Web-based questionnaires: Lessons learned from practical implementation of a pharmacoepidemiological study

Introduction

A pharmacoepidemiological study using a web-based questionnaire has similar characteristics regarding reliability and validity of traditional modes of data collection, such as postal questionnaires^1–5. With the advance of the Internet, web-based questionnaires are also believed to convey many benefits over traditional methods^6,7. They are easier to manage and wide-reaching in comparison to traditional recruitment methods and may be returned more rapidly than postal questionnaires. Anonymity is also fairly well preserved^7,8. Web-based questionnaires can also embed context logic and forced-choice formats, which allows the researcher to adjust the questionnaire based on a previous response. This may significantly improve the quality of the data collected.

Internet-based surveys are supposed to be less costly than performing face-to-face interviews or delivering postal questionnaires to participants, and since the data is automatically stored, errors due to the manual entry of the data in a database are minimised. Nevertheless, the advantages of the online methods depend on factors such as the age of the target population and their familiarity with the Internet^8,9.

In 2015, we developed a pharmacoepidemiological study to estimate the prevalence of methylphenidate use for cognitive enhancement among undergraduate and graduate students attending a major Brazilian university situated in the southeast of the country, the Universidade Federal de Minas Gerais (UFMG) (unpublished study; Candido R, Perini E, Menezes de Pádua C, Junqueira DRJ). We hypothesised that using a web-based questionnaire would offer a practical approach to the research and have greater acceptability among the youth and young adults that were our target audience.

Our experience in this research suggests that researchers designing pharmacoepidemiological studies may be unaware of methodological challenges surrounding the design and implementation of web-based questionnaires. From choosing an online service to design and host the questionnaire, to the delivery of the email invitation and analysis, we faced numerous challenges that could not be fully anticipated, which are discussed in this article. The challenges were not related to any of our research fields or common areas of expertise (epidemiology, pharmacoepidemiology, and statistics) and they could significantly impact any research plans. Scientists from developing countries, or limited access to financial and other kinds of support, may benefit from exercising special care when planning a web-based survey.

Methods

We evaluated five platforms to host our electronic questionnaire: Wuffo^® (https://www.wufoo.com/), KwikSurveys^® (https://kwiksurveys.com/), Survs^® (https://survs.com/), SurveyMonkey^® (https://www.surveymonkey.co.uk/) and Google Forms^® (https://www.google.co.uk/forms/about/). Our assessment considered costs and functionalities to export the database to other programs (eg. Excel^®), availability of logic resources for setting the questions, and resources to set character limits to the open answers. Only KwikSurveys^® and SurveyMonkey^® platforms allowed us to build a structured questionnaire with forced-answer formats and unlimited context logic, but SurveyMonkey^® presented a better cost-benefit. The Platinum modality of SurveyMonkey^® (current renamed to Gold plan) also allowed us to track the respondent email, and this feature was essential to avoid multiple answers from the same participant.

Students enrolled in undergraduate, postgraduate and residency courses were selected through random sampling (using a number generator software) and invited to answer the survey through their electronic addresses (email), which was provided by the university. The invitation email contained an explanation of the research and a link to access the questionnaire hosted at SurveyMonkey^®. Informed consent was obtained from all participants answering the survey. Reminder messages were directed to the email of non-respondents in scheduled periods of 15 days, and after two reminder messages, non-respondents were replaced.

We calculated a sample size of 365 students considering a prevalence of consumption of methylphenidate for cognitive enhancement equal to 7%^10,11, a sampling error of 4%, a significance of 95%, and power of 80%.

Results and Discussion

Choosing the online platform among the many available was the first challenging aspect we faced. There are several platforms available and they can offer many modalities of plans with diverse functionalities (Table 1). Researchers not familiar with Information Science may face hard and confusing decisions. For instance, some plans may limit the amount of questions included in a survey, not offer advanced survey logic (e.g., advanced branching, advanced piping, and other functionalities to enhance data collection), or even limit the number of people in a team to access the survey dashboard or the number of responses per survey. We suggest researchers carefully revise these aspects to ensure they are choosing an online platform suitable to what they need to build and deliver in their survey.

Our questionnaire was fairly straightforward and comprised 29 questions. Nevertheless, setting up the navigation flow and questions structure was time-consuming, requiring several pilot phases and numerous adjustments. Although all online platforms may appear user-friendly, setting up multiple-choice questions, skipping rules and piping the questionnaire is not a straightforward task for a person with regular software skills. We realised that having in our team the support of an Information Science specialist would allow us to build a questionnaire faster and with a better structure and design. However, budget constraints prevented us from accessing this kind of support, and our university did not offer this service.

We collected answers from 378 students; therefore, we achieved our estimated sample size. Our sample was also representative of the university population. However, we had to invite 2,384 students due to recruitment attrition, yielding a response rate of 15.8%. Considering that our target population can be regarded as a selected one, generally easy to reach and fairly collaborative, the response rate was poor. We later learned that spam filters of the university email system and other email providers identified our invitation email as spam because it included the link to the web-based questionnaire. To overcome this problem, we contacted the students' departments of all faculties asking them to advertise the research and encourage the participation of the students. The encouragement message also asked students to check their spam box to notice the survey invitation. We also advertised the research on the university website. We considered the possibility of using a dedicated website or emailing the invitations through the university server to all students; however, we did not have the budget nor expert support to develop a dedicated website, and emailing the invitations to all students would impact our random sampling procedure.

The analytical processing of the data was also an unexpected problematic issue. All export file formats required excessive data cleaning. Remarkably, answers to multiple-choice questions and questions built with context logic would be broken in different lines and columns in a way that required an overall revision of the entries to aggregate data from each single respondent and allow analysis. This process demanded a significant amount of effort and time. Our experience suggests that studies with a larger number of responses would become unviable if facing similar problems. To avoid these problems, researchers may be aware, in advance, that although user-friendly, the platforms available to host web-based surveys require some expert knowledge in the development, manipulation, and retrieval of electronic data and intelligent systems. Basic and even advanced skills in epidemiology, sociology, statistics, and related disciplines may not be sufficient in this context.

Conclusions

The design and implementation of a web-based questionnaire may carry several challenges, and pharmacoepidemiologists may not be familiar with them. Moreover, although more economically viable, the availability of resources to ensure access to an appropriate platform and to specialised support is essential to the success of the survey. The lessons we learned urge us to strongly recommend that this mode of data collection may be carried out by multidisciplinary teams and with the appropriate support from a team of specialists in Information Science. Our experience can also be promptly generalised to other research fields where the features of the web-based questionnaire may be of interest.

Ethical approval and informed consent

Our study received ethical approval from the Universidade Federal de Minas Gerais (approval number, 441.603). Informed consent was obtained from all individual participants included in the study.

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.