Improving the design of an online course with virtual focus group feedback [version 1; peer review: awaiting peer review]

Virtual focus groups played a significant role in guiding the design and development of an innovative professional development program for postdoctoral scholars, called The Postdoc Academy. The primary goal of obtaining feedback from postdocs was to ensure the program content is relevant, approachable, and inclusive for participants of all backgrounds, career stages, professional aspirations, and disciplines. The data collected in 13 focus groups shaped the final content and structure for the Massive Open Online Course (MOOC), ‘The Postdoc Academy: Succeeding as a Postdoc’. Evaluation of participant experience in a post-focus group survey suggests that engaging a target audience is an effective approach to obtain participant feedback and engage learners in the material. Content and activities modified by this feedback were highly rated by course participants in self-reported post-module evaluations. This article describes a method on how structured virtual focus groups of diverse future course participants can provide valuable feedback on developing the content and structure of professional development programming.


Introduction
Professional development is critical for postdoctoral scholars (postdocs) to develop skills beyond the technical focus of their research (Davis, 2009;Gibbs et al., 2015). A recent study highlights the lack of postdoc training in precisely the skills necessary for success in the current job market, including career preparation, leadership, and collaboration (Alund et al., 2020). To address this training gap, our team created online and supplemental in-person professional development programming for postdocs, based on the National Postdoctoral Association core competencies (NPA Core Competencies, n.d.). The Postdoc Academy implements an effective model for online professional development incorporating active learning, participant reflection, and skill building (Hokanson et al., 2019). The goal of the Postdoc Academy is to create a comprehensive and flexible professional development program targeting the needs of postdocs that develops transferable skills to enable their success in a diverse set of careers. Although the program is based in the United States, the program is open to postdocs from all countries. To date, more than 60 countries have been represented by the program registrants.
Prior to the launch of a Massive Open Online Course (MOOC) titled 'The Postdoc Academy: Succeeding as a Postdoc', the program team implemented virtual focus groups to guide program design, development, and implementation. The goals of the postdoc focus groups were to: 1) align course content and design with the needs of the postdoc community; 2) ensure course content is relevant to postdocs of different disciplines, professional aspirations, career stages, and backgrounds; and 3) integrate the voices, experiences, and stories of postdocs in the content. Focus groups as a data collection approach have been used for many years. Recently, researchers have investigated using virtual focus groups as an effective approach to obtain feedback (Forrestal et al., 2015;Lobe, 2017;Matthews et al., 2018;Tates et al., 2009). Compared to in-person focus groups, virtual focus groups have been shown to have equivalent participation rates, participant satisfaction, and consistent numbers of unique ideas generated and relevant comments from participants (Underhill & Olmsted, 2003).
With similar outcomes as in-person focus groups, virtual focus groups have additional advantages. Virtual focus groups can reach a more diverse population compared to in-person focus groups. Based on the target audience of the postdoctoral community, the program team wanted to increase the likelihood of achieving a representative sample. Participants engage with the content online and are isolated in the virtual environment of the focus groups, often working through activities in silence, partially simulating an experience similar to the online course. Forrestal et al. (2015) and Krueger (1994) suggest that, since participants can turn off their video and choose their display name, a sense of privacy exists in virtual focus groups that allows for more open discussion of personal or sensitive topics. Another potential benefit is the lower cost compared to in-person focus groups, since no expenses for travel, room, or food were incurred. This article will share the process used by the Postdoc Academy to facilitate virtual focus groups and how engaging a target audience is an effective approach to obtain participant feedback and engage learners in professional development programming.

Overview of virtual focus groups
Between April and July 2019, virtual focus groups were conducted with participants from the program's intended audience: postdocs. The content reviewed was part of a MOOC, titled the Postdoc Academy: Succeeding as a Postdoc (see the edX course and http://www.postdocacademy.org/explore/). Modules within the MOOC include: 1) finding success as a postdoc; 2) building an actionable career plan; 3) developing resilience; and 4) working effectively in an intercultural environment. Over the course of three weeks, three focus groups were held for each of the four content modules, with the exception of one module (Developing Resilience) that had an additional focus group to review significant modifications implemented based on the initial focus group feedback. Each session was capped at approximately ten participants. On average, focus groups had seven participants, which is consistent with focus group best practices to include six to eight participants in each session (Sweet, 2001).
As inclusivity and transparency are central to the goals of the program, we prioritized these themes in developing this feedback group approach. Social presence, or a sense of community, is critical to maximize participation and foster open discussion in an online environment (Akyol & Garrison, 2008;Gunawardena & Zittle, 1997). Transparency is another key factor to promote affinity in virtual communities (Dalsgaard & Paulsen, 2009). We aimed to promote a sense of transparency in the focus groups such that participants understood the goals and process of content creation, as well as the rationale for focus group design.
Marketing and recruitment for virtual focus groups Social media (Facebook, Twitter, and Instagram) were used to market the virtual focus groups to the postdoc community. In addition, newsletters from the Postdoc Academy, the four collaborating institutions, and the program partners (postdoc associations, postdoc offices, international offices, and professional societies) shared marketing information. Participants were allowed to participate in one session per module. The only criteria for focus group participation was status as a postdoc and focus group registration and attendance.

Pre-session guidelines
Participants were sent an email reminder for the focus group with logistics and the list of questions that would be asked. They completed a pre-session survey that served to collect demographic information for National Institutes of Health reporting, such as gender and ethnicity, time in postdoc position (in years), discipline, and institution.

Focus group facilitation
All virtual focus groups were hosted on the Zoom platform, chosen as an institutionally supported platform and its ease of use. Each focus group was scheduled for two hours and facilitated by two program team members, joining from separate rooms and devices. The lead facilitator reviewed the module content using screen share, guided the discussion, and moderated live polling during the session. The secondary facilitator took notes and responded to participant questions and comments in the chat window.
Beginning the focus group • Introductions: At the beginning of the focus group, each facilitator introduced themselves, with the lead facilitator then calling on each participant to introduce themselves.
• Video optional: While the facilitators had their video turned on, it was communicated to participants this was not a requirement.
• Technical check: The facilitators checked for technical difficulties.
• Program introduction: The lead facilitator shared PowerPoint slides on their screen to provide an overview of the program and the module being discussed. At the time of focus group facilitation, the course content was primarily in Google Docs and Google Slides. Because of this, the participants were asked to be mindful that this work is still in progress and the final course on the edX platform may look different.
• Types of feedback: The participants also were informed that there are no right or wrong answers to the discussion prompts and live poll questions, and that the program team is looking for positive and negative constructive feedback.
• Data use and management: The lead facilitator then informed participants that the session would be recorded. Participants were also informed of how data would be used. The data collected would be anonymized and used for evaluation and improvement of content, as well as conference presentations or manuscripts in support of the pedagogical approach.
• Confidentiality: Participants were asked to maintain confidentiality of the content that was reviewed during the session.
Begin recording. The recording of the focus group then began, and facilitators shared how the session would be facilitated. Participants were encouraged to follow along using an outline of the module content that contained links to video scripts, activities, and discussion prompts. The two formats of obtaining feedback were also discussed: anonymous live polling questions and open discussion questions. The participants also received a worksheet to add additional written feedback after the session (see extended data (Young et al., 2020)).

Content review and discussion
• Module content review: The lead facilitator provided an overview of the module. The learning objectives, context or framing of the content, and main components of the module were shared. The facilitators used incremental content sharing (approximately 10 minutes at a time) with group discussions interspersed. In doing so, the format of interaction was consistently varied throughout the virtual focus group, which may have contributed to high participant engagement.
• Discussion and polling: Live polling and group discussion questions were used to prompt discussion in the focus groups (see extended data (Young et al., 2020)). On average, each session used six live poll questions and 12 discussion prompts. The multiple-choice poll questions and discussion prompts were drafted by each module development team.
• Facilitation methods: Facilitators asked open-ended questions, one at a time, and asked for clarification when a response was unclear. Participants spoke in turn without reliance on hand-raising features. Comments shared in the chat feature were highlighted by facilitators verbally. Facilitators remained neutral and avoided leading questions. If a participant was not participating, the facilitator would call on the participant by name to share their answer.

Concluding the session.
After all of the discussion prompts had been asked, the lead facilitator closed the session with a few reminders. First, participants were reminded of the worksheet that they could use to provide additional written feedback. Second, participants were informed of how they would receive an Amazon gift card for their participation. Finally, participants were thanked for their time and contributions to the program.

Post-session follow-up
After the session, participants were emailed within two days to remind them of the worksheet for providing written feedback and to update them on the status of their gift card. The electronic Amazon gift card ($75 each) was emailed to participants within one week of participating in the focus group.
Synthesizing and sharing feedback After each focus group, the facilitators analyzed the live polling results and each added additional notes about the discussion prompts to capture the thoughts and inputs of both facilitators. Once focus groups were completed for each module, the facilitators summarized the findings and shared them with the module development team. Each module development team reviewed and discussed the feedback to modify content within the online course.
Evaluating the efficacy of virtual focus groups Within two weeks of the conclusion of the focus group series, a Qualtrics survey was emailed to participants to obtain feedback on their virtual focus group experience. This anonymous survey contained 19 multiple choice and free response questions and took an estimated 7 minutes to complete. Upon survey completion, participants had the opportunity to enter a raffle for an additional $75 Amazon gift card. Data were anonymized and quantitative results analyzed by frequency.
Thematic analysis of open-response questions was conducted, using Braun and Clark's framework for thematic data analysis (Braun & Clark, 2006;Clark & Braun, 2013).

Ethical considerations
The focus groups in this article were determined to not be human subjects research by the Boston University IRB (#4046X), thus written consent was not required. An IRB (#5419X) from Boston University has been obtained to perform research on the course data. This article focuses on the methods of the focus groups, rather than creating generalizable knowledge from the course data.

Focus group participants
Average number of participants in each focus group was seven, and ranged from 4 -11. All focus group participants were postdocs at a United States-based institution of higher education (eight institutions represented). A total of 53 postdocs participated in focus groups, with approximately half of respondents (53%) attending one focus group, and 47% attending more than one session. Of those who filled out the pre-focus group survey (94% response rate, 49 out of 53 participants), the majority of respondents (92%, 45 out of 49 respondents) were in their first 3 years as a postdoc. Most respondents (78%, 38 out of 49 respondents) were classified as STEM, 69% (34 out of 49 respondents) identified as female and 31% (15 out of 49 respondents) identified as male (see Table 1). One respondent participated in all four module focus groups, as well as the additional focus group for "Developing Resilience".
Evaluating the efficacy of virtual focus groups Three types of feedback to evaluate the efficacy of virtual focus groups were collected: 1) post-focus group survey; 2) facilitator reflections; and 3) post-module and post-course surveys after the online course. Based on all three sources of feedback, significant changes were made to the course content.
In evaluation of the online course, course participants were highly satisfied with the course.
Results of the post-focus group survey indicate that focus group participants had an overall positive experience. We asked participants about their experience in virtual focus groups, using a 4-point Likert scale from strongly agree to strongly disagree. Of the 30 survey respondents, 33% (10) strongly agreed and 67% (20) agreed that the online format was effective for learning about the content. Next, 33% (10) strongly agreed and 67% (20) agreed that the content presented in the focus groups was suitable for people from different cultural backgrounds. Finally, 60% (18) strongly agreed and 40% (12) agreed that their contributions to the discussion were valued.
To demonstrate the types of feedback we received in focus groups and the changes made to the online course based on that feedback, we will use the module titled "Developing Resilience" because major modifications were made to the course content after virtual focus groups (Table 2). One example of changes to the online course was based on the feedback received when focus group participants were asked if the module content is approachable and relatable for a diverse audience. The feedback indicated that the content seemed to be American-centric. One participant commented "...we need to see lots of different people from different backgrounds, foreign and domestic." Since the feedback suggested that the content is approachable but not relatable for all postdocs, the program  team revised the module content by: 1) incorporating the stories and perspectives of international postdocs; 2) discussing how international postdocs might face barriers to resilience compared to domestic postdocs; and 3) highlighting strategies that are used in other cultures to build resilience.
Another example of modifying course content based on focus group feedback was in alignment of course content with the learning objectives. The program team identified a learning objective to support participant resilience across the worklife continuum, and the feedback from focus group participants indicated that the course content was not aligned. One commented, "...it felt more like work resilience than work-life resilience or work and life resilience." Generally, participants felt that the content was more representative of professional resilience than resilience that spans the personal and professional aspects of life. With this feedback, the program team revised content to emphasize that resilience spans personal and professional components of one's life by: 1) incorporating literature on recovery internal and external to the workplace; 2) sharing interviews of individuals discussing resilience in personal and professional contexts; and 3) highlighting strategies to build resilience beyond work.
Considering the extent of feedback during the focus groups that reviewed "Developing Resilience", the same group of participants were invited to an additional focus group (n=4) to ensure that the revised content was relatable, engaging, and informative for participants. Participants confirmed that the implemented revisions improved the course content. This was an important step in the iterative feedback process before launching and evaluating the online course.
Evaluation of the online course with post-module and postcourse surveys indicated that the changes made based on the focus group feedback were well-received. Not all focus group participants took the online course and course surveys, therefore, the results reported in Table 3 are a summary from all course participants who completed course surveys (Underlying data (Young et al., 2020)). Overall, 89% of survey respondents (94 out of 103) were satisfied with the course. Additionally, 91% of survey respondents (84 out of 92) indicated that they would recommend the course to a friend. Table 3 displays the evaluation results for "Developing Resilience". The survey items are aligned with the learning objectives and short-term outcomes identified for the module. These results indicate that the online course content, including the major modifications made to "Developing Resilience" based on focus group feedback, was well-received by course participants.
Evaluating the experience of focus group participants In addition to evaluating the efficacy of virtual focus groups, we also wanted to evaluate the experience of focus group participants. Figure 1 displays the reasons postdocs said they were motivated to participate in the focus groups. Participants were able to indicate more than one reason. An intrinsic motivation to contribute intellectually as described in the literature is altruism, the enjoyment of helping others (Kankanhalli et al., 2005). One of the most frequent survey responses was to provide feedback to guide program development, which suggests altruism may be a motivator in these virtual focus groups.
To gain insight into the experience of focus group participants and improve the experience for future sessions, the Critical Incident Questionnaire was used (Brookfield, 1995). As displayed in Table 4, participants indicated they were most engaged while interacting with other postdocs and when offering specific feedback. While the half of the survey respondents (53%, 10 out of 19 respondents) indicated there was not a time when they were not engaged, others indicated they were least Table 2. Summary and representative comments of open-ended discussion questions from one set of focus groups that reviewed "Developing Resilience".

Changes to online course based on focus group feedback
Is the content in this module approachable and relatable for a diverse audience?
Since resilience can be defined differently by different cultures, this content seems to be American-centric. It's approachable but might not be relatable for all postdocs.
Representative comment: "This is strongly missing. We need to see lots of different people from different backgrounds, foreign and domestic." What components of the graphic do you like? Are there components of the graphic that could be improved?
Overall yes, but a few small tweaks would make it easier to follow.   Participants could select more than one motivator. engaged while reviewing the material and while slides were being presented.

Discussion
In summary, we developed and implemented a model of high-engagement virtual focus groups, received quality feedback, made significant changes to the course content, and the resulting content was well-received by course participants. We evaluated the experience of participants to ensure a well-defined process for facilitating virtual focus groups and ensure that participants felt valued for their contributions. From this work, multiple recommendations for improving professional development programming emerged.
Virtual focus groups are an effective, low-cost method to receive quality feedback and ensure programming is engaging, inclusive, and relevant to the target audience. Creating an inclusive and transparent environment for focus group participants can lead to higher engagement. Reviewing the content in small amounts and creating space for open discussion are two key takeaways. We envision this process would be beneficial for those who support graduate and postdoctoral professional development.
To improve upon the process described here, the program team plans to expand recruitment to reach an audience that more accurately represents the target audience, the postdoc population. To do so, we will invite postdocs at non-academic institutions (government labs, industry, agencies). Additionally, we will consider including other key stakeholders in addition to future participants, such as faculty and staff who support postdocs in separate focus groups to gain insight to other perspectives.
Although there are many benefits to virtual focus groups, there are limitations for implementing this process that should be considered. One limitation can be the technical barriers with the web conferencing service used to host the focus groups  (Kite & Phongsavan, 2017). It's critical that facilitators consider the pros and cons of different platforms, including strategies to enhance privacy and security, and test the web conferencing service. Additionally, we suggest providing detailed instructions on how to access the platform and allow additional time in each session to ensure there are no technical difficulties.
Another limitation can be the limited number of participants who were included in the virtual focus groups. The postdocs who participated in the focus groups might have a higher level of engagement in professional development activities relative to the postdoc population. Thus, the feedback received and the resulting course content may cater to course participants with similar identities and experiences as the focus group participants. To be more representative of the course participants, the focus groups could be held in a number of different environments. Additionally, not all focus group participants completed the online course, which limits analysis of the changes made based on focus group feedback.
In conclusion, structured virtual focus groups of diverse future participants provided valuable feedback on the content and structure of an online course for postdocs. Evaluating the experience of focus group participants provided insights into the benefits of using this approach and has helped improve the process for the future. Facilitators must consider the benefits and potential limitations when planning, implementing, and evaluating virtual focus groups as a method of soliciting input on a professional development program.

Data availability
The transcripts of the focus groups and facilitator reflections contain identifying information about the participants. In order to keep the focus group participants anonymous, the focus group transcripts and facilitator reflections are not included in the underlying data. Upon request, deidentified transcripts can be provided.

Underlying data
Harvard Dataverse: Improving the design of an online course with virtual focus group feedback. https://doi.org/10.7910/DVN/ ZBWSHS (Young et al., 2020) This project contains the following underlying data: The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com