F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.26289.1 Method Article Articles Improving the design of an online course with virtual focus group feedback

[version 1; peer review: 1 approved, 1 approved with reservations]

 Young Celine Conceptualization Data Curation Formal Analysis Investigation Methodology Project Administration Visualization Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-6956-3966 1 Chesniak Olivia M. Conceptualization Data Curation Formal Analysis Investigation Methodology Visualization Writing – Original Draft Preparation Writing – Review & Editing 2 Drane Denise Conceptualization Methodology Writing – Review & Editing 2 Campa III Henry Writing – Review & Editing 3 Green Noah Writing – Review & Editing 4 Greenler Robin Writing – Review & Editing 5 Middlemis Maher Jessica Writing – Review & Editing 5 McGee Richard Writing – Review & Editing 2 Nunez Antonio Writing – Review & Editing 3 Goldberg Bennett B. Conceptualization Funding Acquisition Methodology Supervision Writing – Review & Editing 2 Chobot Hokanson Sarah Conceptualization Funding Acquisition Methodology Supervision Writing – Review & Editing https://orcid.org/0000-0002-7348-4312 a 1 Boston University, Boston, MA, 02215, USA Northwestern University, Evanston, IL, 60208, USA Michigan State University, East Lansing, MI, 48824, USA Green Scientific and Educational Consulting, Clarksville, MD, 21029, USA University of Wisconsin-Madison, Madison, WI, 53715, USA sch1@bu.edu

No competing interests were disclosed.

 1 10 2020 2020 9 1191 15 9 2020 Copyright: © 2020 Young C et al. 2020 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Virtual focus groups played a significant role in guiding the design and development of an innovative professional development program for postdoctoral scholars, called The Postdoc Academy. The primary goal of obtaining feedback from postdocs was to ensure the program content is relevant, approachable, and inclusive for participants of all backgrounds, career stages, professional aspirations, and disciplines. The data collected in 13 focus groups shaped the final content and structure for the Massive Open Online Course (MOOC), ‘The Postdoc Academy: Succeeding as a Postdoc’. Evaluation of participant experience in a post-focus group survey suggests that engaging a target audience is an effective approach to obtain participant feedback and engage learners in the material. Content and activities modified by this feedback were highly rated by course participants in self-reported post-module evaluations. This article describes a method on how structured virtual focus groups of diverse future course participants can provide valuable feedback on developing the content and structure of professional development programming.

 virtual focus groups massive open online course evaluation instructional design postdoctoral scholars professional development National Institute of General Medical Sciences R25GM121257 This work is funded by the National Institutes of General Medical Sciences of the National Institutes of Health under Award Number R25GM121257. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Introduction

Professional development is critical for postdoctoral scholars (postdocs) to develop skills beyond the technical focus of their research ( Davis, 2009; Gibbs et al., 2015). A recent study highlights the lack of postdoc training in precisely the skills necessary for success in the current job market, including career preparation, leadership, and collaboration ( Alund et al., 2020). To address this training gap, our team created online and supplemental in-person professional development programming for postdocs, based on the National Postdoctoral Association core competencies ( NPA Core Competencies, n.d.). The Postdoc Academy implements an effective model for online professional development incorporating active learning, participant reflection, and skill building ( Hokanson et al., 2019). The goal of the Postdoc Academy is to create a comprehensive and flexible professional development program targeting the needs of postdocs that develops transferable skills to enable their success in a diverse set of careers. Although the program is based in the United States, the program is open to postdocs from all countries. To date, more than 60 countries have been represented by the program registrants.

Prior to the launch of a Massive Open Online Course (MOOC) titled ‘The Postdoc Academy: Succeeding as a Postdoc’, the program team implemented virtual focus groups to guide program design, development, and implementation. The goals of the postdoc focus groups were to: 1) align course content and design with the needs of the postdoc community; 2) ensure course content is relevant to postdocs of different disciplines, professional aspirations, career stages, and backgrounds; and 3) integrate the voices, experiences, and stories of postdocs in the content. Focus groups as a data collection approach have been used for many years. Recently, researchers have investigated using virtual focus groups as an effective approach to obtain feedback ( Forrestal et al., 2015; Lobe, 2017; Matthews et al., 2018; Tates et al., 2009). Compared to in-person focus groups, virtual focus groups have been shown to have equivalent participation rates, participant satisfaction, and consistent numbers of unique ideas generated and relevant comments from participants ( Underhill & Olmsted, 2003).

With similar outcomes as in-person focus groups, virtual focus groups have additional advantages. Virtual focus groups can reach a more diverse population compared to in-person focus groups. Based on the target audience of the postdoctoral community, the program team wanted to increase the likelihood of achieving a representative sample. Participants engage with the content online and are isolated in the virtual environment of the focus groups, often working through activities in silence, partially simulating an experience similar to the online course. Forrestal et al. (2015) and Krueger (1994) suggest that, since participants can turn off their video and choose their display name, a sense of privacy exists in virtual focus groups that allows for more open discussion of personal or sensitive topics. Another potential benefit is the lower cost compared to in-person focus groups, since no expenses for travel, room, or food were incurred.

This article will share the process used by the Postdoc Academy to facilitate virtual focus groups and how engaging a target audience is an effective approach to obtain participant feedback and engage learners in professional development programming.

 Methods Overview of virtual focus groups

Between April and July 2019, virtual focus groups were conducted with participants from the program’s intended audience: postdocs. The content reviewed was part of a MOOC, titled the Postdoc Academy: Succeeding as a Postdoc (see the edX course and http://www.postdocacademy.org/explore/). Modules within the MOOC include: 1) finding success as a postdoc; 2) building an actionable career plan; 3) developing resilience; and 4) working effectively in an intercultural environment. Over the course of three weeks, three focus groups were held for each of the four content modules, with the exception of one module (Developing Resilience) that had an additional focus group to review significant modifications implemented based on the initial focus group feedback. Each session was capped at approximately ten participants. On average, focus groups had seven participants, which is consistent with focus group best practices to include six to eight participants in each session ( Sweet, 2001).

As inclusivity and transparency are central to the goals of the program, we prioritized these themes in developing this feedback group approach. Social presence, or a sense of community, is critical to maximize participation and foster open discussion in an online environment ( Akyol & Garrison, 2008; Gunawardena & Zittle, 1997). Transparency is another key factor to promote affinity in virtual communities ( Dalsgaard & Paulsen, 2009). We aimed to promote a sense of transparency in the focus groups such that participants understood the goals and process of content creation, as well as the rationale for focus group design.

 Marketing and recruitment for virtual focus groups

Social media (Facebook, Twitter, and Instagram) were used to market the virtual focus groups to the postdoc community. In addition, newsletters from the Postdoc Academy, the four collaborating institutions, and the program partners (postdoc associations, postdoc offices, international offices, and professional societies) shared marketing information. Participants were allowed to participate in one session per module. The only criteria for focus group participation was status as a postdoc and focus group registration and attendance.

 Pre-session guidelines

Participants were sent an email reminder for the focus group with logistics and the list of questions that would be asked. They completed a pre-session survey that served to collect demographic information for National Institutes of Health reporting, such as gender and ethnicity, time in postdoc position (in years), discipline, and institution.

 Focus group facilitation

All virtual focus groups were hosted on the Zoom platform, chosen as an institutionally supported platform and its ease of use. Each focus group was scheduled for two hours and facilitated by two program team members, joining from separate rooms and devices. The lead facilitator reviewed the module content using screen share, guided the discussion, and moderated live polling during the session. The secondary facilitator took notes and responded to participant questions and comments in the chat window.

 Beginning the focus group

Introductions: At the beginning of the focus group, each facilitator introduced themselves, with the lead facilitator then calling on each participant to introduce themselves.

Video optional: While the facilitators had their video turned on, it was communicated to participants this was not a requirement.

Technical check: The facilitators checked for technical difficulties.

Program introduction: The lead facilitator shared PowerPoint slides on their screen to provide an overview of the program and the module being discussed. At the time of focus group facilitation, the course content was primarily in Google Docs and Google Slides. Because of this, the participants were asked to be mindful that this work is still in progress and the final course on the edX platform may look different.

Types of feedback: The participants also were informed that there are no right or wrong answers to the discussion prompts and live poll questions, and that the program team is looking for positive and negative constructive feedback.

Data use and management: The lead facilitator then informed participants that the session would be recorded. Participants were also informed of how data would be used. The data collected would be anonymized and used for evaluation and improvement of content, as well as conference presentations or manuscripts in support of the pedagogical approach.

Confidentiality: Participants were asked to maintain confidentiality of the content that was reviewed during the session.

Begin recording. The recording of the focus group then began, and facilitators shared how the session would be facilitated. Participants were encouraged to follow along using an outline of the module content that contained links to video scripts, activities, and discussion prompts. The two formats of obtaining feedback were also discussed: anonymous live polling questions and open discussion questions. The participants also received a worksheet to add additional written feedback after the session (see extended data ( Young et al., 2020)).

Content review and discussion

Module content review: The lead facilitator provided an overview of the module. The learning objectives, context or framing of the content, and main components of the module were shared. The facilitators used incremental content sharing (approximately 10 minutes at a time) with group discussions interspersed. In doing so, the format of interaction was consistently varied throughout the virtual focus group, which may have contributed to high participant engagement.

Discussion and polling: Live polling and group discussion questions were used to prompt discussion in the focus groups (see extended data ( Young et al., 2020)). On average, each session used six live poll questions and 12 discussion prompts. The multiple-choice poll questions and discussion prompts were drafted by each module development team.

Facilitation methods: Facilitators asked open-ended questions, one at a time, and asked for clarification when a response was unclear. Participants spoke in turn without reliance on hand-raising features. Comments shared in the chat feature were highlighted by facilitators verbally. Facilitators remained neutral and avoided leading questions. If a participant was not participating, the facilitator would call on the participant by name to share their answer.

Concluding the session. After all of the discussion prompts had been asked, the lead facilitator closed the session with a few reminders. First, participants were reminded of the worksheet that they could use to provide additional written feedback. Second, participants were informed of how they would receive an Amazon gift card for their participation. Finally, participants were thanked for their time and contributions to the program.

 Post-session follow-up

After the session, participants were emailed within two days to remind them of the worksheet for providing written feedback and to update them on the status of their gift card. The electronic Amazon gift card ($75 each) was emailed to participants within one week of participating in the focus group.

 Synthesizing and sharing feedback

After each focus group, the facilitators analyzed the live polling results and each added additional notes about the discussion prompts to capture the thoughts and inputs of both facilitators. Once focus groups were completed for each module, the facilitators summarized the findings and shared them with the module development team. Each module development team reviewed and discussed the feedback to modify content within the online course.

Evaluating the efficacy of virtual focus groups

Within two weeks of the conclusion of the focus group series, a Qualtrics survey was emailed to participants to obtain feedback on their virtual focus group experience. This anonymous survey contained 19 multiple choice and free response questions and took an estimated 7 minutes to complete. Upon survey completion, participants had the opportunity to enter a raffle for an additional $75 Amazon gift card. Data were anonymized and quantitative results analyzed by frequency. Thematic analysis of open-response questions was conducted, using Braun and Clark’s framework for thematic data analysis ( Braun & Clark, 2006; Clark & Braun, 2013).

 Ethical considerations

The focus groups in this article were determined to not be human subjects research by the Boston University IRB (#4046X), thus written consent was not required. An IRB (#5419X) from Boston University has been obtained to perform research on the course data. This article focuses on the methods of the focus groups, rather than creating generalizable knowledge from the course data.

 Results Focus group participants

Average number of participants in each focus group was seven, and ranged from 4 – 11. All focus group participants were postdocs at a United States-based institution of higher education (eight institutions represented). A total of 53 postdocs participated in focus groups, with approximately half of respondents (53%) attending one focus group, and 47% attending more than one session. Of those who filled out the pre-focus group survey (94% response rate, 49 out of 53 participants), the majority of respondents (92%, 45 out of 49 respondents) were in their first 3 years as a postdoc. Most respondents (78%, 38 out of 49 respondents) were classified as STEM, 69% (34 out of 49 respondents) identified as female and 31% (15 out of 49 respondents) identified as male (see Table 1). One respondent participated in all four module focus groups, as well as the additional focus group for “Developing Resilience”.

Characteristics of focus group participants based on the pre-session survey (n=49, 94% response rate).
Characteristic Frequency % of respondents
Time in postdoc position: Less than 1 year 2–3 years 3–4 years 4–5 years More than 5 years 27 18 0 0 4 55 37 0 0 8
Discipline: STEM Social Sciences Humanities 38 10 1 78 20 2
Gender: Female Male Prefer not to say Prefer to self-describe 34 15 0 0 69 31 0 0
Ethnicity: Asian Black/African American Caucasian Hispanic/Latino Prefer not to say Prefer to self-describe 7 5 33 0 4 0 15 10 67 0 8 0
Number of Sessions Attended: 1 2 3 4 5 26 5 10 7 1 53 10 21 14 2
Evaluating the efficacy of virtual focus groups

Three types of feedback to evaluate the efficacy of virtual focus groups were collected: 1) post-focus group survey; 2) facilitator reflections; and 3) post-module and post-course surveys after the online course. Based on all three sources of feedback, significant changes were made to the course content. In evaluation of the online course, course participants were highly satisfied with the course.

Results of the post-focus group survey indicate that focus group participants had an overall positive experience. We asked participants about their experience in virtual focus groups, using a 4-point Likert scale from strongly agree to strongly disagree. Of the 30 survey respondents, 33% (10) strongly agreed and 67% (20) agreed that the online format was effective for learning about the content. Next, 33% (10) strongly agreed and 67% (20) agreed that the content presented in the focus groups was suitable for people from different cultural backgrounds. Finally, 60% (18) strongly agreed and 40% (12) agreed that their contributions to the discussion were valued.

To demonstrate the types of feedback we received in focus groups and the changes made to the online course based on that feedback, we will use the module titled “Developing Resilience” because major modifications were made to the course content after virtual focus groups ( Table 2). One example of changes to the online course was based on the feedback received when focus group participants were asked if the module content is approachable and relatable for a diverse audience. The feedback indicated that the content seemed to be American-centric. One participant commented “... we need to see lots of different people from different backgrounds, foreign and domestic.” Since the feedback suggested that the content is approachable but not relatable for all postdocs, the program team revised the module content by: 1) incorporating the stories and perspectives of international postdocs; 2) discussing how international postdocs might face barriers to resilience compared to domestic postdocs; and 3) highlighting strategies that are used in other cultures to build resilience.

Summary and representative comments of open-ended discussion questions from one set of focus groups that reviewed “Developing Resilience”.
Prompt: Discussion questions asked to focus group participant Feedback: Summarized feedback from focus group participants in response to prompt Actions to revise content: Changes to online course based on focus group feedback
Is the content in this module approachable and relatable for a diverse audience? Since resilience can be defined differently by different cultures, this content seems to be American-centric. It’s approachable but might not be relatable for all postdocs. Representative comment: “This is strongly missing. We need to see lots of different people from different backgrounds, foreign and domestic.” - Incorporated the stories and perspectives of international postdocs - Discussed how international postdocs might face different barriers to resilience compared to domestic postdocs - Highlighted strategies to build resilience that are used in other cultures
What components of the graphic do you like? Are there components of the graphic that could be improved? Overall yes, but a few small tweaks would make it easier to follow. Include a specific example of what a postdoc might face. Representative comment: “To show the goal of being more resilient, you could minimize those oscillations and have the line trend upward.” - Used more approachable language in the graphic labels - Weaved postdoc narratives throughout the module - When discussing the goal of developing resilience, demonstrated what could results in on the graphic

Another example of modifying course content based on focus group feedback was in alignment of course content with the learning objectives. The program team identified a learning objective to support participant resilience across the work-life continuum, and the feedback from focus group participants indicated that the course content was not aligned. One commented, “...it felt more like work resilience than work-life resilience or work and life resilience.” Generally, participants felt that the content was more representative of professional resilience than resilience that spans the personal and professional aspects of life. With this feedback, the program team revised content to emphasize that resilience spans personal and professional components of one’s life by: 1) incorporating literature on recovery internal and external to the workplace; 2) sharing interviews of individuals discussing resilience in personal and professional contexts; and 3) highlighting strategies to build resilience beyond work.

Considering the extent of feedback during the focus groups that reviewed “Developing Resilience”, the same group of participants were invited to an additional focus group (n=4) to ensure that the revised content was relatable, engaging, and informative for participants. Participants confirmed that the implemented revisions improved the course content. This was an important step in the iterative feedback process before launching and evaluating the online course.

Evaluation of the online course with post-module and post-course surveys indicated that the changes made based on the focus group feedback were well-received. Not all focus group participants took the online course and course surveys, therefore, the results reported in Table 3 are a summary from all course participants who completed course surveys (Underlying data ( Young et al., 2020)). Overall, 89% of survey respondents (94 out of 103) were satisfied with the course. Additionally, 91% of survey respondents (84 out of 92) indicated that they would recommend the course to a friend. Table 3 displays the evaluation results for “Developing Resilience”. The survey items are aligned with the learning objectives and short-term outcomes identified for the module. These results indicate that the online course content, including the major modifications made to “Developing Resilience” based on focus group feedback, was well-received by course participants.

Results of post-module and post-course surveys to evaluate “Developing Resilience”.

Respondents self-reported agreement or disagreement on a 5-point Likert Item. N=number of survey respondents.

Survey item Respondents who strongly agree or somewhat agree, % Respondents who neither agree nor disagree, % Respondents who strongly disagree or somewhat disagree, %
I can describe the process of resilience as it pertains to personal and professional experiences.      n=93 99 1 0
I can describe behaviors and thoughts that support building resilience.      n=93 96 3 1
I can describe stressors that exist internal and external to the workplace.      n=93 99 1 0
I am more aware of the behaviors that promote and detract from my resilience.      n=93 92 7 1
I have applied the concept of resilience to my own personal and professional experiences.      n=88 91 7 2
I have practiced recovering from stressors that are internal and external to the workplace.      n=88 83 14 3
I feel less stressed about current challenges I am facing.      n=88 65 23 12
Evaluating the experience of focus group participants

In addition to evaluating the efficacy of virtual focus groups, we also wanted to evaluate the experience of focus group participants. Figure 1 displays the reasons postdocs said they were motivated to participate in the focus groups. Participants were able to indicate more than one reason. An intrinsic motivation to contribute intellectually as described in the literature is altruism, the enjoyment of helping others ( Kankanhalli et al., 2005). One of the most frequent survey responses was to provide feedback to guide program development, which suggests altruism may be a motivator in these virtual focus groups.

Participants’ motivation for participating in focus groups in a post-session survey (67% response rate, n=33).

Participants could select more than one motivator.

To gain insight into the experience of focus group participants and improve the experience for future sessions, the Critical Incident Questionnaire was used ( Brookfield, 1995). As displayed in Table 4, participants indicated they were most engaged while interacting with other postdocs and when offering specific feedback. While the half of the survey respondents (53%, 10 out of 19 respondents) indicated there was not a time when they were not engaged, others indicated they were least engaged while reviewing the material and while slides were being presented.

Participants report of when they were most engaged and least engaged during the sessions.
Most engaged during Number of respondents (n=29)
Interacting with other postdocs Representative comment: “When listening to other participants share their thoughts and opinions, which helped me see other points of view regarding the various topics.” 9
When offering feedback Representative comment: “After information was presented and then followed by a series of specific questions about the content.” Representative comment: “When we were asked to do a follow up session - I liked seeing the changes based on the feedback and felt extra appreciated.” 6
Discussing working effectively in an intercultural environment 3
Live polling 3
Receiving the ‘checklist’ for transitioning into a postdoc or other content that was helpful to my own professional development 3
Discussing the impact of the materials 2
All of it 2
Least engaged during Number of respondents: (n=19)
Never 10
During slides 3
Reviewing the material 4
“When I felt like I didn’t have the expertise to offer suggestions about the topic” 1
“When the materials weren’t fully developed - it was hard to understand some of the material based on just schematics or general descriptions” 1
Discussion

In summary, we developed and implemented a model of high-engagement virtual focus groups, received quality feedback, made significant changes to the course content, and the resulting content was well-received by course participants. We evaluated the experience of participants to ensure a well-defined process for facilitating virtual focus groups and ensure that participants felt valued for their contributions. From this work, multiple recommendations for improving professional development programming emerged.

Virtual focus groups are an effective, low-cost method to receive quality feedback and ensure programming is engaging, inclusive, and relevant to the target audience. Creating an inclusive and transparent environment for focus group participants can lead to higher engagement. Reviewing the content in small amounts and creating space for open discussion are two key takeaways. We envision this process would be beneficial for those who support graduate and postdoctoral professional development.

To improve upon the process described here, the program team plans to expand recruitment to reach an audience that more accurately represents the target audience, the postdoc population. To do so, we will invite postdocs at non-academic institutions (government labs, industry, agencies). Additionally, we will consider including other key stakeholders in addition to future participants, such as faculty and staff who support postdocs in separate focus groups to gain insight to other perspectives.

Although there are many benefits to virtual focus groups, there are limitations for implementing this process that should be considered. One limitation can be the technical barriers with the web conferencing service used to host the focus groups ( Kite & Phongsavan, 2017). It’s critical that facilitators consider the pros and cons of different platforms, including strategies to enhance privacy and security, and test the web conferencing service. Additionally, we suggest providing detailed instructions on how to access the platform and allow additional time in each session to ensure there are no technical difficulties.

Another limitation can be the limited number of participants who were included in the virtual focus groups. The postdocs who participated in the focus groups might have a higher level of engagement in professional development activities relative to the postdoc population. Thus, the feedback received and the resulting course content may cater to course participants with similar identities and experiences as the focus group participants. To be more representative of the course participants, the focus groups could be held in a number of different environments. Additionally, not all focus group participants completed the online course, which limits analysis of the changes made based on focus group feedback.

In conclusion, structured virtual focus groups of diverse future participants provided valuable feedback on the content and structure of an online course for postdocs. Evaluating the experience of focus group participants provided insights into the benefits of using this approach and has helped improve the process for the future. Facilitators must consider the benefits and potential limitations when planning, implementing, and evaluating virtual focus groups as a method of soliciting input on a professional development program.

 Data availability

The transcripts of the focus groups and facilitator reflections contain identifying information about the participants. In order to keep the focus group participants anonymous, the focus group transcripts and facilitator reflections are not included in the underlying data. Upon request, deidentified transcripts can be provided.

Underlying data

Harvard Dataverse: Improving the design of an online course with virtual focus group feedback. https://doi.org/10.7910/DVN/ZBWSHS ( Young et al., 2020)

This project contains the following underlying data:

Deidentified Postdoc Academy Feedback Session Survey.tab

Post-Module and Post-Course Survey Results.tab

 Extended data

Harvard Dataverse: Improving the design of an online course with virtual focus group feedback. https://doi.org/10.7910/DVN/ZBWSHS ( Young et al., 2020)

This project contains the following extended data:

Extended data.pdf (contains Appendix A: Example of Focus Group Questions and Appendix B: Example of Live Poll Results).

Postdoc Academy Feedback Session Worksheet.pdf (Sent to participants prior to focus group.)

Postdoc_Academy_Feedback_Session_Survey.pdf (Blank survey sent to participants within 2 weeks of completion of focus groups.)

Postdoc_Academy_Post-Module_4_Survey.pdf (Completed by course participants at the end of Module 4 within “The Postdoc Academy: Succeeding as a Postdoc”.)

Postdoc_Academy_Post-Course_Survey.pdf (Completed by course participants at the end of “The Postdoc Academy: Succeeding as a Postdoc”.)

Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).

 Akyol Z Garrison DR : The Development of a Community of Inquiry over Time in an Online Course: Understanding the Progression and Integration of Social, Cognitive and Teaching Presence. Journal of Asynchronous Learning Networks. 2008;12(3-4). Reference Source Alund M Emery N Jarrett BJM : Academic ecosystems must evolve to support a sustainable postdoc workforce. Nat Ecol Evol. 2020;4:777–781. 32284583 10.1038/s41559-020-1178-6 Braun V Clarke V : Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. 10.1191/1478088706qp063oa Brookfield SD : Becoming a Critically Reflective Teacher. San Francisco: Jossey-Bass.1995. Reference Source Clarke V Braun V : Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. Psychologist. 2013;26(2):120–123. Reference Source Dalsgaard C Paulsen MF : Transparency in Cooperative Online Education. The International Review of Research in Open and Distributed Learning. 2009;10(3). 10.19173/irrodl.v10i3.671 Davis G : Improving the Postdoctoral Experience: An Empirical Approach. Science and Engineering Careers in the United States: An Analysis of Markets and Employment. In: R.B. Freeman and D.L. Goroff (Eds.). Chicago: University of Chicago Press.2009;99–127. Reference Source Forrestal SG D’Angelo AV Vogel LK : Considerations for and Lessons Learned from Online, Synchronous Focus Groups. Survey Practice. 2015;8(3):1–8. 10.29115/SP-2015-0015 Gibbs KD McGready J Griffin K : Career Development among American Biomedical Postdocs. CBE Life Sci Educ. 2015;14(4):ar44. 26582238 10.1187/cbe.15-03-0075 4710405 Gunawardena CN Zittle FJ : Social presence as a predictor of satisfaction within a computer‐mediated conferencing environment. Am J Distance Educ. 1997;11(3):8–26. 10.1080/08923649709526970 Hokanson SC Grannan S Greenler R : A Study of Synchronous, Online Professional Development Workshops for Graduate Students and Postdocs Reveals the Value of Reflection and Community Building. Innovative Higher Education. 2019;44:385–398. 10.1007/s10755-019-9470-6 Kankanhalli A Tan BCY Kwok-Kee W : Contributing knowledge to electronic knowledge repositories: an empirical investigation. MIS Quarterly. 2005;29(1):113–143. 10.2307/25148670 Kite J Phongsavan P : Insights for conducting real-time focus groups online using a web conferencing service [version 1; peer review: 2 approved with reservations]. F1000Res. 2017;9(6):122. 28781752 10.12688/f1000research.10427.1 5527981 Krueger RA : Focus Groups: A Practical Guide for Applied Research.Thousand Oaks, CA: SAGE Publications.1994. Lobe B : Best Practices for Synchronous Online Focus Groups. In: R. Barbour and D. Morgan (Eds.) A New Era in Focus Group Research. London: Palgrave Macmillan.2017;227–250. 10.1057/978-1-137-58614-8_11 Matthews KL Baird M Duchesne G : Using Online Meeting Software to Facilitate Geographically Dispersed Focus Groups for Health Workforce Research. Quantitative Health Research. 2018;28(10):1621–1628. 10.1177/1049732318782167 NPA Core Competencies. (n.d.). Reference Source Sweet C : Designing and conducting virtual focus groups. Quantitative Market Research. 2001;4(3):130–135. 10.1108/13522750110393035 Tates K Zwaanswijk M Otten R : Online focus groups as a tool to collect data in hard-to-include populations: examples from paediatric oncology. BMC Med Res Methodol. 2009;9(15). 19257883 10.1186/1471-2288-9-15 2653071 Underhill C Olmsted MG : An Experimental Comparison of Computer-Mediated and Face-to-Face Focus Groups. Soc Sci Comput Rev. 2003;21(4):506–512. 10.1177/0894439303256541 Young C Chesniak OM Drane D : Improving the design of an online course with virtual focus group feedback. Harvard Dataverse, V1.2020. http://www.doi.org/10.7910/DVN/ZBWSHS 10.5256/f1000research.29018.r115651 Reviewer response for version 1 Quispe-Juli Cender Udai 1 Referee https://orcid.org/0000-0003-0633-8339 Universidad Peruana Cayetano Heredia, Av. Honorio Delgado 430, San Martín de Porres, 15102, Peru

Competing interests: No competing interests were disclosed.

 18 1 2022 Copyright: © 2022 Quispe-Juli CU 2022 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

The bibliography reviewed is not up-to-date and gives the impression that a systematic review of previous research was not done. The argument developed in the introduction is not enough to support the idea that it is a new method, the investigation could be better justified.

Within the limitations, it must be recognized that the conduct of the participants may vary in digital environments, conditioning their participation and objectivity. It should also be recognized that focus group feedback is not the only method that could be used to improve an online course. Other indicators such as eye-tracking responses, cue points and hotspots within the platform where the course is located could be very helpful.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Partly

Reviewer Expertise:

Online Courses; Medical Education, Digital Health

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

 10.5256/f1000research.29018.r92657 Reviewer response for version 1 Tudevdagva Uranchimeg 1 2 Referee https://orcid.org/0000-0001-9239-0760 Power Engineering School, Mongolian University of Science and Technology, Ulaanbaatar, Mongolia Faculty of Computer Science, Chemnitz University of Technology, Str. der Nationen, Chemnitz, Germany

Competing interests: No competing interests were disclosed.

 9 9 2021 Copyright: © 2021 Tudevdagva U 2021 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

This paper aims to figure out how positive constructive feedback from participants can influence and improve the offering of professional development programs.

Data collected from 13 focus groups which are reasonable for analyses and conclusions. In total 53 postdoc researchers took part in the offered course. For data collection, authors are using an online survey with support of a promotion method: Amazon card for free, which I found interesting, but not really usual for scientific data collection.

Technical writing of the paper is excellent. Transparency of used methods for evaluation and activities with focus groups is well done. It is easy to read and follow sections to understand the content of work.

Based on collected data authors did analyze professional development programs and did some change in program with the aim to improve it. In summary, virtual groups for studying professional development programs have many advantages for participants.

Most positive part of the paper is several ways to collect fundamental data from target groups. And excellent comments on data analysis. But some data is not really available for free, if the reader wants to follow all details.

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

e-learning, evaluation model, evaluation of e-learning

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.