An international survey and modified Delphi process revealed editors’ perceptions, training needs, and ratings of competency-related statements for the development of core competencies for scientific editors of biomedical journals

Participants

We approached current or former scientific editors of journals, defined as editors who make decisions on the content and policies of a journal – including editors-in-chief and associate/academic editors. Recruitment advertisements were sent to editorial organizations and groups having a large scientific editor membership from around the world. These organizations forwarded the advertisement about the survey of editors to their members through a distribution list email or an announcement on a listserv or message board. The organizations are listed in Box 1:

Phase 1: Survey of Editor Perceptions and Training Needs

We developed an online-based assessment of editors’ perceptions and training needs that was anonymous and self-administered (Supplementary File S1). The questionnaire was developed based on data collected in our previous scoping review4, as well as with input from our research team, comprising scientific editors, representatives from publishing houses, educational experts, and specialists in publication science. Questions were designed to broadly cover major areas associated with the scientific editor role, including editors’ knowledge, expertise, skills, and experience. The questionnaire was not validated; however, it was piloted among five experienced scientific editors of biomedical journals and subsequently revised based on their feedback. The revised questionnaire was uploaded to SurveyMonkey (www.surveymonkey.com) and the survey URL was sent to the editorial organizations for distribution.

The survey contained 19 demographic questions relating to participants’ age, sex, education level, job title, editorial experience, the journal they edit, and their editorial training experience. The perceptions of respondents were also examined in four areas: Participants indicated in the first two instances how important they thought a series of competency-related statements was to the scientific editor role, and, in the latter two instances, how much they thought they possessed these same competencies. Response options on a 1–7 Likert scale were provided (as indicated):

Finally, participants were asked to create a ranked list of their top 10 training needs (from #1 being the most important to #10 being least important). Participants were also asked if they would be willing to participate in a Delphi process to rate the importance of a much larger and more detailed list of competency-related statements for scientific editors of biomedical journals. If so, they were asked to provide their email addresses and were included in the list of potential participants for the modified Delphi process.

Phase 2: Modified Delphi Process

We carried out a three-round modified Delphi process in order to rate and refine the list of potential competencies derived in a previous scoping review we conducted⁴. A Delphi process typically involves experts and takes place in the form of iterations or “rounds” (normally 2 to 4) in which data is collected anonymously, often on-line, and then fed back to the group in an aggregated and de-identified way, along with individual participants’ comments¹³. In the current project, we modified the Delphi in several ways: First, we solicited the involvement of any scientific editors of biomedical journals, not only experts, as is normally the case with a Delphi process. Next, we did not require all participants to be involved in all rounds of the modified Delphi. In addition, the number of items included in the modified Delphi was much greater than a Delphi would generally include. Finally, for the sake of efficiency (due to the large number of items), in the third round we did not ask participants to re-rate items that had reached consensus for inclusion or exclusion in Round 2.

Interested Phase 1 participants were invited to participate in the online modified Delphi, which was also administered via SurveyMonkey. For each Delphi round, an invitation was sent to the entire list of potential participants, regardless of whether they responded in the previous round.

Round 1. Participants were asked to rate each competency-related statement on a 5-point Likert scale, from 1 (Not at all Important) to 5 (Absolutely Essential). At the end of each section, an open text box was provided for participants to include comments relating to items in that section if desired. Participants were also asked to name any potential competencies that were not included in the Delphi. Participants were reminded that all competency-related statements exclusively related to the position of Editor-in-Chief had been intentionally removed from the Delphi, as was the case for the scoping review on which the Delphi was built.

Round 2. Phase 1 participants were invited to participate in Round 2 of the Delphi, regardless of whether they completed Round 1. Along with the email invitation, they were provided the mean score for each of the items from Round 1, the participant’s own score for each of the items, if relevant, and collated (de-identified) comments from the text boxes. Participants were then asked to re-rate and provide their rationale for disagreement only for those items for which they disagreed with the mean score from Round 1. Any new competency-related statements arising from the final question in Round 1 were included for rating in the Round 2 survey. Participants were asked to rate these new items in the same way that items had been rated in Round 1, that is, on a 5-point Likert scale from 1 (Not at all Important) to 5 (Absolutely Essential).

Round 3. Phase 1 participants were invited by email to participate in Round 3 of the Delphi, regardless of whether they completed Round 1 and/or Round 2. Attached to the e-mail was a document listing the Round 1 and Round 2 average scores for all items and participants’ comments from previous rounds. Items that did not reach consensus for inclusion or exclusion in Round 2 were highlighted in yellow. Participants were asked to re-rate these highlighted items in Round 3 using a 3-point Likert scale from 1 (Less Important) to 3 (Essential). The shift to a 3-point Likert scale was aimed at simplifying the process for participants by limiting the number of options to the manner in which the ratings of each competency statement would be analysed (i.e., consensus on a ‘1’ would mean the item was excluded and consensus on a 3 would mean the item was included).

Analysis

To establish inclusion and exclusion criteria for both the survey of editors and the modified Delphi, we pre-specified the consensus level at 80% of respondents. This decision was based on the use of an 80% cut-off rate in previous Delphi studies in healthcare¹⁴ and education¹⁵.

The data on the 76 perception-related survey questions were summarized by calculating mean scores for each of the questions. These scores were then classified as reaching consensus for inclusion (≥6 on a 7-point Likert scale) or exclusion (≤3 on a 7-point Likert scale), for use in a future consensus meeting. The ranked top 10 lists of training needs were collated by one author (JG) by regrouping similar statements and these groupings were then verified by another author (KDC).

In Round 1 of the Delphi, means were calculated for each of the items and consensus for inclusion (≥4 on a 5-point Likert scale) or exclusion (≤2 on a 5-point Likert scale) was determined. In Round 2, means were calculated for each of the items from Round 1 and consensus for inclusion and exclusion was updated. For any new items suggested by participants in Round 1, mean scores were calculated and consensus for inclusion and exclusion was determined in Round 2. In Round 3, means were calculated for items that had not reached consensus for inclusion or exclusion in Round 2. Final consensus for inclusion was set at 80% of participants selecting a ‘3’ on a 3-point Likert scale, while exclusion was set at 80% selecting a ‘1’ on the scale. A final list of included and excluded items was created for use in a future consensus meeting. Due to the large volume of included items in the final list, a post-hoc decision was made to create a shorter list of “highly rated” items that reached 90% consensus for inclusion.

Ethical considerations

All participants provided consent through an online form preceding the survey of editors and Delphi. In order to stimulate participation and thank participants, an iPad Mini was awarded as a draw prize after the survey of editors and after each round of the Delphi. Participants who wished to be entered into the draw were asked to provide their email address. These email addresses were used for the sole purpose of the draw and removed from the data prior to analysis. All data from this research was pooled for presentation in the results section.

Editors’ Perceptions and Ranking of their Training Needs

Demographic data. A total of 148 editors from around the world accessed and completed the survey of their perceptions (Table 1 & Supplementary File S2). Respondents were mainly those replying to the notices for participation posted by the World Association of Medical Editors (WAME) and Cochrane. Nearly 2/3 of respondents were male. Close to 2/3 had more than seven years of experience as an editor. The majority of editors indicated that their journal had provided formal or informal training related to their job and that they had sought both formal and informal training beyond what was provided by their employer. The large majority indicated having formal or informal training in research methods and statistics.

Editor Perceptions. The degree to which participants perceive that expertise-related items are important to their job as editor. Of the 18 questions related to participants’ perceptions of the importance of particular expertise-related items to their job as editor, two items reached consensus for inclusion (Table 2). The highest consensus was for ‘expertise in research methods’ (85.4%), followed by ‘expertise in dealing with publication ethics, including conflicts of interest of authors, reviewers, and editors and the editorial board’ (80.4%). None of the items reached consensus for exclusion.

The degree to which participants perceive that particular skills and experience are important to their job as editor. Of the 20 questions related to perceptions of the importance of particular skills and experience related to their job as editor, four items reached consensus for inclusion (Table 2). The highest consensus was for ‘behaving with integrity/professionalism’ (94.4%), followed by ‘using good judgment in decision-making’ (93.7%), ‘language/writing skills’ (90.1%), and ‘author and peer reviewer correspondence; how to evaluate peer reviews, draft a revision letter and evaluate an author response letter and revision’ (86.6%). No items reached consensus for exclusion.

The degree to which participants perceive they possess particular expertise-related items related to their job as editor. Of the 18 questions related to participants’ perceptions of how much they possess particular expertise related to their job as editor, no items reached consensus for inclusion (Table 2). The highest rated item was ‘expertise in research methods’ (72.1%), followed by ‘expertise in the publication process (decision-making aspects) for research papers, commentary, and correspondence’ (70.6%), ‘expertise in the subject areas in which your journal publishes’ (69.9%), and ‘expertise in dealing with authorship issues’ (60.7%). No items reached consensus for exclusion.

The degree to which participants perceive they perform particular skills and possess particular experience related to their job as editor. Of the 20 questions related to participants’ perceptions of how much they thought they performed particular skills and possessed particular experience related to their job as editor, three items reached consensus for inclusion (Table 2). The highest consensus was for ‘behaving with integrity/professionalism’ (90.2%), followed by ‘using good judgment in decision-making’ (87.5%), and ‘language/writing skills’ (81.5%). No items reached consensus for exclusion.

Ranked training needs. Training needs from participants’ ranked top ten lists were categorized into 109 unique items (Dataset 1). Of the 114 respondents to this question, the top priority listed was training in statistics; mentioned by 36.8% of respondents, with a median ranking of 2 (IQR=2). The second ranked need was for training in research methods; mentioned by 21.9% of respondents, with a median ranking of 2 (IQR=1.5). The third highest training need was in publication ethics; mentioned by 20.2% of respondents, with a median ranking of 3 (IQR=2). The fourth highest need was in recruiting and dealing with peer reviewers; mentioned by 17.5% of respondents, with a median ranking of 3 (IQR=4). The fifth highest training need was in indexing of journals; mentioned by 15.8% of respondents, with a median ranking of 2 (IQR=1).

Modified Delphi

We compiled a list of 202 unique competency-related statements identified in our previous research⁹ and 12 additional statements identified by participants in the survey of editors in Phase 1 (Supplementary File S3). These 214 items were categorized into seven areas:

Round 1. Eighty-three people participated, all of whom had completed the survey of editors. Of the 214 items listed, 88 items reached consensus for inclusion (≥4 on a 5-point Likert scale). Only one item – ‘act with integrity and accountability’ - was rated as 5 out of 5 by more than 80% of respondents. The items with the highest average score were ‘act with integrity and accountability’ (4.84), ‘identify and address allegations of fraud or plagiarism’ (4.70), ‘act on concerns about plagiarism, data fabrication, or an authorship issue and follow up with authors and then institutions’ (4.65), and ‘request full disclosure of potential conflicts of interest by the authors’ (4.65). No items reached consensus for exclusion (≤2 on a 5-point Likert scale) (see Supplementary File S3 for participants’ comments from all 3 rounds of the Delphi).

Round 2. Eighty-three people participated, 80 of whom had completed the survey of editors and 68 of whom participated in Round 1. Of the 214 items listed, 99 items reached consensus for inclusion (≥4 on a 5-point Likert scale). Sixteen items were added for assessment based on the suggestions made by Round 1 participants, of which 4 reached consensus for inclusion, bringing the total of included items to 103. The items with the highest average score in Round 2 were ‘act with integrity and accountability’ (4.82), ‘identify and address allegations of fraud or plagiarism’ (4.68), ‘demonstrate the ability to assess the quality of papers’ (4.64), ‘Demonstrate accountability to authors and ensure they are treated with fairness, courtesy, and objectivity’ (4.63), and ‘request full disclosure of potential conflicts of interest by the authors’ (4.61). Again, no items achieved consensus for exclusion (≤2 on a 5-point Likert scale).

Round 3. Round 3 included 73 participants, 72 of whom participated in the survey of editors, 58 of whom participated in both previous rounds, and 10 of whom participated in only one of the two previous rounds. The 103 items that reached consensus in Round 2 were not re-rated in Round 3. Additionally, 5 items with >80% consensus but a rating of between 3.95 and 3.99, as well as one item rated 4.0 but with only 78% consensus were inadvertently left out of the Round 3 Delphi and were therefore added (with a note) to the final list of included competency-related statements. This, therefore, left a total of 121 items to be rated in Round 3. None of these items reached consensus in Round 3, leaving a total of 109 included items at the end of Round 3. Similar to Rounds 1 and 2, no items achieved consensus for exclusion in this round.

Due to the large volume of included items after 3 rounds of the Delphi, the post-hoc decision was made (JG, KDC, LS, DM) to further narrow down the list to a more manageable size. This was done by identifying items that achieved 90% consensus for inclusion in Rounds 2 or 3. This produced 23 “highly rated” items, (Table 3), leaving 86 “included” items with 80% consensus (Dataset 2). With no items having reached consensus for exclusion, the Delphi exercise was completed with 121 items that did not reach consensus for inclusion or exclusion.

Limitations

There were a number of limitations in this research. The limited number of respondents for both the survey of editors’ perceptions and the modified Delphi, the fact that the study was conducted in English, and the fact that the survey was completed primarily by medical journal editors may limit the generalizability of the findings to the wider pool of scientific editors around the world. Additionally, for the modified Delphi, we chose to only invite respondents from the survey of editors (however we did not restrict participation to only this group). This decision was made since we believed we had used all of the most pertinent communication channels to recruit participants for the survey of editors, therefore (given our biomedical focus) we were unlikely to gain many more respondents by putting out a further call for participation in the modified Delphi. Moreover, although efforts were made to regroup similar items from participants’ top 10 lists of training needs, this subjective process may have failed to regroup some items that could be seen as similar while combining other items that could be judged to be different from one another. A further limitation is in regards to the interpretation of competency-related statements to be rated by participants in the Delphi. While efforts were made to preserve the original wording of competency-related statements in the scoping review, some of the participants’ comments relayed difficulties in understanding some items and misinterpreting others. This may have led to the mis-rating of a few items by some participants. This limitation should be offset at least to some extent by our use of median scores rather than means.