We obtained research ethics board approval for the proposed study from the Ottawa Health Science Network Research Ethics Board (REB ID# 20220132) on April 06, 2022. The final protocol was registered on April 28, 2022, using the Open Science Framework (OSF) ⁹ at https://osf.io/fsmzc. Anonymized voting data and participant responses from rounds 1 and 2, along with the round 3 thematic analysis, were shared publicly using OSF at https://osf.io/8v63x/. We followed the ACcurate COnsensus Reporting Document (ACCORD) checklist in reporting our findings. ¹⁰

We conducted a modified three-round Delphi study, similar to the approach used in a previously published study. ¹¹ Diamond et al. have proposed key methodologic criteria to report in publications of Delphi studies, ¹² which were used to inform the present study. The first two rounds of the Delphi involved completing an online independent survey using SurveyLet. ¹³ The third round consisted of a virtual consensus group meeting with participants from earlier rounds.

Participants were recruited from a diverse group of academic publishing houses. Purposeful sampling based on this list of potential participants and snowball sampling were used to increase representation and target outreach/invitations. To be eligible, participants needed to be fluent in English. Representatives were invited from publishing houses to participate in our Delphi study using a study advertisement and standardized recruitment script (see https://osf.io/6r3np). Members of the public, patients, or carers were not invited for recruitment in any of the three rounds. Participants were not provided financial compensation for taking part in the study, but those who agreed to participate in the third round –that is, the virtual consensus group meeting– were invited to co-author the present research report.

Our advertisement and recruitment script (see https://osf.io/6r3np) were disseminated (IJA, AM, MW) to the publisher community via two large publishing societies: the Open Access Scholarly Publishing Association (OASPA) ¹⁴ and the International Association of Scientific, Technical, and Medical Publishers (STM), ¹⁵ as well as the World Association of Medical Editors (WAME) ( https://www.wame.org/). OASPA provides an international community for open-access publishing and seeks to advance open-access publishing solutions. STM is also global in reach, with more than 140 members, which publish nearly 66% of all journal articles and work to advance trusted research worldwide. WAME is an international, nonprofit voluntary association of editors of peer-reviewed medical journals that seeks to foster international cooperation among and the education of medical journal editors. OASPA, STM, and WAME assisted with recruitment by targeting and sharing the invitation via email with their members. In addition, a member of the research team (JYN) extracted all journals catalogued in Directory of Open Access Journals (DOAJ) and used the RAND function ¹⁶ on Microsoft Excel to select a random sample of 750 journals. Publishers of these selected journals, which were manually checked to ensure they met the eligibility criteria (RF, MMessih, AP), were also sent our advertisement and recruitment script for participation in the study. Individuals interested in participating were redirected from the invitation to an online survey where they were provided with more information about the study, including participant inclusion criteria, and asked to complete an online informed consent form prior to proceeding.

Round 1

Participants were asked to complete an online survey (see https://osf.io/6r3np), which asked about their needs and preferences for a JTT. This survey was purpose-built for the study and included 30 items that addressed the following: (1) participant demographic characteristics (e.g., employment information, years working in the publishing industry, and areas of expertise); (2) JTT user features (i.e., how the JTT user interface should look, how data automation can facilitate the metrics the JTT reports, and how to disseminate a completed JTT to the community and track its uptake); and (3) JTT metrics (i.e., metrics that users will access about each individual journal on the JTT to make informed decisions regarding the use of that journal for clinical or research purposes). These survey items were informed and chosen by a team of experts in journalology (JYN, KDC, DM). Survey items regarding JTT metrics were posed in a question format, with responses provided on a 9-point Likert scale ¹⁷ with endpoints labeled ‘strongly agree’ and ‘strongly disagree.’ Participants were provided with a textbox to provide any comments on each item. Respondents also had the opportunity to suggest additional items for the JTT for participants to consider in round 2 of the survey. This survey was pilot tested by a group of researchers who were not part of the study prior to administration.

Participants were sent the survey on May 23, 2022, followed by two reminder emails, each spaced 1 week apart. Responses to the round 1 Delphi survey were collated, with all information presented in aggregate. For items assessing publishers’ needs and preferences for a JTT, we defined consensus on an item as 80% agreement to include or exclude an item. This cutoff was selected based on findings from a systematic review of Delphi studies. ¹² In our design, all items with 80% of responses in the top third (7-9) or bottom third (1-3) of our 9-point scale were considered to have reached consensus for inclusion or exclusion. Once items met consensus, they were not included in subsequent rounds. Furthermore, open-ended survey responses from round 1 were analyzed to provide feedback and incorporate any new suggestions for round 2 participants.

Round 2

Participants who completed round 1 of the survey were emailed an invitation and link to the complete round 2 of the Delphi survey. Members of the research team (JYN, HL, DM, KDC) created round 2 based on the responses obtained in round 1. It contained all survey items in round 1 that did not reach consensus, as well as any new items or clarifications required based on comments participants made in round 1. Comments provided by individual participants to explain responses to items from the round 1 survey were presented back to all participants for consideration as they re-voted on each item. Participants were also provided the opportunity to suggest new JTT items for respondents in Round 3 to vote on.

Participants were provided with the percentage of scores on each third of the scale for each item not in consensus from round 1, along with a collated and de-identified version of any comments made on each item. Participants were sent the survey on March 6, 2023, followed by two reminder emails, each spaced 1 week apart. We processed responses resulting from round 2 and defined consensus in the same way as in round 1.

Round 3

We invited all respondents from round 1 to take part in an online consensus group meeting. If unavailable, an option was provided to participants to send another representative from their publishing organization to take part in round 3 in their place. Participants met virtually on Zoom on December 6, 2023, to establish consensus on any remaining JTT metric items from rounds 1 and 2 of the Delphi. The virtual consensus meeting was moderated by a steering committee with an expertise in journalology (JYN, JA, GLB, QC, WJNM, JW, KDC, DM) to ensure that participants could express their views on any outstanding items while ensuring that the focus of the meeting and goals of final consensus on items were met.

We provided all participants with a summary of the results obtained in round 2 approximately one week prior to the meeting. This summary listed all the JTT metrics that did and did not reach consensus. Items that did not reach consensus were presented with the number and percentage of round 2 participants that identified that specific JTT metric as ‘unimportant’ (1-3 points), ‘neutral’ (4-6 points), and ‘important’ (7-9 points) for inclusion on the tool on a 9-point scale. Participants voted through the anonymous polling feature on the Zoom platform, where they were provided three options to select from during the voting of each item: (1) support for inclusion on the JTT, (2) support for exclusion on the JTT, or (3) abstain from voting. The group’s voting results were presented to participants immediately following receipt of all responses. Members of the steering committee did not vote during round 3, and a separate research member (HL) took field notes during the session.

We report the number of participants that completed each round of the Delphi and their basic demographic characteristics using frequencies and percentages. Data for each round of the Delphi was analyzed successively and presented independently. Quantitative items are presented using frequencies and percentages. For each survey item assessing needs and preferences for the JTT, we indicated whether consensus was reached and, if so, at what round.

Open-question survey responses from rounds 1 and 2 were analyzed using a thematic content analysis. ¹⁸ Two members of the research team (JYN, HL) aggregated and independently coded responses. Themes were confirmed through multiple iterative meetings between research members (JYN, HL), with all items reaching consensus through discussion. Similarly, for the round 3 online consensus group, an automated Zoom transcript, which was reviewed for accuracy (MMasood), was used to conduct a thematic content analysis. ¹⁸ One member of the research team (MMasood) coded, combined, and uploaded online consensus group notes into Microsoft Excel and established themes based on the codes. Two members of the research team (JYN, MMasood) subsequently discussed the generated themes to confirm their accuracy and relevance.

Challenges with gaining a sufficient participant sample size led to two deviations from the protocol. First, within round 1, additional participants were recruited from a random sample of 750 journals extracted from DOAJ. Similarly, to increase the sample size, participants in round 3 were recruited from all respondents from round 1, rather than from a random, stratified sample of respondents (see Table 1 for participant demographic characteristics).

Time and resource constraints also resulted in additional changes to the round 3 online consensus group. First, round 3 was conducted within a single day as opposed to two half-days to accommodate participant availability. Second, JTT user features (i.e., preferences for a website or API interface, fee-for-usage, the use of automation, and registration for access on the tool; see Table 2) and JTT metric items (see Table 3) that did not reach consensus in round 2, along with novel JTT metrics suggested by round 2 participants, were originally intended to be discussed and voted in round 3. Instead, only JTT metrics items that did not reach consensus in round 2 (see Table 3 for items) were discussed and voted on within round 3.

Participants

A total of 86 participants completed the round 1 survey (see Table 1 for participant demographic characteristics). Respondents voted on JTT user feature ( Table 2) and JTT metric item ( Table 3) preferences and answered open-ended survey questions regarding the JTT. Please view the following link for complete round 1 responses: https://osf.io/92zh8.

Participants had less than 1 year (n = 1, 1.2%), 1 to 3 years (n = 10, 11.6%), 4 to 10 years (n = 34, 39.5%), 11 to 20 years (n = 21, 24.4%), and over 20 years (n = 20, 23.3%) of experience working in the publishing industry, with most being involved as an editor and/or with completing editorial tasks (n = 47, 55.3%). Editorial processes/peer review (n = 72, 81.8%), the publication process (n = 68, 77.3%), and open access (n = 64, 72.7%) were the top three areas of expertise in publishing that respondents identified with. Participants worked with over 70 international publishers/publishing organizations at the time of the study (see https://osf.io/92zh8 for the complete, unedited list of institutions).

JTT user features

Participants voted on four JTT user feature items in round 1 ( Table 2). None of the JTT user feature items reached the 80% consensus threshold after both rounds of voting. Participants provided the most support for: hosting the JTT on a website over an application programming interface (API)/browser plugin interface (n = 51, 58.0%); not paying to use the tool (n = 52, 61.2%); and fully automating the JTT (n = 41, 48.2%). The use of registration to access the JTT platform was contentious, with roughly equal proportions of participants being in support (n = 33, 38.8%), neutral (n = 22, 25.9%), and opposed to its use (n = 30, 35.3%). Open-survey responses regarding the JTT user format suggested that participants were predominantly concerned with ensuring that the JTT will be user-friendly and accessible.

JTT metrics

Of the 17 JTT metric items that participants voted on within round 1, five reached consensus ( Table 3). The following items were agreed upon as ‘important’ to include in the JTT on a 9-point scale: a metric that reports whether (1) the journal describes its approach to publication ethics or not; (2) the journal editors are listed or not; (3) the journal uses fake digital object identifiers (DOIs) or not; (4) the journal reports misleading scholarly metrics or not; and (5) there is verifiable contact information or not.

Participants suggested seven novel JTT metrics to vote on, including metrics that report: (1) whether the journal has any article submission or processing fees and information about the journal’s funding model; (2) the journal’s citation metrics; (3) the journal’s policies surrounding equity, diversity, and inclusion; (4) the journal’s policies on retractions/corrections; (5) the journal publisher’s size and how the JTT’s evaluation may negatively affect smaller publishers just by virtue of their size; (6) the journal’s policies regarding reporting ethics, funding, and conflicts of interest; and (7) the journal’s peer review model and metrics. All seven JTT metrics were added in for participants to vote for in round 2. Within open-ended survey responses, participants also raised concerns about how ‘transparency’ is defined by the tool and how metric scoring may negatively affect smaller publishers/journals.

Participants

Forty-three (50% of round 1) participants completed the round 2 survey. Respondents voted on JTT metric and user feature items that did not reach consensus in round 1, as well as any additional items suggested by participants in round 1 (see Table 3 notes for newly added items). Please view the following link for complete round 2 responses: https://osf.io/nh6cv.

Most respondents had more than 4 years of experience working in the publishing industry (n = 39, 90.7%). Similar to round 1, most participants were involved as an editor and/or completed editorial tasks (n = 26, 60.5%), and the top three areas of expertise within publishing participants identified were editorial processes/peer review (n = 38, 88.4%), the publication process (n = 35, 81.4%), and open access (n = 34, 79.1%). At the time of the study, participants in this round worked for 30 international publishers/publishing organizations (see https://osf.io/nh6cv for the complete, unedited list of institutions).

JTT user features

In round 2, participants voted on four JTT user features ( Table 2). While none of the items reached the 80% consensus threshold, respondents displayed similar voting preferences to round 1. Most participants supported hosting the JTT on a website over an API/browser plugin interface (n = 33, 76.7%), not paying to use the tool (n = 32, 76.2%), and fully automating the JTT (n = 24, 55.8%). Furthermore, similar to round 1, the use of registration to access the JTT platform was contentious, with roughly equal proportions of participants being in support (n = 17, 39.5%), neutral (n = 10, 23.3%), and opposed to (n = 16, 37.2%) its use. Open-survey responses regarding the JTT user format suggested that participants were predominantly concerned with ensuring that the JTT would be user-friendly, easy to understand, and concise.

JTT metrics

In round 2, participants voted on their preferences regarding 19 JTT metric items (12 metric items that did not reach consensus in round 1 and seven novel items added based on round 1 responses). Of these 19 items, four met the 80% consensus threshold. Participants found the inclusion of the following items to be ‘important’ on a 9-point scale: a metric reporting (1) the journal’s policies regarding reporting ethics, funding, and conflicts of interest; (2) the journal’s policies on retractions/corrections; (3) the journal’s peer review model and metrics; and (4) whether the journal uses DOIs or not ( Table 3).

In open-ended survey responses, participants suggested eight novel JTT metrics to include for voting (see https://osf.io/nh6cv for the complete list). These new items were not voted on within round 3 due to time and resource constraints. Participants also suggested that the JTT metric scoring strategy should be openly shared to provide transparency and allow users to understand how metrics are evaluated in the tool’s assessment process. Furthermore, similar to round 1, participants raised concerns with how ‘transparency’ is defined. They added that the JTT should adopt a widely recognized definition of ‘transparency’ to ensure a clear understanding of the concept and its application within the tool.

Participants

In round 3, 21 participants discussed and voted on JTT metric items that did not reach consensus after round 2 ( Table 3) within an online consensus group meeting. Specifically, round 3 participants included EC, IC, PEC, JG, RBH, BSK, AM, EP, AP, JDBS, PRS, PS, JW, MW, and seven other publishers. Demographic characteristics were not collected for round 3 participants.

JTT user features

In round 3, JTT user feature items that did not meet consensus in round 2 were not discussed nor voted on due to time and resource constraints.

JTT metrics

During round 3, participants discussed and voted on the inclusion of JTT metrics within an online group meeting. There were 15 items that had not reached consensus in round 2. After discussing each item, two individual items were modified to be expanded into two items each, resulting in four items in total, and two novel items were added for voting (see notes on modifications made in Table 3). With these revisions, participants ultimately voted on 19 JTT metric items in round 3.

Of the 19 JTT metric items voted on in round 3, six met the 80% consensus threshold. Participants supported the inclusion of four items: a metric (1) reporting whether the journal is a member of COPE or not; (2) reporting whether the journal has any article submission or processing fees; (3) reporting whether the journal uses ORCIDs or not; and (4) reporting whether the journal is listed in the DOAJ or not. Participants supported the exclusion of two items: (1) an option for the JTT to collect/share journal incidents; and (2) a metric reporting whether the written content presented on the website is clear or not. In total, 15 out of the 30 JTT metrics reached consensus after all three rounds of voting (see Table 4 for the complete list of consensus items).

A thematic content analysis ¹⁸ of the online group meeting transcripts resulted in two themes (see https://osf.io/978vs for the thematic analysis and Table 5 for a summary of the themes). First, several factors influenced the support individuals had for JTT metrics. This included how relevant the metric was to transparency and open practices; if the information gathered by the metric was already captured within another one; and how challenging it would be to gain membership for that metric when applicable (e.g., to be indexed on DOAJ). Concerns were raised within multiple rounds regarding how certain metrics could be exclusionary towards journals (e.g., small publishers) based on selection criteria that are not related to adherence to transparency and open practices (e.g., language, costs, geographic location). For items that reached consensus for exclusion in round 3, the primary issue raised by participants was the inability to objectively define and evaluate the relevant metrics using a clear criterion on the JTT. Within the second theme, participants provided suggestions to increase user clarity. Respondents stated that, with each JTT metric, users should be provided clear descriptions of what the metric is, how it is measured on the JTT, and potential biases that may be associated with the metric. Further, participants suggested that the naming of the JTT and its components should be clear and accurate. Some respondents stated that the tool should be renamed (i.e., the tool should not be called the ‘Journal Transparency Tool’) to increase clarity, but alternative names were not proposed.