Dear reviewer,

We would like to thank you for taking the time to review and evaluate our manuscript. We have addressed all your comments and have revised the manuscript accordingly. An updated version of the manuscript highlighting the changes is attached. Please find below the responses to your comments.

Major comments:

1. Problematic Comparison: Hybrid vs. Gold OA JournalsThe current design compares Gold OA journals against Hybrid OA journals. However, hybrid journals by definition combine two publishing models: Gold OA and subscription-based. They are not a homogeneous category. Comparing them directly to full Gold OA journals is methodologically unsound, as it conflates two distinct business models within the hybrid group. As a result, the analysis is unable to isolate the effect of open access publishing at the journal level. I strongly recommend to the authors to redefine the study design. At a minimum, distinguish between:

Gold OA journals,

Response: The missing definitions were added to the definitions subsection of the methods section to add further clarification of these distinguished terms.

2.  Incomplete and Biased Treatment of APCs

The manuscript discusses article processing charges (APCs) only in the context of Gold OA journals, implicitly portraying them as costly. Yet:

Response: Claims and recommendations to publishers about costs were removed from the discussion section as this was also suggested by reviewer 1.

3. Lack of Transparency in Data and Methodology Several methodological details are missing or insufficiently explained:

Journal model transitions: The manuscript refers to journal age and model history but does not specify how or from which source the transition dates were determined.

Response: Please see below the respective response to each point:

Journal model transitions: In our dataset, we did not include journals that transitioned from hybrid to gold OA to clearly distinguish between journals that have remained hybrid and those that remained gold OA at the time of data collection and analysis. We only mentioned this in the discussion section to clarify the current trend of certain journals. To clarify this issue, we added to the methods section the fact that we excluded journals that transitioned from hybrid to gold OA.

4. Problematic and Unsupported Claims

The policy recommendations presented in the manuscript are not evidence-based and, in some cases, are contradictory or misleading. For example (not exhaustive list):

The authors suggest that “readers can still obtain the full text through institutional subscription, by paying for an individual article, or through contacting the corresponding author via email or researcher platforms like ResearchGate and Academia.edu.” This recommendation ignores key issues: access by payment contradicts the critique of APCs as a barrier, and informal sharing through ResearchGate or Academia.edu disregards copyright restrictions and the fact that legally shareable versions are often limited to preprints or accepted manuscripts (after embargo).

I suggest to remove or thoroughly revise the recommendations. They are normative, inconsistent with the results, and contradict much of the existing OA literature.

Response: Regarding the first point, we added a sentence following this one reiterating the difficulty of accessing articles not published as OA in hybrid journals, such as in the case of lack of response from authors when contacted to share their articles, as well as certain embargos imposed by publishers.

Regarding the second point, the sentence was removed.

Regarding the third point, this sentence was removed and replaced by the following sentence after mentioning that publishing model wasn’t associated with impact factor or CiteScore “However, lack of assessing the impact of articles published OA in hybrid journals may limit the generalizability of this interpretation.” Similarly, the following sentence was removed from the first paragraph of the discussion based on a suggestion from reviewer 1 “Therefore, such publishers should consider lowering fees to cover necessary editorial processing and offer waivers for unfunded authors provided they provide proof of lack of funds regardless of their country of origin (as many gold OA journals already offer waivers or discounts for authors from low-income countries).”

5. Misrepresentation of Prior Literature The manuscript claims to be the first to study the effect of publishing models at the journal level. This is demonstrably false:

Existing literature showed that Gold OA publishing shows mixed effects on journal impact, with positive results in Medicine and Life Sciences but lower impact in Social Sciences and Humanities. For example (see below the list of related papers I am aware of), Bautista-Puig et al. (2020) report a significant post‐conversion increase in relative citation impact (p < 0.05), and Chua et al. (2017) find that gold OA journals receive 1.3 times more citations (p = 0.039) compared with non–open access journals. Studies by Gumpen- berger et al. (2013) and Momeni et al. (2021) note upward trends in metrics such as Journal Impact Factor, SCImago Journal Rank, and other normalized citation measures after a shift to gold OA, even if precise effect sizes are not given.

In contrast, Davis (2011) finds no effect on citation counts (point estimate = 1.01; 95% CI, 0.95–1.07), and Björk and Solomon (2012) observe that initial differences in citation rates between open and subscription journals disappear when factors like discipline and age are controlled. Findings differ by field: journals in Medicine and Life Sciences typically achieve higher impact with gold OA, whereas Social Sciences and Humanities tend to score lower.

The novelty claim must be corrected, and these references properly integrated.

Moreover, the study ignores the role of Sci-Hub and shadow libraries, which provide access to paywalled articles and reduce the measurable citation advantage of OA (see Maddi et Sapinho 2023). To have more robust results, this effect should be neutralized.

Response: We reworded the first sentence of the discussion and removed the word “first.” We also removed it from the last paragraph. We included the studies suggested in the second paragraph of the discussion with a brief description of their findings as stated (The study by Chua et al was already cited and discussed in the first paragraph, but we added the observation stated in this comment to it). We also included a brief sentence on shadow libraries in the discussion section as suggested and we cited the reference mentioned.

Minor comments

1. Write “Sugimoto” instead of “Suigmoto”.

Response: Corrected.

2. The title is misleading: it implies a causal claim (“Does OA increase impact?”) while the design is more observational.

Response: The first part of the title was removed. It was reworded to “Association of hybrid versus gold open-access publishing models of medical journals with their impact” to avoid implying casualty.

3. The abstract misinterprets results (e.g., underreporting the significant association in Scopus quartiles).

Response: This part was corrected based on the above comment, as well as a comment from reviewer 1.

4. To sum up, the paper requires major restructuring: from redefining the comparison groups to improving the statistical models, correcting the literature claims, and removing unsupported recommendations. As it stands, the article is misleading and fails to meet the methodological rigor required for indexing in F1000Research.

Response: We appreciate the valuable comments made. We hope that the revised manuscript is an improved version of our work.

Respectfully;

The corresponding author

Dear reviewer,

We would like to thank you for taking the time to review and evaluate our manuscript. We have addressed all your comments and have revised the manuscript accordingly. An updated version of the manuscript highlighting the changes is attached. Please find below the responses to your comments.

1. The work is presented clearly and well-organized, facilitating reading, but the literature review is still limited. The authors do not explore recent references discussing the differences between hybrid and fully open access models, nor do they include debates on alternative metrics such as altmetrics or indicators of digital visibility. Incorporating more recent studies would strengthen the theoretical and contextual basis of the manuscript.

Response: Additional studies with brief explanation of their findings were added to the end of the second paragraph of the discussion, including a study that evaluated the impact on altmetric score. We also included that the evaluation on altmetrics is warranted in future studies in the last paragraph of the discussion before the conclusion.

2. The study design is relevant but presents weaknesses that compromise its technical soundness. The criteria for selecting the journals are not sufficiently clear, and there is no detail on which subject areas were included or excluded. Furthermore, potential confounding factors such as journal age, editorial policy, or internationalization, which could influence the indicators, were not considered. A more robust methodological clarification and a discussion of these biases are necessary.

Response: We believe that the “Journal selection” subsection of the methods section clearly lists the inclusion and exclusion criteria. We specifically mentioned that we included medical journals published in English and indexed in WOS and Scopus. We also mentioned that we excluded journals of basic sciences, ethics, healthcare systems management, administration, and policy. For further clarification, we added the exclusion criterion of the journals that transitioned from hybrid to gold OA.

With regards to the factors, we did include journal age as a factor in all the analyses. However, we listed “the lack of assessing of editorial policy and internationalization as potential confounding factor” as a limitation in the last paragraph of the discussion.

3. The current methodological description does not allow other researchers to replicate the study. Precise information about the data sources used, the date of collection, the inclusion and exclusion criteria, and how journals were classified as hybrid or gold OA is missing. Without these elements, the study loses transparency and reproducibility. Detailing these aspects is essential to ensure the credibility of the work.

Response: We added a new subsection to the methods section titled “Data collection”, where we mentioned the data collected, source of the data collected, and the dates the study was conducted. Regarding inclusion and exclusion criteria and how the journals were classified, this was already described under the “Journal selection” subsection of the methods.

4. About statistical analysis, the study presents only descriptive comparisons without applying significance tests to support the observed differences. This absence weakens the conclusions, which are more speculative than scientific. Applying appropriate tests, such as t-test, ANOVA, or multivariate regression, would allow the hypotheses to be validated or refuted with greater robustness. If the authors choose not to include statistical analyses, the conclusions need to be reformulated in more cautious terms.

Response: We employed Chi-square and Mann-Whitney tests for comparison purposes. However, for the assessment of factors and their association with different metrics, we employed multivariable generalized linear model and binary logistic regression. We consulted a statistician who improved our statistics in this revised version of the manuscript after a similar comment was made by reviewer 4. We updated the results accordingly as shown by the marked text.

5. Another critical point is the lack of raw data used in the analysis. Without these materials, it is impossible to verify the results' consistency or ensure the study's reproducibility. It is recommended that the authors provide a dataset with the collected indicators, specifying the source and year of each data point, preferably in an open-access repository. This would increase the transparency and scientific utility of the work.

Response: We indeed uploaded the dataset as a supplementary material. However, we apologize for not citing that in the main text. We have now added that clarification in the first sentence of the results.

6. The conclusions presented are broader than the results can support. Given the absence of statistical analysis and the methodological limitations, it is impossible to state categorically that gold OA journals have a higher impact than hybrid ones. The conclusions should be reformulated to reflect the preliminary and exploratory nature of the findings, acknowledging the study's limitations and avoiding excessive generalizations.

Response: We added the following sentence to the conclusion section to tone down our conclusion “However, lack of assessing the impact of articles published OA in hybrid journals may limit the generalizability of this interpretation.”

7. The article addresses a relevant and timely issue that can contribute to the debate on scientific publishing models but requires substantial revisions before it can be considered solid. Expanding the literature review, detailing the methodology, including statistical analyses, making the raw data available, and reformulating the conclusions are indispensable steps. If these points are adequately addressed, the work may significantly contribute to discussing the impact of open access models in medical journals.

Response: We appreciate all the comments and suggestions. We have addressed them all and hope that the revised version is an improved version of the manuscript.

Respectfully;

The corresponding author

Dear reviewer,

We would like to thank you for taking the time to review and evaluate our manuscript. We have addressed all your comments and have revised the manuscript accordingly. An updated version of the manuscript highlighting the changes is attached. Please find below the responses to your comments.

1. Some sentences are difficult to follow and there are still a few spelling and grammar errors in the manuscript.

Response: The manuscript underwent proofreading to correct the errors.

2. Please completely remove language suggestive of causality (e.g. effect, affect, impact), as the study design doesn’t allow for a causal interpretation of the associations found.

Response: The language was toned down, and these words were removed where appropriate to avoid casual interpretation.

3. Please remove from the text the redundancy of providing both 95% CI and P values by removing all p-values from the manuscript when also a 95% CI is provided.

Response: Done.

4. One sentence in the abstract seems to be wrong: However, gold OA model wasn’t significantly associated with Scopus Q1 ranking (OR, 0.49; 95% CI, 0.25-0.94; P=0.032), indicating that hybrid journals were more likely to have such ranking. In the Results section you say about this: …hybrid journals were associated with ranking in Q1 of Scopus as the odds of gold OA journals being ranked as Q1 was less than one (OR, 0.49; 95% CI 0.25–0.94; P = 0.032). These statements contradict each other. I believe that the second one is correct, as the 95% CI doesn’t include 1. Furthermore, the sentence in the Results section refers to Table 4, but the table doesn’t contain the point estimate and 95% CI cited.

Response: We reviewed the two sentences. The first one was the correct one. So, we edited the one in the results section and rephrased it. The result is listed in Table 5 and Table 4.

5. The final sentence of the conclusion (Gold OA journals are encouraged to reduce their fees to facilitate global research access) in the abstract is unrelated to the study results and should be removed.

Response: Removed as suggested.

6. Table 2-4 are not very clear. Please explain whether the factors listed in the table are adjusted for the influence of the others. In other words: are these univariate or multivariate estimates? Furthermore, you specify only the reference category of the reported OR for the first factor in each table but fail to explain what the reference category is for the other factors included.

Response: The analysis was multivariable generalized linear model. This was clarified in the statistical analysis section of the methods. For the other factors it is either a yes or no (e.g., affiliation with an organization or lack of affiliation, either indexed in EMBASE or not, etc.). In case of linear regression, there’s no reference. We believe this is clear to the reader. Therefore, no changes were made.

7. Not only the P values, but also the coefficients and standard errors are redundant in tables 1-4 and can be removed.

Response: After consulting a statistician and based on one of the comments by reviewer 3, we realized a misunderstanding in reporting the results of the analysis. Therefore, we removed the OR column since it was incorrect and kept the β coefficient column but replaced the standard error with 95% confidence intervals.

8. In Table 3 the OR (95% CI) for the factor journal published both clinical and basic sciences is missing.

Response: We apologize for missing that. However, this whole column of OR is now removed.

9. The Discussion section contains a number of recommendations to lower subscription rates and APCs or to give waivers to specific groups. While I sympathize with these pleas, there is no ground for them in the outcomes of the study, and they should be removed from the manuscript.

Response: We appreciate this comment. These parts were removed as suggested, but the calls for lower APCs in the literature was kept with appropriate citation of relevant references.

Respectfully;

The corresponding author

Dear Dr. Simard,

We would like to thank you for taking the time to review and evaluate our manuscript. We have addressed all your comments and have revised the manuscript accordingly. An updated version of the manuscript highlighting the changes is attached. Please find below the responses to your comments.

1. I think criticism of gold OA is warranted, but I think the approach used by the authors is misguided because it ignores the market forces that are at play as well as other sociological factors that hybrid journals benefit from (e.g. publisher and journal reputation, prestige, green OA, transformative agreements, etc.).

Response: Based on this comment and the comments made by reviewer 1, we have revised the manuscript to reflect the impact at the journal level, rather than the article level. Specifically, we commented in the discussion on the relation of the publishing model and their impact with journals’ reputation and prestige (lines 206-208 and 274-278).

2. I also believe that the introduction is too short or not detailed enough and fails to grasp some of the main issues related to the topic such as publication prestige, impact, preprints, more inclusive data sources (such as OpenAlex and Dimensions), the for-profit nature of the system, article processing charges, OA policies and community infrastructure, transformative agreements, etc. The article would benefit from a more in-depth literature review. OA is a fast-changing topic involving multiple actors which the introduction does not accurately reflect.

Response: We have expanded the introduction to include many of the suggested topics in the reviewer’s comment (lines 59-62, 67, and 74-83).

3. I also find the title/research question (Are gold journals worth it?) to be confusing as the article never really investigate more than research impact, which is only one of the facets (and I would argue it is the shallowest) of what makes OA/open science a more ethical choice. Furthermore, this link between research impact and publishing in OA journals is based on highly controversial and flawed journal-based metrics such as the impact factor and citescore, which are not a good predictor of individual article impact because the distribution of citations among journal articles is highly skewed (see Larivière and Sugimoto, 2018 [Ref 3]). Another problem with the design is that it doesn’t not take into consideration the fact that hybrid OA articles can also be OA (via green or APC payment), and generally benefit from greater historical prestige than gold journals. Multiple studies have shown that green OA papers tend to receive more citations on average than other types of OA for that reason (e.g. Piwowar et al. 2018).

Response: After the extensive revision that made to the manuscript, we have also revised the title to reflect the content and the consideration of the impact at the journal level rather than at the article level. We discussed the controversy regarding OA publishing based on the suggested reference (Sugimoto and Larivière) and another reference (Casadevall, et al) in the introduction (lines 78-83), as well as in the discussion (lines 272-278). Regarding the last point of considering that hybrid journals also have gold OA option, we have already acknowledged this as one of the limitations of our study (lines 284-287).

4. In my opinion, the discussion/conclusion sections raise some valid points, but they are often too superficial and could use more depth and comprehension of the complex market forces at play. For instance, calling on gold OA publishers to lower APCs, while noble, is not a realistic or a sustainable approach because it doesn’t consider the for-profit nature of the system and how these actors are taking advantage of OA to generate more profits (e.g. see Butler et al. 2023 [Ref 2]). There is also very little criticism of the hybrid model (and no APC average, when previous research has shown that they are higher than gold), when it is arguably even worse than gold journals: their APCs are higher, they charge both readers and authors, and are also parts of unsustainable transformative agreements. Why not also call them out on reducing their APC/subscription/transformative agreement prices?

Response: We completely agree with the argument presented by the reviewer. Therefore, and to address this comment, we added to the introduction the concern regarding the high APCs of hybrid journals and the high prices of transformative agreements (lines 59-62). Additionally, we also added this argument and the call to reduce these fees by hybrid journals in the same paragraph that called gold OA journals to reduce their APCs in the discussion (lines 265-271).

5. The abstract makes the false assumption that hybrid and gold are the most popular open access models, when both green and diamond are essentially just as big as the other two. The main difference is that they are generally not-for-profit, so they don’t benefit from the same visibility or attention as the other two models.

Response: We edited the sentence and removed “the most” before “common publishing models (line 2).

6. There are some minor false information or assumption about OA:

Green OA doesn’t necessarily mean embargos

7. Ultimately, I appreciate what the authors were trying to accomplish with this study, but I do believe that some major flaws in the study design (e.g. focusing on journal-based metrics) and assumptions that were made (e.g. ignoring specific factors about hybrid OA) hold it back to the point where I cannot approve of this article.

Response: We appreciate the reviewer’s feedback. As mentioned above, we have extensively revised the manuscript to describe the results from the journal-level perspective. We also added to the discussion the arguments regarding the financial barriers posed by both gold OA and hybrid journals. We hope that these revisions have improved the manuscript and its credibility.

Respectfully;

The corresponding author

Dear Prof. Bouter,

We would like to thank you for taking the time to review and evaluate our manuscript. We have addressed all your comments and have revised the manuscript accordingly. An updated version of the manuscript highlighting the changes is attached. Please find below the responses to your comments.

1. In this article 201 hybrid and 201 gold medical open access journals are compared in terms of several citation-based indicators. The findings indicate that gold open access journals have a slightly higher Impact Factor and are more likely to be in Quartile 1 of Scopus. Strangely that first key finding is not mentioned in the abstract, while the second finding is formulated in a confusing way (‘…wasn’t significantly associated..”).

Response: We added the slight difference in IF; however, it was actually the hybrid journals that tended to have Q1 ranking in Scopus more than gold OA journals. We added both results to the abstract (lines 12-14).

2. The narrative is somewhat difficult to follow, as there is a disconnect between the research question and the design of the study. The title (‘Is publishing gold open-access worth it?’) doesn’t state for whom this question will be answered. It turns out to boil down to the question whether the Article Processing Charge (APC) offers value for money in terms of impact from the author perspective. And this is where the disconnect comes in: the authors study impact at journal level, while authors will be interested in impact at article level. The problem is that indicators like the Impact Factor and other journal level indicators are poor predictors of the number of citations which a specific article in the journal at issue will get. There seems to be a growing consensus that gold open access publication leads to more citations and more citation diversity compared to paywalled publications. (See: Jeffrey Brainard. Open-access papers draw more citations from a broader readership. Science 24 January 2024. https://www.science.org/content/article/open-access-papers-draw-more-citations-broader-readership).

Response: We highly appreciate the reviewer’s highlight and the differentiation between the article level vs. the journal’s level impact, particularly in terms of citation and the diversity of citations. While we did emphasize the impact of the model on the impact at the journal level, we have edited the claims that our findings have an impact at the article level and changed them to reflect the impact at the journal level. We have revised the manuscript, including the title, to reflect that our work focused on the impact at the journal level rather than at the article level (lines 74-83 in the introduction, as well as lines 152 and 220 in the discussion). We also cited the suggested article in the updated objective statement (line 78). In addition, we added to the discussion a paragraph on the differentiation between the two levels of impact and cited some of the studies mentioned in the article you kindly provided (lines 193-214).

3. But there are more reasons why the study design is less suitable to answer the research question:

Hybrid journals are a mix of gold open access and paywalled publications, which diminishes the contrast studied.

Response: We agree that we haven’t given diamond OA sufficient attention and emphasized its importance given the financial barriers associated with both gold OA and subscription-based publishing. Therefore and to fill this gap, we added to the introduction a sentence defining diamond OA (lines 32-33). We also edited the statements defining green OA in the introduction and added the appropriate citations including the website suggested by the reviewer (lines 33-38). Moreover, we added a part discussing diamond OA and its importance in overcoming the financial barriers to the discussion section in comparison to gold OA and subscription-based publishing (lines 258-265).

5. Finally, I sympathize with the plea to lower APC, but don’t see the connection to the findings of the study. In summary, the research question cannot be answered well with the study design at issue and already seems to be answered by others.

Response: After adding the part discussing diamond OA and that it is still growing with a very limited number of journals adopting it compared with the number of gold OA journals, we believe that the context and the flow of this paragraph in the discussion provides a justification for this plea (lines 247-271).

Respectfully;

The corresponding author