Role, function, and expectations of research funding committees: Perspectives from committee members

Amanda Blatch-Jones; Cherish Boxall; Katie Meadmore

doi:10.12688/f1000research.154665.3

Home Browse Role, function, and expectations of research funding committees: Perspectives...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Brief Report

Revised

Role, function, and expectations of research funding committees: Perspectives from committee members

[version 3; peer review: 2 approved, 1 approved with reservations, 1 not approved]

Amanda Blatch-Jones ¹, Cherish Boxall², Katie Meadmore¹

PUBLISHED 06 Mar 2025

Author details Author details

¹ National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, England, S)16 7NS, UK
² Southampton Clinical Trials Unit, University of Southampton, Southampton, England, SO16 6YD, UK

Amanda Blatch-Jones
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Cherish Boxall
Roles: Data Curation, Formal Analysis, Validation, Writing – Original Draft Preparation

Katie Meadmore
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Research funding committees play an integral role in the research funding process, consisting of a range of skills, knowledge, and expertise (e.g., professional, and public contributors). Although there is some evidence that has explored the efficiency and effectiveness of funding committees in terms of the funding process, there is a lack of published evidence about the purpose, role, and function of funding committees, from the perspective of committee members.

A subset of survey data from a cohort of six National Institute for Health and Care Research (NIHR) research programmes, exploring the purpose of a funding committee, and the expectations and role of a funding committee member between October 2020 to December 2021. All committee members were eligible to participate in the survey.

Using an inductive approach, 50 completed responses (22.5% response rate) were analysed, focusing on the role of a funding committee member and the functions of a funding committee. Participants highlighted seven key areas for the purpose of a funding committee: prioritising and recommending what research to fund (n=36) and assessment of quality (n=24) being the most common responses. Four areas were considered important to the expectations and role of funding committee members, with reviewing, critically appraising, and discussing applications (n=44); and being fair, objective, and unbiased (n=27) being the most common responses.

The findings offer a unique insight into committee members’ expectations about the role, purpose and function of a funding committee and their contribution to the funding recommendation process. There was high agreement that the purpose and role of committees and their members was to offer expert advice to make fair, impartial decisions on which research should be prioritised. Exploring the purpose, role, and function of funding committees has relevance and importance for funding organisations seeking to enhance and optimise the decision-making practice of funding committees.

Keywords

Funding committees, survey, virtual committees, grant applications, peer review, qualitative, expectations, research funding

Corresponding author: Amanda Blatch-Jones

Competing interests: No competing interests were disclosed.

Grant information: This research was funded by the National Institute for Health and Care Research Coordinating Centre (NIHRCC), based at the University of Southampton, through its Research on Research programme of work. The views and opinions expressed in the discussion are those of the authors and do not necessarily reflect those of the Department of Health and Social Care. The NIHRCC had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2025 Blatch-Jones A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Blatch-Jones A, Boxall C and Meadmore K. Role, function, and expectations of research funding committees: Perspectives from committee members [version 3; peer review: 2 approved, 1 approved with reservations, 1 not approved]. F1000Research 2025, 13:1066 (https://doi.org/10.12688/f1000research.154665.3) First published: 17 Sep 2024, 13:1066 (https://doi.org/10.12688/f1000research.154665.1) Latest published: 06 Mar 2025, 13:1066 (https://doi.org/10.12688/f1000research.154665.3)

Revised Amendments from Version 2

We would like to thank the reviewers for their thoughtful and encouraging comments and feedback. We have addressed their comments and submitted a revised version of the article on the two questions from the netnographic survey data. We have addressed the comments individually and have also updated all sections (introduction, methods, discussion, and reference list) to address the feedback received. We have included more references from the peer review and funding practice literature, as offered by one reviewer. We are happy that the revised version of the article now provides a better, more readable version for the reader and addresses the points raised by both reviewers.

See the authors' detailed response to the review by Mario Coccia
See the authors' detailed response to the review by Adrian Barnett
See the authors' detailed response to the review by Santiago Vasco-Morales
See the authors' detailed response to the review by Junwen Luo

Introduction

Research funding committees (also referred to as grant panels or boards) are a core component of the peer review process. Peer review elicits expert, professional and public contributor opinions on whether to recommend or award a research proposal for funding. These committees play an important role for research funding organisations and aid in the funding recommendation and decision-making process. Abdoul et al. reported the role of the committee was to provide a clear summary of the proposal and the associated reviews, to act as an additional assessor and to establish a consensus decision.¹ However, there are variations between how funding organisations manage and conduct their funding committees; for example, some require decisions on which research proposals to award funding, whilst others provide recommendations on which proposals to fund, with the final decision laying elsewhere.² Several funding organisations provide terms of reference and guidance to support committee members’ understanding of their role, purpose and function as a member of a funding committee. For example, the National Institute for Health and Care Research (NIHR) provides information packs on their website for public contributors (https://www.nihr.ac.uk/documents/public-committee- member-roles-information-pack-for-members-of-the-public/26580) and professional committee members (https://www.nihr.ac.uk/researchers/have-your-say-on-our-research/become-a-professional-committee-member.htm).

Funding organisations may also differ in the number of peer review stages that they have, and the type of reviewer required (e.g., external review, funding committee review or community focused review). Some funding organisations use a two-stage application assessment process (stage 1 involves an outline application and stage 2 involves a full application), which involves external peer reviewing and funding committee review. It has been observed that the primary purpose of the committee at stage one is to assess the quality and value of the research question. By comparison, for stage two applications, the role of the committee is to decide which applications to recommend for funding.³

The purpose and role of the funding committee may also influence the skills needed to make an effective committee member² and how the expectations and experiences of peer reviewers (including committee members) do not always align with the stated role and purpose of the committee.⁴^–⁶ Studies that have explored motivation and participation in funding committees demonstrate differences in these expectations.²^,³^,⁷^–¹¹ For example, Gallo et al. identified that 87-92% of their participants felt that serving as a reviewer on a peer review panel had positively impacted their career, and this was influenced through improvements in writing grant applications and increased exposure to new scientific ideas compared to networking and collaboration opportunities.¹² These opportunities can unintentionally be seen to benefit or give greater advantage to those on the panel, potentially introducing the Matthew effect in research funding.⁶^,¹³

There has been considerable evidence addressing and analysing the research funding process, particularly in terms of efficiency and effectiveness of peer review by funding organisations.¹⁴^–¹⁹ However, there is still a significant lack of evidence around the funding allocation procedures more generally and there are several reasons for this.²⁰^,²¹ For example, Guthrie et al. and others have highlighted the sensitivity that surrounds funding organisations’ allocation and recommendations to fund research making this a particularly challenging context to undertake research.²⁰^,²² Others have highlighted the challenges with the accessibility to conducting real world research on grant review panels and have had to use methods such as computer modelling to explore panel choices, bias and the peer review processes across different grant review models.²¹^–²⁶

Despite these challenges, understanding how the purpose, role, and function of funding committees is perceived by committee members themselves is integral to the research funding process, and can contribute to an important research evidence gap.

This paper addresses this important gap in the evidence and presents data from an online survey conducted as part of a larger study reported elsewhere that explored the role of virtual funding committee meetings for the allocation of NIHR research funding.³ Where previous research has mainly focused on peer reviewer choice, using datasets on reviewer scores and funding decisions,¹⁴^,¹⁵^,²¹^,²⁶^–²⁸ there is limited evidence exploring what the purpose and expectation is of funding committee meetings from the perspective of funding committee members themselves.²⁹ For this reason, the aim of this paper is to provide some context around what funding committee members from several NIHR programmes consider the purpose of a funding committee to be, and what they consider the expectations of their role as funding committee member are. The insights drawn from the analysis provides a unique opportunity to explore the function and role of funding committees from a committee member perspective, which has received limited attention in the management and funding of research practice literature.

Methods

The survey data presented in this paper is a sub-sample from the main study exploring the NIHR virtual funding committee practices.³ The purpose of the survey (as part of a netnographic study) was to gain further insight and understanding of virtual funding committees and explore the social interactions, expectations, and perceptions in a virtual setting of funding committee practice. The sub-sample addressed here, focuses on funding committee members expectations and role as a member of a committee and the purpose of a funding committee generally, from committee meetings conducted by the NIHR during October 2020 to December 2021. For a full account of the study methods, please see the main study³).

Study design

An online survey conducted as part of a netnographic study. This paper only reports on the questions from an online survey about the purpose, expectations and role of funding committees and their members and not specifically related to virtual or face-to-face committee meetings (see data analysis section for the two questions). All NIHR health and social care research programme funding committees that took place between October 2020 to December 2021 were eligible to participate in the study. A single NIHR funding committee was treated as a single online community.

Data collection

All funding committee members from the NIHR research programmes who consented to participate in the netnographic study were sent a link to the survey within four weeks of the virtual funding committee taking place. The survey was open from October 2020 to January 2022. Due to the nature and sensitivity of funding committees and being able to identify respondents, no personal data were collected, unless they offered their contact details to be included in the interview part of the main study. All respondents were asked to complete an online consent form before they could access the survey. The survey was accessible for anyone with an internet connection and was held on a University of Southampton server, using Microsoft Forms. The survey included a total of 16 open and closed-ended questions, including Likert scale responses. We estimated from the pilot of the survey, it would take approximately 15-20 minutes to complete. All respondents were given three weeks to respond to the survey (with two reminders).

Data analysis

For the analysis reported here, responses from two questions relating to the purpose, role, and function of NIHR funding committees and their members were analysed using an inductive (the data drove the coding of responses) qualitative content analysis approach. The survey data was downloaded from Microsoft Forms and moved to NVivo for analysis. These two open-ended text questions were:

1. As a member of a funding committee, what do you consider the purpose of a funding committee to be?
2. What do you consider the expectations and role of a funding committee member to be?

All responses received on these two questions were coded using Nvivo software (version 1.6.1) (A free alternative Computer-Assisted Qualitative Data Analysis Software (CAQDAS) to using Nvivo is Microsoft Excel or QualCoder that is open source (https://qualcoder.wordpress.com/)). One researcher read through the responses for each question and created initial categories and themes. Relevant quotes were highlighted and coded to the initial themes, which were then reviewed by a second member of the team. The theme names and associated quotes were then revised through discussion until consensus was met. A review of the final themes were discussed, and tables were produced to determine what quotes would be used to illustrate the specific theme. All quotes were checked by both members of the team to make sure no identifiable information was present.

Results

From a potential cohort of 222 invited committee members, 50 responses were received (response rate of 22.5%) from a total of six NIHR research programmes (Efficacy and Mechanism Evaluation (EME); Evidence Synthesis (ES); Global Health Research (GHR); Health and Social Care Delivery Research (HSDR) (formerly known as Health Service and Delivery Research (HS&DR)); Health Technology Assessment (HTA); Public Health Research (PHR)) further details are reported elsewhere³). A high cloaking level was taken across all forms of data analysis, and where there was potential association to a funding committee or individual, verbatim quotes were amended from the online survey open-ended questions. It is on this basis that common themes across all six NIHR programmes were created to understand the purpose and expectations of funding committees, rather than focusing on specific NIHR programmes or specific roles of the funding committee (e.g., clinician, statistician, methodologist, health care professional, patient and public contributor).

Purpose of a funding committee

Several explanations describing the purpose of a funding committee were reported by the respondents (see Table 1 for a summary of responses). There was variation in how the respondents explained the purpose and role of a funding committee meeting; however, most respondents thought that prioritisation; assessment of the quality of an application; and prioritising applications that answer questions of importance and relevance were key functions of a funding committee to ensure recommendations of research funding were for the best science that would maximise patient benefit. Although some respondents mentioned the importance of fair and transparent reviews, ‘value for money’ and feedback to applicants, prioritising the need and benefit to the NHS, staff, patients, and the public were seen as key factors during decision-making and form the core focus of discussion at these funding committee meetings.

Table 1. Qualitative content analysis of responses on the purpose of a funding committee.

Categorisation	No. of responses	Direct quotes
To prioritise and recommend which research to fund	36	“… ultimately decide whether the research should be funded.” “To recommend and prioritise research applications for funding and/or further information.”
Assessment of quality	24	“To assess the quality of funding applications and fund the best science.” “A fair and quality assessment (technical, ethical, value for money) of the proposal.”
To prioritise research questions of importance and relevance	21	“Allocation of funds to maximise benefit with respect to reducing uncertainty in healthcare/medical knowledge.” “… answering questions of clinical importance and that are relevant and important to patients.”
To provide expert advice and opinion through review and shared discussion with a diverse group of individuals	16	“To review research project proposals and share views of multidisciplinary team (clinicians, academics, statisticians, economist, patients).” “To provide expert opinions on funding applications to decide which applications should receive public money, determining need, feasibility and rigor of the research proposed. “
To provide fair and transparent review	7	“Make fair decisions to direct funding to the strongest research proposals which lie within the remit of the funding stream.” “To observe a fair and transparent process that enables public resources to be targeted at areas of genuine health (or social care) need.”
Assessment of value for money (e.g., cost effective)	7	“Assessing proposals for appropriateness, rigour, feasibility and likely value for money, and prioritising for funding.” “To ensure that applications meet a public need, are cost effective, methodologically sound and involve the public in all aspects of the project.”
Provide and offer feedback to applicants	5	“To provide feedback to applicants to improve the research plans to maximise its potential.” “For unsuccessful applicants, feedback to support their development.”

Expectation and role of a funding committee member

Exploring how respondents described their role as a member of a committee offered insight to what they expected from the role and whether this was comparable to how the purpose of a funding committee was reported (see Table 2 for a summary of responses). By contrast to the purpose of the funding committee as a whole, the majority of responses indicated that participants felt their role was to review and critically appraise applications and to discuss these views at the committee meeting. Several respondents mentioned how the role involves being able to provide an expert opinion and an objective and unbiased review (constructive criticism). Prioritising research applications according to clinical need and patient benefit was seen as important to the committee member role but this was not frequently mentioned compared to the purpose of the committee.

Table 2. Qualitative content analysis of responses on the expectation and role of a funding committee member.

Categorisation	No. of responses	Direct quotes
To review, critically appraise and discuss applications	44	“We are expected to present and/or comment on a specific set of proposals, and we are expected to have carefully reviewed these proposals because the rest of the committee will likely follow our suggestion. In addition, we are expected to revise all the other proposals, listen carefully to other presenters, contribute to the discussion and give a fair assessment to each of the proposals. “To review and critique research proposals; summarise and communicate strengths and weaknesses of the proposals to the committee.
To be fair, objective, and unbiased	27	“Provide an honest, unbiased and thorough review of funding applications highlighting strengths and weaknesses of each application.” “… to provide fair and objective comments around its strengths and weaknesses, to articulate clearly to other committee members the overall impressions of the application, to engage in meaningful dialogue/discussion about the applications to facilitate its fair assessment.”
To provide and share expertise in reviews and discussions	21	“To bring their strengths, skills and knowledge to bear on each application.” “Provide assessment of applications, based on the member's area of expertise.”
To prioritise and recommend funding	11	“… to assist the chair in recommending and prioritising research applications for funding/further information.” “To achieve consensus on funding where possible and to have sound and evidenced reasons for dissonance where necessary.”

Discussion

This paper provides a unique insight into the purpose, role, and function of funding committees and committee members from the perspective of funding committee members, from several programmes across the NIHR. Exploring these two aspects has provided new insights into what funding committees and their members look for during the decision-making process but also how they see their role and function as part of the funding committee and in addition to funding committees’ terms of reference. Overall, from those that responded to the survey (22.5%) there was high agreement that the purpose and function of committees and their members was to offer expert advice to make fair, impartial decisions on which research should be prioritised for making funding decisions or make recommendations for funding. The results from the survey were part of the larger study, and the findings from the two questions reported here enabled committee members to provide greater clarity of their own experiences and perspectives of the committee that have received limited attention in the past.²⁵

Reassuringly, the roles of the committee reported were in line with general NIHR assessment criteria of the need for evidence, scientific rigour and value for money, and members perceived the main purpose of the committee meeting itself to be to prioritise and recommend which research to fund by assessing quality and ensuring the research is in areas of genuine need and clinical importance (based on NIHR guidance for funding committee members). Furthermore, the findings reported here also coincide and were complementary to the role of formality, process and structure of funding committee meetings in the main study.³

Although NIHR does provide applicants with feedback, which is stated in their guidelines, this was the least reported purpose of the funding committee. Providing feedback to applicants has been described as an important step for funders to undertake. Not only does it facilitate applicants in revising their research proposals (and thus increasing the quality of research proposals) but it also increases the attractiveness of the application process.³⁰ As such, it may be beneficial for committees to remind their committee members of the importance of providing high quality constructive feedback.

Interestingly, value for money was equally valued the same as providing a fair and transparent review, which is often a feature in the feedback given to applicants.³¹ Value for money is an important assessment criteria and is noted on the NIHR websites for committee members. In addition, the roles of the committee and its purpose also aligned to the NIHR terms of reference for joining a committee, including public contributors, demonstrating transparency from the NIHR in terms of the assessment criteria on how funding decisions are made by the funding committees.

In line with the role of committees, the role of committee members also touched on providing reviews that were fair, drew on expertise and prioritised the most relevant topics. However, the majority of participants felt that the role of committee members was to review, critically appraise and engage in discussions on NIHR applications. This highlights a key difference between the roles of committee members and committees as a whole, with committee members providing the evidence from which committees as a whole make recommendations of funding and prioritisation. Furthermore, although committee members perceive that part of their role is to make fair and unbiased appraisals of applications, previous literature⁹^,¹³^,²⁰^,²⁹ indicates that many aspects of peer review are susceptible and succumb to bias.¹⁸^,¹⁹^,²² Future work could explore the differences between roles, expectations, benefits, and motivations of a committee member to tease these important issues out. Better understanding of these may also allow funding organisations to create better strategies and incentives for recruiting to their committees and may also help to sustain transparency and objectivity around funding committees and the decision-making recommendations of funded research for the research community as a whole.³²^,³³

Strengths and limitations

The data reported in this paper come from two questions collected as part of a larger survey. Although the number of responses to the survey was acceptable, especially in the context of the wider study, a response rate of 22.5% is relatively low and the sample contained only those who were members of six NIHR funding committees, which could have introduced bias (those more interested and engaged could have been more likely to complete the survey and may be more inclined to report positive experiences) and limit the generalisability of the findings. Although the sample only included one funding organisation, the findings incorporated several contexts by involving six NIHR programmes, across different research types (e.g., evidence syntheses, health assessment, global health, and public health).²⁰ The data collection from 2020-2021 could also be seen as a limitation of the findings reported in this paper. Whilst the generalisability of our findings needs to be taken with caution, they provide a unique and reflective account on the purpose, role, and function of committee members that has not otherwise been explored previously. The findings add valuable contributions to the broader evidence around decision-making practices of funding organisations but more importantly addresses a gap in the evidence that goes beyond process measures and measuring effectiveness.²⁰

Variability in responses might have been due to different roles on the funding committee, terms of reference, committee inductions for new members and chairs, how long they have been a member and their professional position within academia, public community, clinical, or social care setting. Whilst the variability could also be an indicator of how funding committees are well briefed on the role and function as a member, it was not possible to explore this in more detail as no personal data were collected as part of the main study. A future study may wish to compare responses by exploring protective characteristics to better understand the different groups of people who join committees, and how these may in turn enhance the decision-making practices of funding committees and look across multiple funding organisations. To build on the existing evidence presented here from the perspective of the NIHR, a broader context from other funding organisations, or indeed multiple funding organisations across diverse contexts could generate more generalisable findings and comparative data.

Conclusion

Funding committees play an important role in the decision-making process to fund research, yet there is limited published evidence exploring their integral role, purpose, or function as committee members. The findings reported here contribute new evidence and insights about the value and importance of exploring funding committee members expectations and understanding of their role as members but also the function and purpose of a funding committee meeting. Exploring the social processes and gaining such insights also contributes to the open research agenda literature, particularly the transparency and fairness of research funding practices that are often challenging to undertake.²⁰^,³⁴ The limitations of previous research, such as Pier et al. Gallo et al. (e.g., limited access to National Institutes of Health (NIH) data and only using one study method to collect data) demonstrates the complexity of conducting research about funding research practices.⁷^,²⁷^,³⁵^,³⁶

However, although the survey data reported here saw variation across respondents’ responses, there was clear consensus about what matters in order to make informed decision-making recommendations in relation to their role as a committee member but also how, as a collective group of experts, decisions are made that are fair and transparent, offering a considered and balanced discussion to make recommendations on what applications merit funding. Enhancing the current understanding in this space deserves more reflection and reporting, particularly from other funding organisations, to determine how and in what circumstances research applications are assessed, validated, and critically appraised to inform the decision-making process, whilst also appreciating the sensitivities around these funding allocation processes.

Ethical considerations and consent

All respondents were asked to complete an online consent form before they could access the survey.

Consent from each participant of the survey was gained prior to completing the online survey. The study was approved by the University of Southampton, Faculty of Medicine Ethics Committee (ID 57541, 3^rd August 2020). This study was conducted according to the principles expressed in the Declaration of Helsinki.

Data availability

In line with our ethical approval, the survey data (alongside the interview and observational data) cannot be shared publicly to maintain confidentiality of our participants.

Full details of the survey guide are provided in OSF: Exploring virtual funding committee practices in the allocation of National Institute for Health and Care Research funding: A netnographic study. https://doi.org/10.17605/OSF.IO/RZ6VT.³⁷

The project contains the following data:

• Appendix 4 Survey questions.docx

Specified materials and guides are available under the terms of the CC0 1.0 Public domain dedication (https://creativecommons.org/publicdomain/zero/1.0/). Additional requirements or requests are available from the corresponding author: A.J.Blatch-Jones@southampton.ac.uk) for researchers who meet the criteria for access to the data (e.g., researchers organisational affiliation, intended use of the data, and/or type of data).

This main study is registered at RoR Registry and Hub (study number 805), accessible here: https://ror-hub.org/study/805/.

Reporting guidelines: OSF repository. SRQR checklist for “Exploring virtual funding committee practices in the allocation of National Institute for Health and Care Research funding: A netnographic study.” (https://osf.io/rz6vt/³⁷).

Data are available under the terms of the Crreative Commons Attribution International License (CC BY 4.0).

Acknowledgements

We would like to thank all respondents that took part in the online survey. We would also like to thank those who reviewed early draft versions of the article, and Emmanuel Asante for his contribution to observations reported in the main netnography study.

References

1. Abdoul H, Perrey C, Amiel P, et al. et al.: Peer review of grant applications: criteria used and qualitative study of reviewer practices.2012.
2. Steiner Davis ML, Conner TR, Miller-Bains K, et al.: What makes an effective grants peer reviewer? An exploratory study of the necessary skills. PLoS One. 2020; 15(5): e0232327. PubMed Abstract | Publisher Full Text | Free Full Text
3. Blatch-Jones AJ, Boxall C, Asante E, et al.: Exploring virtual funding committee practices in the allocation of National Institute for Health and Care Research funding: A netnographic study. medRxiv. 2023. 2023.11.17.23298707.
4. Coveney J, Herbert DL, Hill K, et al.: ‘Are you siding with a personality or the grant proposal?’: observations on how peer review panels function. Res. Integr. Peer Rev. 2017; 2(1): 19. Publisher Full Text
5. Herbert DL, Graves N, Clarke P, et al.: Using simplified peer review processes to fund research: a prospective study. BMJ Open. 2015; 5(7): e008380. PubMed Abstract | Publisher Full Text
6. Giannelos K, Wiarda M, Doorn N: Challenges to ethical public engagement in research funding: a perspective from practice. Open Res Eur. 2024; 4: 179. PubMed Abstract | Publisher Full Text | Free Full Text
7. Gallo SA, Carpenter AS, Glisson SR: Teleconference versus face-to-face scientific peer review of grant application: effects on review outcomes. PLoS One. 2013; 8(8): e71693. PubMed Abstract | Publisher Full Text | Free Full Text
8. Gallo SA, Schmaling KB, Thompson LA, et al.: Grant reviewer perceptions of the quality, effectiveness, and influence of panel.2020.
9. Vallée-Tourangeau G, Wheelock A, Vandrevala T, et al.: Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences. Humanit. Soc. Sci. Commun. 2022; 9(1): 1–11.
10. Gallo SA, Schmaling KB, Thompson LA, et al.: Grant review feedback: Appropriateness and usefulness. Sci. Eng. Ethics. 2021; 27: 1–20.
11. Abma-Schouten R, Gijbels J, Reijmerink W, et al.: Evaluation of research proposals by peer review panels: broader panels for broader assessments? Sci. Public Policy. 2023; 50(4): 619–632. Publisher Full Text
12. Gallo SA, Thompson LA, Schmaling KB, et al.: The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey. Sci. Eng. Ethics. 2020; 26(2): 761–782. PubMed Abstract | Publisher Full Text
13. Bol T, de Vaan M , van de Rijt A : The Matthew effect in science funding. Proc. Natl. Acad. Sci. 2018; 115(19): 4887–4890. PubMed Abstract | Publisher Full Text | Free Full Text
14. Shankar K, Luo J, Ma L, et al.: SPRING 2020 survey: peer review of grant proposals.Figshare.2021. [Last accessed 16/12/2024]]. Publisher Full Text
15. Feliciani T, Luo J, Shankar K: Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study. Quant. Sci. Stud. 2022; 3(3): 832–856. Publisher Full Text
16. Holbrook JB, Hrotic S: Blue skies, impacts, and peer review. RT A J. Res. Policy Eval. 2013; 1(1). Publisher Full Text
17. Obrecht M, Tibelius K, D'Aloisio G: Examining the value added by committee discussion in the review of applications for research awards. Res. Eval. 2007; 16(2): 70–91. Publisher Full Text
18. Schmaling KB, Evenson GR, Marble BK, et al.: Perceptions of grant peer reviewers: a mixed methods study. Res. Eval. 2024; 33. Publisher Full Text
19. Gallo S, Thompson L, Schmaling K, et al.: Risk evaluation in peer review of grant applications. Environ. Syst. Decis. 2018; 38(2): 216–229. Publisher Full Text
20. Guthrie S, Ghiga I, Wooding S: What do we know about grant peer review in the health sciences? F1000Res. 2017; 6: 6. Publisher Full Text
21. Feliciani T, Morreau M, Luo J, et al.: Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model. Res. Policy. 2022; 51(4): 104467. Publisher Full Text
22. Cristall N, Drozdowska BA, Fladt J, et al.: Priorities and expectations of researchers, funders, patients and the public regarding the funding of medical research: results from the PERSPECT qualitative study. BMJ open. 2024; 14(11): e084655. PubMed Abstract | Publisher Full Text | Free Full Text
23. Langfeldt L: Expert panels evaluating research: decision-making and sources of bias. Res. Eval. 2004; 13(1): 51–62. Publisher Full Text
24. Lee CJ: Commensuration Bias in Peer Review. Philos. Sci. 2015; 82(5): 1272–1283. Publisher Full Text
25. Langfeldt L: The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome. Soc. Stud. Sci. 2001; 31(6): 820–841. Publisher Full Text
26. Luo J, Feliciani T, Reinhart M, et al.: Analyzing sentiments in peer review reports: Evidence from two science funding agencies. Quant. Sci. Stud. 2021; 2(4): 1271–1295. Publisher Full Text
27. Pier EL, Brauer M, Filut A, et al.: Low agreement among reviewers evaluating the same NIH grant applications. Proc. Natl. Acad. Sci. 2018; 115(12): 2952–2957. Publisher Full Text
28. van den Besselaar P , Leydesdorff L: Past performance, peer review and project selection: a case study in the social and behavioral sciences. Res. Eval. 2009; 18(4): 273–288. Publisher Full Text
29. van den Besselaar P , Sandström U, Schiffbaenker H: Studying grant decision-making: a linguistic analysis of review reports. Scientometrics. 2018; 117(1): 313–329. PubMed Abstract | Publisher Full Text | Free Full Text
30. Fackrell K, Meadmore K, Recio-Saucedo A, et al.: Identification and comparison of key criteria of feedback of funding decisions: mixed-methods analysis of funder and applicant perspectives. BMJ Open. 2021; 11: e048979. PubMed Abstract | Publisher Full Text | Free Full Text
31. National Institute for Health and Care Research (NIHR): Guidance for NIHR Committee Member Development Scheme (CMDS) Participant applicants.2020. Reference Source
32. Bouter LM: Fostering responsible research practices is a shared responsibility of multiple stakeholders. J. Clin. Epidemiol. 2018; 96: 143–146. PubMed Abstract | Publisher Full Text
33. Munafò MR, Nosek BA, Bishop DVM, et al.: A manifesto for reproducible science. Nat. Hum. Behav. 2017; 1(1): 0021. PubMed Abstract | Publisher Full Text | Free Full Text
34. Guthrie S, Rincon DR, McInroy G, et al.: Measuring bias, burden and conservatism in research funding processes. F1000Research. 2019; 8(851): 851. Publisher Full Text
35. Wilsdon J: Are we all metascientists now? The shifting landscape for research on research.2024. [Last accessed: 16/12/2024]. Publisher Full Text
36. Ioannidis JPA: Meta-research: Why research on research matters. PLoS Biol. 2018; 16(3): e2005468. PubMed Abstract | Publisher Full Text | Free Full Text
37. Blatch-Jones A; Katie Meadmore, Cherish BoxallAsante E: Exploring Virtual Funding Committee Practices in the Allocation of National Institute for Health and Care Research Funding: A Netnographic Study. OSF. July 31 2024. Publisher Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 17 Sep 2024

Author details Author details

Cherish Boxall
Roles: Data Curation, Formal Analysis, Validation, Writing – Original Draft Preparation

Katie Meadmore
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This research was funded by the National Institute for Health and Care Research Coordinating Centre (NIHRCC), based at the University of Southampton, through its Research on Research programme of work. The views and opinions expressed in the discussion are those of the authors and do not necessarily reflect those of the Department of Health and Social Care. The NIHRCC had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (3)

version 3

Revised

Published: 06 Mar 2025, 13:1066

https://doi.org/10.12688/f1000research.154665.3

version 2

Revised

Published: 02 Jan 2025, 13:1066

https://doi.org/10.12688/f1000research.154665.2

version 1

Published: 17 Sep 2024, 13:1066

https://doi.org/10.12688/f1000research.154665.1

© 2025 Blatch-Jones A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Blatch-Jones A, Boxall C and Meadmore K. Role, function, and expectations of research funding committees: Perspectives from committee members [version 3; peer review: 2 approved, 1 approved with reservations, 1 not approved]. F1000Research 2025, 13:1066 (https://doi.org/10.12688/f1000research.154665.3)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 3

VERSION 3

PUBLISHED 06 Mar 2025

Revised

Views

Reviewer Report 11 Mar 2025

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

Approved

https://doi.org/10.5256/f1000research.178710.r369816

No ... Continue reading

CITE

Report a concern

Respond or Comment

Version 2

VERSION 2

PUBLISHED 02 Jan 2025

Revised

Views

Reviewer Report 13 Feb 2025

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

Approved with Reservations

https://doi.org/10.5256/f1000research.176120.r365675

This paper has a clear question and is well reported. Funding panels are an understudied area relative to their importance, so evidence in this area is welcome. The study is relatively small in terms of sample size and the number of questions asked. Some results are fundamental, for example the commonly mentioned purposes in Table 1.

A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.

Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?

The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.

Citation number 3 can be updated from the preprint.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: statistics, research funding

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

06 Mar 2025

Author Response

Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems ... Continue reading Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.
Response: Thank you for your comment. We have added additional detail of the potential bias in the data to the limitations section.
Please note that this brief report only reports on two questions from a survey about the role, function and expectations of funding committee members. The full survey covered additional questions that did elicit both positive and negative responses. This is reported in the full paper that can be found here: https://f1000research.com/articles/13-338
Along with the supplementary material outlining the survey questions: https://osf.io/rz6vt/

Comment 2: Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?
Response: Applicants receive feedback from the panel committee as part of the funding allocation process. This is highlighted and explained in the NIHR guidelines. We have now added a paragraph to the discussion that discusses feedback.

Comment 3:The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.
Response: Thank you. We have added some additional text to the revised version to reflect your comment.
(those more interested and engaged could have been more likely to complete the survey)
Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.
Response: Thank you for your comment. We have added additional detail of the potential bias in the data to the limitations section.
Please note that this brief report only reports on two questions from a survey about the role, function and expectations of funding committee members. The full survey covered additional questions that did elicit both positive and negative responses. This is reported in the full paper that can be found here: https://f1000research.com/articles/13-338
Along with the supplementary material outlining the survey questions: https://osf.io/rz6vt/

Comment 2: Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?
Response: Applicants receive feedback from the panel committee as part of the funding allocation process. This is highlighted and explained in the NIHR guidelines. We have now added a paragraph to the discussion that discusses feedback.

Comment 3:The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.
Response: Thank you. We have added some additional text to the revised version to reflect your comment.
(those more interested and engaged could have been more likely to complete the survey)
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

06 Mar 2025

Author Response

Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems ... Continue reading Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.
Response: Thank you for your comment. We have added additional detail of the potential bias in the data to the limitations section.
Please note that this brief report only reports on two questions from a survey about the role, function and expectations of funding committee members. The full survey covered additional questions that did elicit both positive and negative responses. This is reported in the full paper that can be found here: https://f1000research.com/articles/13-338
Along with the supplementary material outlining the survey questions: https://osf.io/rz6vt/

Comment 2: Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?
Response: Applicants receive feedback from the panel committee as part of the funding allocation process. This is highlighted and explained in the NIHR guidelines. We have now added a paragraph to the discussion that discusses feedback.

Comment 3:The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.
Response: Thank you. We have added some additional text to the revised version to reflect your comment.
(those more interested and engaged could have been more likely to complete the survey)
Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.
Response: Thank you for your comment. We have added additional detail of the potential bias in the data to the limitations section.
Please note that this brief report only reports on two questions from a survey about the role, function and expectations of funding committee members. The full survey covered additional questions that did elicit both positive and negative responses. This is reported in the full paper that can be found here: https://f1000research.com/articles/13-338
Along with the supplementary material outlining the survey questions: https://osf.io/rz6vt/

Comment 2: Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?
Response: Applicants receive feedback from the panel committee as part of the funding allocation process. This is highlighted and explained in the NIHR guidelines. We have now added a paragraph to the discussion that discusses feedback.

Comment 3:The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.
Response: Thank you. We have added some additional text to the revised version to reflect your comment.
(those more interested and engaged could have been more likely to complete the survey)
Competing Interests: None Close
Report a concern

Views

Reviewer Report 22 Jan 2025

Mario Coccia, CNR -- National Research Council of Italy, Collegio Carlo Alberto, Moncalieri, Italy; Arizona State University - Downtown Phoenix Campus, Phoenix, Arizona, USA

Not Approved

https://doi.org/10.5256/f1000research.176120.r356498

Role, function, and expectations of research funding committees: Perspectives from committee members

The topics of this paper are interesting, though well known. The structure and content must be revised, and results have to be better explained by authors before to be reconsidered.

The title has to be shorter and should indicate the areas of these committees.

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background. After that authors can focus on the topics of this study to provide a correct analysis for fruitful discussion (See suggested readings that must be all read and used in the text).

Methods of this study can be clarified. The section of Materials and methods must be re-structured with the following three sections, clarifying the logic of the inquiry:
•   Sample and data. Accurate description, also indicating the research fields and expertise.
•   Measures of variables
•   Data analysis procedure.

Results
Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections).
I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.

Discussion.
First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies.
Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Authors have to comment on whether the results were expected for each set of findings; go into greater depth to explain unexpected findings. Moreover, either compare your results with the findings from other studies or use the studies to support these results. It is important to discuss how a gap in literature has been addressed. Expectations should be clarified on what it is based on and the logic of the decision process. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices.

Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.

I strongly suggest improving the paper by using all comments (suggested papers are included to be read and used all).

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.

Schmaling, K.B., Evenson, G.R., Marble, B.K., Gallo, S.A. 2024 (Ref-1). Perceptions of grant peer reviewers: a mixed methods study. Research Evaluation, 33, rvae050

Gallo, S., Thompson, L., Schmaling, K., Glisson, S. 2018 (Ref-2). Risk evaluation in peer review of grant applications. Environment Systems and Decisions, 38(2), pp. 216–229

Coccia M. 2023 (Ref-3). New Perspectives in Innovation Failure Analysis: A taxonomy of general errors and strategic management for reducing risks. Technology in Society, vol. 75, n. 102384, https://doi.org/10.1016/j.techsoc.2023.102384

Cristall, N., Drozdowska, B.A., Fladt, J., ... Goyal, M., Ganesh, A. 2024 (Ref-4). Priorities and expectations of researchers, funders, patients and the public regarding the funding of medical research: results from the PERSPECT qualitative study.BMJ Open, 14(11), e084655

Coccia M. 2024 (Ref-5). Theory of errors in innovation failure and strategic management of winning by failing. Technology Analysis & Strategic Management. https://doi.org/10.1080/09537325.2024.2383604

Giannelos, K., Wiarda, M., Doorn, N. 2024 (Ref-6). Challenges to ethical public engagement in research funding: a perspective from practice.Open Research Europe, 4, 179

Ardito L., Coccia M., Messeni Petruzzelli A. 2021 (Ref-7). Technological exaptation and crisis management: Evidence from COVID-19 outbreaks. R&D Management, vol. 51, n. 4, pp. 381-392, https://doi.org/10.1111/radm.12455

Schweiger, G., Barnett, A., van den Besselaar, P., ... Sandström, U., Conix, S. 2024 (Ref-8). The costs of competition in distributing scarce research funds   Proceedings of the National Academy of Sciences of the United States of America, 121(50), e2407644121 2024

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Schmaling K, Evenson G, Marble B, Gallo S: Perceptions of grant peer reviewers: a mixed methods study. Research Evaluation. 2024; 33. Publisher Full Text
2. Gallo S, Thompson L, Schmaling K, Glisson S: Risk evaluation in peer review of grant applications. Environment Systems and Decisions. 2018; 38 (2): 216-229 Publisher Full Text
3. Coccia M: New Perspectives in Innovation Failure Analysis: A taxonomy of general errors and strategic management for reducing risks. Technology in Society. 2023; 75. Publisher Full Text
4. Cristall N, Drozdowska BA, Fladt J, Jaroenngarmsamer T, et al.: Priorities and expectations of researchers, funders, patients and the public regarding the funding of medical research: results from the PERSPECT qualitative study.BMJ Open. 2024; 14 (11): e084655 PubMed Abstract | Publisher Full Text
5. Coccia M: Theory of errors in innovation failure and strategic management of winning by failing. Technology Analysis & Strategic Management. 2024. 1-15 Publisher Full Text
6. Giannelos K, Wiarda M, Doorn N: Challenges to ethical public engagement in research funding: a perspective from practice.Open Res Eur. 2024; 4: 179 PubMed Abstract | Publisher Full Text
7. Ardito L, Coccia M, Messeni Petruzzelli A: Technological exaptation and crisis management: Evidence from COVID‐19 outbreaks. R&D Management. 2021; 51 (4): 381-392 Publisher Full Text
8. Schweiger G, Barnett A, van den Besselaar P, Bornmann L, et al.: The costs of competition in distributing scarce research funds.Proc Natl Acad Sci U S A. 2024; 121 (50): e2407644121 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: science and research policy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Author Response 06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

06 Mar 2025

Author Response

The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the ... Continue reading The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the title further, we have not added the funding committee research areas. The research areas have now been clearly articulated in the document (health and social care).

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.
The purpose of the brief report was to provide further clarification on two specific questions from the survey (that is reported in full elsewhere: https://f1000research.com/articles/13-338) as the role, function and expectations from the perspective of committee members is limited. The abstract provides insight for funders’ to ensure they optimise the decision-making practices for the allocation of funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background.
Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. Paragraph 4 indicates the gaps in the evidence and why it is challenging to undertake research in the context of funding committee’s role and function.
The full article provides greater detail, and the brief report is linked to that article (link provided above for convenience).

Methods of this study can be clarified. Sample and data. Accurate description, also indicating the research fields and expertise.
Due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. As such we are unable to provide details of the individual committee members who took part in this survey. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Results Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections). I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.
The data presented is qualitative due to the nature of the survey questions being open-ended and are not reporting any data that would be conducive to graphs. The results are provided under the two questions as stated in the introduction and methods section. The two tables offer insight to the qualitative content analysis, including examples of quotes from the two survey questions. This is a common way to present data from qualitative studies.

Discussion. First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies. Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.
Following the brief report guidelines, we have provided a summary of what was presented in the brief report, and why it is relevant to existing literature. We also compared what was reported by committee members to the NIHR guidelines.
Due to brief reports having a word limit of 2500, the guideline for a discussion includes “The results should be discussed in the context of existing literature. Strengths and weaknesses of the study should be presented and discussed. Future directions for work should be raised.” We have included the risk of potential bias under limitations and have included a statement about what direction this could be.
This was a qualitative study that was exploring committee member perspectives of the purpose, role, and function of funding committees and committee members, and was analyzed in an inductive way. As such, had no prior expectations/framework of what committee members would report. In the discussion and within the word limitation, we have tried to discuss the findings in the context of prior literature where possible.
Furthermore, due to the qualitative nature of the study, this means that the findings from this study cannot be generalised in the same way as other studies. This has been mentioned in the limitations. However, we hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices.
As described in the discussion, to date, there is very limited evidence that describes the purpose, role, and function of committees and their members from a committee member perspective. This study is therefore a step in addressing this gap.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices. Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.
As this brief report is based data from two qualitative questions, it was not appropriate to extract generalizable recommendations. Furthermore, as per brief report guidelines, a conclusion section is optional, but we have included this to offer some future considerations for the reader. We hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The full paper provides a more detailed conclusion based on your suggestions. However, in relation to this brief paper, we felt that it is not relevant to include these aspects. Thank you for suggesting them.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.
We have responded to your comments individually.

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.
Thank you for your suggestions.

Where appropriate we have added relevant references to explain how previous research has focused on specific areas, although there is limited evidence directly from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Peer review is not the feature of the article however some of the references have been added to further demonstrate the gap in the evidence. Thank you.
The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the title further, we have not added the funding committee research areas. The research areas have now been clearly articulated in the document (health and social care).

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.
The purpose of the brief report was to provide further clarification on two specific questions from the survey (that is reported in full elsewhere: https://f1000research.com/articles/13-338) as the role, function and expectations from the perspective of committee members is limited. The abstract provides insight for funders’ to ensure they optimise the decision-making practices for the allocation of funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background.
Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. Paragraph 4 indicates the gaps in the evidence and why it is challenging to undertake research in the context of funding committee’s role and function.
The full article provides greater detail, and the brief report is linked to that article (link provided above for convenience).

Methods of this study can be clarified. Sample and data. Accurate description, also indicating the research fields and expertise.
Due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. As such we are unable to provide details of the individual committee members who took part in this survey. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Results Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections). I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.
The data presented is qualitative due to the nature of the survey questions being open-ended and are not reporting any data that would be conducive to graphs. The results are provided under the two questions as stated in the introduction and methods section. The two tables offer insight to the qualitative content analysis, including examples of quotes from the two survey questions. This is a common way to present data from qualitative studies.

Discussion. First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies. Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.
Following the brief report guidelines, we have provided a summary of what was presented in the brief report, and why it is relevant to existing literature. We also compared what was reported by committee members to the NIHR guidelines.
Due to brief reports having a word limit of 2500, the guideline for a discussion includes “The results should be discussed in the context of existing literature. Strengths and weaknesses of the study should be presented and discussed. Future directions for work should be raised.” We have included the risk of potential bias under limitations and have included a statement about what direction this could be.
This was a qualitative study that was exploring committee member perspectives of the purpose, role, and function of funding committees and committee members, and was analyzed in an inductive way. As such, had no prior expectations/framework of what committee members would report. In the discussion and within the word limitation, we have tried to discuss the findings in the context of prior literature where possible.
Furthermore, due to the qualitative nature of the study, this means that the findings from this study cannot be generalised in the same way as other studies. This has been mentioned in the limitations. However, we hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices.
As described in the discussion, to date, there is very limited evidence that describes the purpose, role, and function of committees and their members from a committee member perspective. This study is therefore a step in addressing this gap.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices. Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.
As this brief report is based data from two qualitative questions, it was not appropriate to extract generalizable recommendations. Furthermore, as per brief report guidelines, a conclusion section is optional, but we have included this to offer some future considerations for the reader. We hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The full paper provides a more detailed conclusion based on your suggestions. However, in relation to this brief paper, we felt that it is not relevant to include these aspects. Thank you for suggesting them.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.
We have responded to your comments individually.

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.
Thank you for your suggestions.

Where appropriate we have added relevant references to explain how previous research has focused on specific areas, although there is limited evidence directly from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Peer review is not the feature of the article however some of the references have been added to further demonstrate the gap in the evidence. Thank you.
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

06 Mar 2025

Author Response

The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the ... Continue reading The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the title further, we have not added the funding committee research areas. The research areas have now been clearly articulated in the document (health and social care).

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.
The purpose of the brief report was to provide further clarification on two specific questions from the survey (that is reported in full elsewhere: https://f1000research.com/articles/13-338) as the role, function and expectations from the perspective of committee members is limited. The abstract provides insight for funders’ to ensure they optimise the decision-making practices for the allocation of funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background.
Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. Paragraph 4 indicates the gaps in the evidence and why it is challenging to undertake research in the context of funding committee’s role and function.
The full article provides greater detail, and the brief report is linked to that article (link provided above for convenience).

Methods of this study can be clarified. Sample and data. Accurate description, also indicating the research fields and expertise.
Due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. As such we are unable to provide details of the individual committee members who took part in this survey. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Results Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections). I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.
The data presented is qualitative due to the nature of the survey questions being open-ended and are not reporting any data that would be conducive to graphs. The results are provided under the two questions as stated in the introduction and methods section. The two tables offer insight to the qualitative content analysis, including examples of quotes from the two survey questions. This is a common way to present data from qualitative studies.

Discussion. First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies. Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.
Following the brief report guidelines, we have provided a summary of what was presented in the brief report, and why it is relevant to existing literature. We also compared what was reported by committee members to the NIHR guidelines.
Due to brief reports having a word limit of 2500, the guideline for a discussion includes “The results should be discussed in the context of existing literature. Strengths and weaknesses of the study should be presented and discussed. Future directions for work should be raised.” We have included the risk of potential bias under limitations and have included a statement about what direction this could be.
This was a qualitative study that was exploring committee member perspectives of the purpose, role, and function of funding committees and committee members, and was analyzed in an inductive way. As such, had no prior expectations/framework of what committee members would report. In the discussion and within the word limitation, we have tried to discuss the findings in the context of prior literature where possible.
Furthermore, due to the qualitative nature of the study, this means that the findings from this study cannot be generalised in the same way as other studies. This has been mentioned in the limitations. However, we hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices.
As described in the discussion, to date, there is very limited evidence that describes the purpose, role, and function of committees and their members from a committee member perspective. This study is therefore a step in addressing this gap.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices. Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.
As this brief report is based data from two qualitative questions, it was not appropriate to extract generalizable recommendations. Furthermore, as per brief report guidelines, a conclusion section is optional, but we have included this to offer some future considerations for the reader. We hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The full paper provides a more detailed conclusion based on your suggestions. However, in relation to this brief paper, we felt that it is not relevant to include these aspects. Thank you for suggesting them.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.
We have responded to your comments individually.

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.
Thank you for your suggestions.

Where appropriate we have added relevant references to explain how previous research has focused on specific areas, although there is limited evidence directly from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Peer review is not the feature of the article however some of the references have been added to further demonstrate the gap in the evidence. Thank you.
The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the title further, we have not added the funding committee research areas. The research areas have now been clearly articulated in the document (health and social care).

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.
The purpose of the brief report was to provide further clarification on two specific questions from the survey (that is reported in full elsewhere: https://f1000research.com/articles/13-338) as the role, function and expectations from the perspective of committee members is limited. The abstract provides insight for funders’ to ensure they optimise the decision-making practices for the allocation of funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background.
Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. Paragraph 4 indicates the gaps in the evidence and why it is challenging to undertake research in the context of funding committee’s role and function.
The full article provides greater detail, and the brief report is linked to that article (link provided above for convenience).

Methods of this study can be clarified. Sample and data. Accurate description, also indicating the research fields and expertise.
Due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. As such we are unable to provide details of the individual committee members who took part in this survey. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Results Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections). I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.
The data presented is qualitative due to the nature of the survey questions being open-ended and are not reporting any data that would be conducive to graphs. The results are provided under the two questions as stated in the introduction and methods section. The two tables offer insight to the qualitative content analysis, including examples of quotes from the two survey questions. This is a common way to present data from qualitative studies.

Discussion. First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies. Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.
Following the brief report guidelines, we have provided a summary of what was presented in the brief report, and why it is relevant to existing literature. We also compared what was reported by committee members to the NIHR guidelines.
Due to brief reports having a word limit of 2500, the guideline for a discussion includes “The results should be discussed in the context of existing literature. Strengths and weaknesses of the study should be presented and discussed. Future directions for work should be raised.” We have included the risk of potential bias under limitations and have included a statement about what direction this could be.
This was a qualitative study that was exploring committee member perspectives of the purpose, role, and function of funding committees and committee members, and was analyzed in an inductive way. As such, had no prior expectations/framework of what committee members would report. In the discussion and within the word limitation, we have tried to discuss the findings in the context of prior literature where possible.
Furthermore, due to the qualitative nature of the study, this means that the findings from this study cannot be generalised in the same way as other studies. This has been mentioned in the limitations. However, we hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices.
As described in the discussion, to date, there is very limited evidence that describes the purpose, role, and function of committees and their members from a committee member perspective. This study is therefore a step in addressing this gap.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices. Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.
As this brief report is based data from two qualitative questions, it was not appropriate to extract generalizable recommendations. Furthermore, as per brief report guidelines, a conclusion section is optional, but we have included this to offer some future considerations for the reader. We hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The full paper provides a more detailed conclusion based on your suggestions. However, in relation to this brief paper, we felt that it is not relevant to include these aspects. Thank you for suggesting them.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.
We have responded to your comments individually.

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.
Thank you for your suggestions.

Where appropriate we have added relevant references to explain how previous research has focused on specific areas, although there is limited evidence directly from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Peer review is not the feature of the article however some of the references have been added to further demonstrate the gap in the evidence. Thank you.
Competing Interests: None Close
Report a concern

Views

Reviewer Report 03 Jan 2025

Santiago Vasco-Morales, Neonatología, Hospital Gineco Obstétrico Isidro Ayora, Quito, Pichincha, Ecuador; Universidad Central del Ecuador, Quito, Pichincha, Ecuador

Approved

https://doi.org/10.5256/f1000research.176120.r355325

This report evaluates an article on the research funding committees of the National Institute for Health and Care Research (NIHR). The article was restructured by including more details about the survey design and providing supplemental materials. The discussion of the ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 17 Sep 2024

Views

Reviewer Report 25 Nov 2024

Santiago Vasco-Morales, Neonatología, Hospital Gineco Obstétrico Isidro Ayora, Quito, Pichincha, Ecuador; Universidad Central del Ecuador, Quito, Pichincha, Ecuador

Approved with Reservations

https://doi.org/10.5256/f1000research.169716.r341502

Summary of the Article
This study explores the roles, expectations, and functions of research funding committees, focusing on the perspectives of committee members from six National Institute for Health and Care Research (NIHR) programs. Using data collected via an online survey, the authors identify key themes in the purpose of funding committees, such as prioritizing high-quality research, providing fair and unbiased reviews, and aligning decisions with public and clinical needs. The results contribute to the understanding of funding committee dynamics and highlight gaps in current practices, particularly regarding transparency and efficiency in decision-making.
The study design employs qualitative content analysis to analyze responses, providing insights into the expectations and experiences of committee members. While the methodology is sound, several areas for improvement have been identified to enhance clarity, reproducibility, and scientific rigor.
Detailed Review
The following is an expanded evaluation of the article based on the key review questions:

1. Is the work clearly and accurately presented, and does it cite the current literature?
Answer: Partly
Criticisms:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.

Recommendations:
Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings andsubstantiate claims.
Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.

2. Is the study design appropriate, and is the work technically sound?
Answer: Partly
Criticisms:

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.

Recommendations:
Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives.
Discuss the potential impact of the low response rate and how it may affect the representativeness of findings.
Clearly separate analysis and discussion of individual versus collective roles.

3. Are sufficient details of methods and analysis provided to allow replication by others?
Answer: Partly
Criticisms:

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.

Recommendations:
Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics).
Provide an appendix or supplementary materials with the full survey questionnaire.
Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized.

4. If applicable, is the statistical analysis and its interpretation appropriate?
Answer: Not applicable
Since the study focuses on qualitative content analysis, statistical analysis is not a core component. However, descriptive statistics, such as response frequencies, are used appropriately.

5. Are all the source data underlying the results available to ensure full reproducibility?
Answer: Partly
Criticisms:

Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.

Recommendations:
Share anonymized or synthesized data to the extent possible, ensuring confidentiality.
Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.

6. Are the conclusions drawn adequately supported by the results?
Answer: Partly
Criticisms:

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.

Recommendations:
Deepen the discussion of how findings apply to funding committee practices and align with existing research.
Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.

Points That Must Be Addressed
To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:
- Include detailed descriptions of the survey design and coding process.
- Provide more illustrative examples of data categorization.
Improve Literature Integration:
- Expand the discussion to connect findings with existing studies on peer review and funding committees.
Acknowledge Limitations:
- Discuss the impact of the low response rate on findings.
- Clearly state the limitations in generalizing results beyond the NIHR context.
Provide Additional Data:
- Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Bouter LM: Fostering responsible research practices is a shared responsibility of multiple stakeholders.J Clin Epidemiol. 2018; 96: 143-146 PubMed Abstract | Publisher Full Text
2. Munafò MR, Nosek BA, Bishop DVM, Button KS, et al.: A manifesto for reproducible science.Nat Hum Behav. 2017; 1 (1): 0021 PubMed Abstract | Publisher Full Text
3. Ioannidis JPA: Meta-research: Why research on research matters.PLoS Biol. 2018; 16 (3): e2005468 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: porque Pediatrics, neonatology, medical education

CITE

Report a concern

Author Response 02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

02 Jan 2025

Author Response
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader ... Continue reading
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Thank you for your comment. We have revised the literature and added additional references.

Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.
We have revised the content of the article to address this point. Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Recommendations: Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings and substantiate claims. Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.
Thank you. We have addressed your comment (and others) in relation to the wider literature and added in some additional text to the discussion. Due to the nature of brief papers, we did not go in depth as the survey complemented the wider netnographic study. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Only two survey questions are related to this article (both reported under data analysis). Both questions are provided in the article. Links to the main study are provided. Data collection is included in the article, including timeframe and process of the complete survey.

Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
In this brief paper, the two survey questions that are reported explore this distinction. Q1 relates to the funding committee and Q2 to the individual, and Table 1 and 2 respectively show the findings for how the roles of the committee and the individuals are perceived by the funding committee members. We have made this distinction clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

In addition, due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.
This is covered under strengths and limitations section.

Recommendations: Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives. Discuss the potential impact of the low response rate and how it may affect the representativeness of findings. Clearly separate analysis and discussion of individual versus collective roles.
This is reported in the brief paper but due to the 2500-word limit, providing the full details of the methods is not feasible, and it is reported elsewhere. We have reported the low response rate under the discussion section. It is not possible to report findings at an individual level, as reported above. However, we have made the distinction of the findings clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
A summary is provided in the brief paper. As stated, full details are reported elsewhere along with the wider themes of the main study. The purpose of the brief paper is to highlight the findings from two questions, as the current literature lacks evidence from real-world, first-hand understanding of what funding committee members perceive their role and function to be as a committee member and what the function and expectation is of a funding committee panel. As detailed, most of the literature is based on modelling, data, peer review, grant proposals, scoring etc. These studies do not address the two questions asked in the survey. By understanding the perceptions from committee members directly can help funding organisations enhance their guidance and consider where improvements are needed.

Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.
These are reported in the main study and in the OSF repository as stated under the data availability statement. All materials are provided but F1000Research policy does not allow supplementary materials.

Recommendations: Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics). Provide an appendix or supplementary materials with the full survey questionnaire. Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized. Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.
All relevant materials that comply with ethical procedures are provided in the OSF repository and linked to the main article as well.

Recommendations: Share anonymized or synthesized data to the extent possible, ensuring confidentiality. Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.
This is available on request as stated under data availability, to comply with ethical procedures, and to protect the identity of all participants. This is not a requirement for publication due to the sensitive nature of the data. However, we clearly state it is available on request.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
We have revised sections of the brief paper to reflect your comments and feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.
Thank you for your comment. This is noted under strengths and limitations. However, as with previous literature, most of the research is with one funder (several references listed demonstrate this point), and the context of each funding organisation will always be a limitation for all research studies. We have already stated this within the article.

Recommendations: Deepen the discussion of how findings apply to funding committee practices and align with existing research. Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.
The netnographic study goes into depth on the aspects you mention. However, as noted this is a brief paper and is limited to 2500 words. Also, as per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The generalisability aspect is already noted in the article but we have extended a sentence to cover your comment.

Points That Must Be Addressed: To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:

Include detailed descriptions of the survey design and coding process.

Provide more illustrative examples of data categorization.

Improve Literature Integration:

Expand the discussion to connect findings with existing studies on peer review and funding committees.

Acknowledge Limitations:

Discuss the impact of the low response rate on findings.

Clearly state the limitations in generalizing results beyond the NIHR context.

Provide Additional Data:

Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Please also see more detailed comments above that relate to these points and how we have addressed them. As described in the paper, please refer to the main study and the OSF repository, for a more comprehensive description of the whole survey design and analysis.

We have included additional references throughout the paper. However, due to the nature of brief papers, we did not go in depth on these.

Low response rate and generalisability are now covered under the strengths and limitations section.

All details and supplementary material are provided in the main study and the OSF repository. F1000Research policy does not allow supplementary materials.

Thank you for your suggestions and detailed feedback.
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Thank you for your comment. We have revised the literature and added additional references.

Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.
We have revised the content of the article to address this point. Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Recommendations: Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings and substantiate claims. Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.
Thank you. We have addressed your comment (and others) in relation to the wider literature and added in some additional text to the discussion. Due to the nature of brief papers, we did not go in depth as the survey complemented the wider netnographic study. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Only two survey questions are related to this article (both reported under data analysis). Both questions are provided in the article. Links to the main study are provided. Data collection is included in the article, including timeframe and process of the complete survey.

Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
In this brief paper, the two survey questions that are reported explore this distinction. Q1 relates to the funding committee and Q2 to the individual, and Table 1 and 2 respectively show the findings for how the roles of the committee and the individuals are perceived by the funding committee members. We have made this distinction clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

In addition, due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.
This is covered under strengths and limitations section.

Recommendations: Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives. Discuss the potential impact of the low response rate and how it may affect the representativeness of findings. Clearly separate analysis and discussion of individual versus collective roles.
This is reported in the brief paper but due to the 2500-word limit, providing the full details of the methods is not feasible, and it is reported elsewhere. We have reported the low response rate under the discussion section. It is not possible to report findings at an individual level, as reported above. However, we have made the distinction of the findings clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
A summary is provided in the brief paper. As stated, full details are reported elsewhere along with the wider themes of the main study. The purpose of the brief paper is to highlight the findings from two questions, as the current literature lacks evidence from real-world, first-hand understanding of what funding committee members perceive their role and function to be as a committee member and what the function and expectation is of a funding committee panel. As detailed, most of the literature is based on modelling, data, peer review, grant proposals, scoring etc. These studies do not address the two questions asked in the survey. By understanding the perceptions from committee members directly can help funding organisations enhance their guidance and consider where improvements are needed.

Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.
These are reported in the main study and in the OSF repository as stated under the data availability statement. All materials are provided but F1000Research policy does not allow supplementary materials.

Recommendations: Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics). Provide an appendix or supplementary materials with the full survey questionnaire. Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized. Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.
All relevant materials that comply with ethical procedures are provided in the OSF repository and linked to the main article as well.

Recommendations: Share anonymized or synthesized data to the extent possible, ensuring confidentiality. Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.
This is available on request as stated under data availability, to comply with ethical procedures, and to protect the identity of all participants. This is not a requirement for publication due to the sensitive nature of the data. However, we clearly state it is available on request.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
We have revised sections of the brief paper to reflect your comments and feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.
Thank you for your comment. This is noted under strengths and limitations. However, as with previous literature, most of the research is with one funder (several references listed demonstrate this point), and the context of each funding organisation will always be a limitation for all research studies. We have already stated this within the article.

Recommendations: Deepen the discussion of how findings apply to funding committee practices and align with existing research. Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.
The netnographic study goes into depth on the aspects you mention. However, as noted this is a brief paper and is limited to 2500 words. Also, as per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The generalisability aspect is already noted in the article but we have extended a sentence to cover your comment.

Points That Must Be Addressed: To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:

Include detailed descriptions of the survey design and coding process.

Provide more illustrative examples of data categorization.

Improve Literature Integration:

Expand the discussion to connect findings with existing studies on peer review and funding committees.

Acknowledge Limitations:

Discuss the impact of the low response rate on findings.

Clearly state the limitations in generalizing results beyond the NIHR context.

Provide Additional Data:

Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Please also see more detailed comments above that relate to these points and how we have addressed them. As described in the paper, please refer to the main study and the OSF repository, for a more comprehensive description of the whole survey design and analysis.

We have included additional references throughout the paper. However, due to the nature of brief papers, we did not go in depth on these.

Low response rate and generalisability are now covered under the strengths and limitations section.

All details and supplementary material are provided in the main study and the OSF repository. F1000Research policy does not allow supplementary materials.

Thank you for your suggestions and detailed feedback.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

02 Jan 2025

Author Response
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader ... Continue reading
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Thank you for your comment. We have revised the literature and added additional references.

Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.
We have revised the content of the article to address this point. Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Recommendations: Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings and substantiate claims. Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.
Thank you. We have addressed your comment (and others) in relation to the wider literature and added in some additional text to the discussion. Due to the nature of brief papers, we did not go in depth as the survey complemented the wider netnographic study. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Only two survey questions are related to this article (both reported under data analysis). Both questions are provided in the article. Links to the main study are provided. Data collection is included in the article, including timeframe and process of the complete survey.

Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
In this brief paper, the two survey questions that are reported explore this distinction. Q1 relates to the funding committee and Q2 to the individual, and Table 1 and 2 respectively show the findings for how the roles of the committee and the individuals are perceived by the funding committee members. We have made this distinction clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

In addition, due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.
This is covered under strengths and limitations section.

Recommendations: Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives. Discuss the potential impact of the low response rate and how it may affect the representativeness of findings. Clearly separate analysis and discussion of individual versus collective roles.
This is reported in the brief paper but due to the 2500-word limit, providing the full details of the methods is not feasible, and it is reported elsewhere. We have reported the low response rate under the discussion section. It is not possible to report findings at an individual level, as reported above. However, we have made the distinction of the findings clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
A summary is provided in the brief paper. As stated, full details are reported elsewhere along with the wider themes of the main study. The purpose of the brief paper is to highlight the findings from two questions, as the current literature lacks evidence from real-world, first-hand understanding of what funding committee members perceive their role and function to be as a committee member and what the function and expectation is of a funding committee panel. As detailed, most of the literature is based on modelling, data, peer review, grant proposals, scoring etc. These studies do not address the two questions asked in the survey. By understanding the perceptions from committee members directly can help funding organisations enhance their guidance and consider where improvements are needed.

Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.
These are reported in the main study and in the OSF repository as stated under the data availability statement. All materials are provided but F1000Research policy does not allow supplementary materials.

Recommendations: Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics). Provide an appendix or supplementary materials with the full survey questionnaire. Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized. Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.
All relevant materials that comply with ethical procedures are provided in the OSF repository and linked to the main article as well.

Recommendations: Share anonymized or synthesized data to the extent possible, ensuring confidentiality. Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.
This is available on request as stated under data availability, to comply with ethical procedures, and to protect the identity of all participants. This is not a requirement for publication due to the sensitive nature of the data. However, we clearly state it is available on request.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
We have revised sections of the brief paper to reflect your comments and feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.
Thank you for your comment. This is noted under strengths and limitations. However, as with previous literature, most of the research is with one funder (several references listed demonstrate this point), and the context of each funding organisation will always be a limitation for all research studies. We have already stated this within the article.

Recommendations: Deepen the discussion of how findings apply to funding committee practices and align with existing research. Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.
The netnographic study goes into depth on the aspects you mention. However, as noted this is a brief paper and is limited to 2500 words. Also, as per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The generalisability aspect is already noted in the article but we have extended a sentence to cover your comment.

Points That Must Be Addressed: To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:

Include detailed descriptions of the survey design and coding process.

Provide more illustrative examples of data categorization.

Improve Literature Integration:

Expand the discussion to connect findings with existing studies on peer review and funding committees.

Acknowledge Limitations:

Discuss the impact of the low response rate on findings.

Clearly state the limitations in generalizing results beyond the NIHR context.

Provide Additional Data:

Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Please also see more detailed comments above that relate to these points and how we have addressed them. As described in the paper, please refer to the main study and the OSF repository, for a more comprehensive description of the whole survey design and analysis.

We have included additional references throughout the paper. However, due to the nature of brief papers, we did not go in depth on these.

Low response rate and generalisability are now covered under the strengths and limitations section.

All details and supplementary material are provided in the main study and the OSF repository. F1000Research policy does not allow supplementary materials.

Thank you for your suggestions and detailed feedback.
Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Thank you for your comment. We have revised the literature and added additional references.

Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.
We have revised the content of the article to address this point. Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Recommendations: Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings and substantiate claims. Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.
Thank you. We have addressed your comment (and others) in relation to the wider literature and added in some additional text to the discussion. Due to the nature of brief papers, we did not go in depth as the survey complemented the wider netnographic study. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Only two survey questions are related to this article (both reported under data analysis). Both questions are provided in the article. Links to the main study are provided. Data collection is included in the article, including timeframe and process of the complete survey.

Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
In this brief paper, the two survey questions that are reported explore this distinction. Q1 relates to the funding committee and Q2 to the individual, and Table 1 and 2 respectively show the findings for how the roles of the committee and the individuals are perceived by the funding committee members. We have made this distinction clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

In addition, due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.
This is covered under strengths and limitations section.

Recommendations: Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives. Discuss the potential impact of the low response rate and how it may affect the representativeness of findings. Clearly separate analysis and discussion of individual versus collective roles.
This is reported in the brief paper but due to the 2500-word limit, providing the full details of the methods is not feasible, and it is reported elsewhere. We have reported the low response rate under the discussion section. It is not possible to report findings at an individual level, as reported above. However, we have made the distinction of the findings clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
A summary is provided in the brief paper. As stated, full details are reported elsewhere along with the wider themes of the main study. The purpose of the brief paper is to highlight the findings from two questions, as the current literature lacks evidence from real-world, first-hand understanding of what funding committee members perceive their role and function to be as a committee member and what the function and expectation is of a funding committee panel. As detailed, most of the literature is based on modelling, data, peer review, grant proposals, scoring etc. These studies do not address the two questions asked in the survey. By understanding the perceptions from committee members directly can help funding organisations enhance their guidance and consider where improvements are needed.

Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.
These are reported in the main study and in the OSF repository as stated under the data availability statement. All materials are provided but F1000Research policy does not allow supplementary materials.

Recommendations: Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics). Provide an appendix or supplementary materials with the full survey questionnaire. Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized. Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.
All relevant materials that comply with ethical procedures are provided in the OSF repository and linked to the main article as well.

Recommendations: Share anonymized or synthesized data to the extent possible, ensuring confidentiality. Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.
This is available on request as stated under data availability, to comply with ethical procedures, and to protect the identity of all participants. This is not a requirement for publication due to the sensitive nature of the data. However, we clearly state it is available on request.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
We have revised sections of the brief paper to reflect your comments and feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.
Thank you for your comment. This is noted under strengths and limitations. However, as with previous literature, most of the research is with one funder (several references listed demonstrate this point), and the context of each funding organisation will always be a limitation for all research studies. We have already stated this within the article.

Recommendations: Deepen the discussion of how findings apply to funding committee practices and align with existing research. Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.
The netnographic study goes into depth on the aspects you mention. However, as noted this is a brief paper and is limited to 2500 words. Also, as per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The generalisability aspect is already noted in the article but we have extended a sentence to cover your comment.

Points That Must Be Addressed: To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:

Include detailed descriptions of the survey design and coding process.

Provide more illustrative examples of data categorization.

Improve Literature Integration:

Expand the discussion to connect findings with existing studies on peer review and funding committees.

Acknowledge Limitations:

Discuss the impact of the low response rate on findings.

Clearly state the limitations in generalizing results beyond the NIHR context.

Provide Additional Data:

Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Please also see more detailed comments above that relate to these points and how we have addressed them. As described in the paper, please refer to the main study and the OSF repository, for a more comprehensive description of the whole survey design and analysis.

We have included additional references throughout the paper. However, due to the nature of brief papers, we did not go in depth on these.

Low response rate and generalisability are now covered under the strengths and limitations section.

All details and supplementary material are provided in the main study and the OSF repository. F1000Research policy does not allow supplementary materials.

Thank you for your suggestions and detailed feedback.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 23 Oct 2024

Junwen Luo, Economics, University College Cork Business School, Cork, County Cork, Ireland

Approved with Reservations

https://doi.org/10.5256/f1000research.169716.r328098

This research is drawn from a small size survey which serves a larger research projects involving observations and interviews of the same pool of research funding reviewers. Overall, its research method, objective, and analysis are solid but can provide deeper thoughts and reflections in several ways.

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the prorgamme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Langfeldt L: Expert panels evaluating research: decision-making and sources of bias. Research Evaluation. 2004; 13 (1): 51-62 Publisher Full Text
2. Lee C: Commensuration Bias in Peer Review. Philosophy of Science. 2015; 82 (5): 1272-1283 Publisher Full Text
3. Langfeldt L: The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome. Social Studies of Science. 2001; 31 (6): 820-841 Publisher Full Text
4. Luo J, Feliciani T, Reinhart M, Hartstein J, et al.: Analyzing sentiments in peer review reports: Evidence from two science funding agencies. Quantitative Science Studies. 2021; 2 (4): 1271-1295 Publisher Full Text
5. van den Besselaar P, Leydesdorff L: Past performance, peer review and project selection: a case study in the social and behavioral sciences. Research Evaluation. 2009; 18 (4): 273-288 Publisher Full Text
6. S, Hrotic J B, Holbrook: Blue skies, impacts, and peer review. https://riviste.unimi.it/index.php/roars/article/view/2914. 2013. Publisher Full Text
7. Obrecht M, Tibelius K, D'Aloisio G: Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation. 2007; 16 (2): 70-91 Publisher Full Text
8. Pier E, Brauer M, Filut A, Kaatz A, et al.: Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences. 2018; 115 (12): 2952-2957 Publisher Full Text
9. Feliciani T, Luo J, Shankar K: Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study. Quantitative Science Studies. 2022; 3 (3): 832-856 Publisher Full Text
10. Feliciani T, Morreau M, Luo J, Lucas P, et al.: Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model. Research Policy. 2022; 51 (4). Publisher Full Text
11. van den Besselaar P, Sandström U, Schiffbaenker H: Studying grant decision-making: a linguistic analysis of review reports. Scientometrics. 2018; 117 (1): 313-329 Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: research evaluation, impact assessment, peer review

CITE

Report a concern

Author Response 02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

02 Jan 2025

Author Response

In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s ... Continue reading In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?
Response: Thank you. We have extended the discussion to take account of your feedback.

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11
Response: Thank you for these additional references. Where appropriate we have added these references to explain how previous research has focused on specific areas, although there is limited evidence from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?
Response: Thank you for your feedback. The survey data was at an individual level, and it is only this data presented in the brief report, as the full survey as part of the netnographic study is reported elsewhere.
Under data analysis we explain the two questions used in this article, and the process we used.

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the programme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.
Response: All NIHR programmes were invited to participate and six NIHR programmes participated in the netnographic study. To make this clearer all six NIHR programmes have been added in the results section for transparency purposes. Given the six NIHR programmes have different health focuses and aims, we have not included additional details or analysis on this. However, alignment to overall NIHR aims is discussed in paragraph 2 of the discussion. We have revised the discussion section based on your feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).
Response: Thank you for sharing this survey on peer review.
In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?
Response: Thank you. We have extended the discussion to take account of your feedback.

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11
Response: Thank you for these additional references. Where appropriate we have added these references to explain how previous research has focused on specific areas, although there is limited evidence from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?
Response: Thank you for your feedback. The survey data was at an individual level, and it is only this data presented in the brief report, as the full survey as part of the netnographic study is reported elsewhere.
Under data analysis we explain the two questions used in this article, and the process we used.

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the programme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.
Response: All NIHR programmes were invited to participate and six NIHR programmes participated in the netnographic study. To make this clearer all six NIHR programmes have been added in the results section for transparency purposes. Given the six NIHR programmes have different health focuses and aims, we have not included additional details or analysis on this. However, alignment to overall NIHR aims is discussed in paragraph 2 of the discussion. We have revised the discussion section based on your feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).
Response: Thank you for sharing this survey on peer review.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

02 Jan 2025

Author Response

In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s ... Continue reading In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?
Response: Thank you. We have extended the discussion to take account of your feedback.

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11
Response: Thank you for these additional references. Where appropriate we have added these references to explain how previous research has focused on specific areas, although there is limited evidence from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?
Response: Thank you for your feedback. The survey data was at an individual level, and it is only this data presented in the brief report, as the full survey as part of the netnographic study is reported elsewhere.
Under data analysis we explain the two questions used in this article, and the process we used.

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the programme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.
Response: All NIHR programmes were invited to participate and six NIHR programmes participated in the netnographic study. To make this clearer all six NIHR programmes have been added in the results section for transparency purposes. Given the six NIHR programmes have different health focuses and aims, we have not included additional details or analysis on this. However, alignment to overall NIHR aims is discussed in paragraph 2 of the discussion. We have revised the discussion section based on your feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).
Response: Thank you for sharing this survey on peer review.
In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?
Response: Thank you. We have extended the discussion to take account of your feedback.

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11
Response: Thank you for these additional references. Where appropriate we have added these references to explain how previous research has focused on specific areas, although there is limited evidence from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?
Response: Thank you for your feedback. The survey data was at an individual level, and it is only this data presented in the brief report, as the full survey as part of the netnographic study is reported elsewhere.
Under data analysis we explain the two questions used in this article, and the process we used.

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the programme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.
Response: All NIHR programmes were invited to participate and six NIHR programmes participated in the netnographic study. To make this clearer all six NIHR programmes have been added in the results section for transparency purposes. Given the six NIHR programmes have different health focuses and aims, we have not included additional details or analysis on this. However, alignment to overall NIHR aims is discussed in paragraph 2 of the discussion. We have revised the discussion section based on your feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).
Response: Thank you for sharing this survey on peer review.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 17 Sep 2024

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3	4
Version 3 (revision) 06 Mar 25				read
Version 2 (revision) 02 Jan 25		read	read	read
Version 1 17 Sep 24	read	read

Junwen Luo, University College Cork Business School, Cork, Ireland
Santiago Vasco-Morales, Hospital Gineco Obstétrico Isidro Ayora, Quito, Ecuador; Universidad Central del Ecuador, Quito, Ecuador
Mario Coccia, Collegio Carlo Alberto, Moncalieri, Italy; Arizona State University - Downtown Phoenix Campus, Phoenix, USA
Adrian Barnett, Queensland University of Technology, Brisbane, Australia

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

0 Views

11 Mar 2025 | for Version 3

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

0 Views Cite this report Responses(0)

Approved

No further comments.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistics, research funding

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

11 Views

13 Feb 2025 | for Version 2

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

11 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistics, research funding

Respond to this report

Responses (1)

Author Response

06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

Comment 1: A limitation that is not addressed is that neither of the two qualitative questions (page 4) prompted the reviewers to consider negative aspects of their experiences and/or systems changes that could be made. Hence the results may give an overly positive view of panels and the expectations of reviewers.
Response: Thank you for your comment. We have added additional detail of the potential bias in the data to the limitations section.
Please note that this brief report only reports on two questions from a survey about the role, function and expectations of funding committee members. The full survey covered additional questions that did elicit both positive and negative responses. This is reported in the full paper that can be found here: https://f1000research.com/articles/13-338
Along with the supplementary material outlining the survey questions: https://osf.io/rz6vt/

Comment 2: Feedback was mentioned as a purpose, but is this also mentioned in the NIHR guidelines? And what feedback to applicants get?
Response: Applicants receive feedback from the panel committee as part of the funding allocation process. This is highlighted and explained in the NIHR guidelines. We have now added a paragraph to the discussion that discusses feedback.

Comment 3:The small sample size may “introduce bias” (page 7), but it would be better to indicate the direction. The likely issue is that more interested and engaged panel members completed the survey, hence again the results may be slanted towards those who take their role more seriously.
Response: Thank you. We have added some additional text to the revised version to reflect your comment.
(those more interested and engaged could have been more likely to complete the survey)

View more View less

Competing Interests

None

Back to all reports

Reviewer Report

13 Views

22 Jan 2025 | for Version 2

Mario Coccia, CNR -- National Research Council of Italy, Collegio Carlo Alberto, Moncalieri, Italy; Arizona State University - Downtown Phoenix Campus, Phoenix, Arizona, USA

13 Views Cite this report Responses(1)

Not Approved

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

science and research policy

Respond to this report

Responses (1)

Author Response

06 Mar 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

The title has to be shorter and should indicate the areas of these committees.
The title is within the requirements for F1000Research brief reports.
So as not to lengthen the title further, we have not added the funding committee research areas. The research areas have now been clearly articulated in the document (health and social care).

Abstract has to clarify the goal and implications of research policy of results to improve decision process in funding.
The purpose of the brief report was to provide further clarification on two specific questions from the survey (that is reported in full elsewhere: https://f1000research.com/articles/13-338) as the role, function and expectations from the perspective of committee members is limited. The abstract provides insight for funders’ to ensure they optimise the decision-making practices for the allocation of funding.

Introduction has to better clarify the research questions of this study, indicating the gap presents in literature that this study endeavors to cover, and provide more theoretical background.
Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. Paragraph 4 indicates the gaps in the evidence and why it is challenging to undertake research in the context of funding committee’s role and function.
The full article provides greater detail, and the brief report is linked to that article (link provided above for convenience).

Methods of this study can be clarified. Sample and data. Accurate description, also indicating the research fields and expertise.
Due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. As such we are unable to provide details of the individual committee members who took part in this survey. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Results Authors have to avoid subheadings that create fragmentation and confusion. If necessary, can use bullet points (same comments for all sections). I suggest also to represent some basic results with appropriate graphical representation, such as bar graphs et. to be clear for readers.
The data presented is qualitative due to the nature of the survey questions being open-ended and are not reporting any data that would be conducive to graphs. The results are provided under the two questions as stated in the introduction and methods section. The two tables offer insight to the qualitative content analysis, including examples of quotes from the two survey questions. This is a common way to present data from qualitative studies.

Discussion. First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies. Although the Results section provides a detailed description, there needs to be a more critical synthesis and comparison of the findings with the literature. A discussion about possible biases in decision processes is important as well as considering the risk of failure in the research project funded, considering main errors to avoid. Now the results are too vague and general. In general the expectations are associated with research fields and considering the overall ecosystem, also end users, if there are scarce resources or if there are emergencies (such as COVID) and committees can apply exaptation approaches, etc..., all these aspects have to be clarified.
Following the brief report guidelines, we have provided a summary of what was presented in the brief report, and why it is relevant to existing literature. We also compared what was reported by committee members to the NIHR guidelines.
Due to brief reports having a word limit of 2500, the guideline for a discussion includes “The results should be discussed in the context of existing literature. Strengths and weaknesses of the study should be presented and discussed. Future directions for work should be raised.” We have included the risk of potential bias under limitations and have included a statement about what direction this could be.
This was a qualitative study that was exploring committee member perspectives of the purpose, role, and function of funding committees and committee members, and was analyzed in an inductive way. As such, had no prior expectations/framework of what committee members would report. In the discussion and within the word limitation, we have tried to discuss the findings in the context of prior literature where possible.
Furthermore, due to the qualitative nature of the study, this means that the findings from this study cannot be generalised in the same way as other studies. This has been mentioned in the limitations. However, we hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices.
As described in the discussion, to date, there is very limited evidence that describes the purpose, role, and function of committees and their members from a committee member perspective. This study is therefore a step in addressing this gap.

Conclusion is slightly short. Conclusion has not to be a summary, but authors have to focus on manifold limitations (move here section presents in previous paragraph). Moreover, the Conclusion does not adequately discuss the theoretical and managerial implications of the study. Authors should claim for how the results can be applied more generally. They should clarify and systematize lessons learned in tables, proposing recommendations that can help improve a situation, or highlighting best practices. Make sure you create 3 subsections in the Conclusion: 1) Theoretical Implications, 2) Managerial or Policy Implications, and 3) Future Research.
As this brief report is based data from two qualitative questions, it was not appropriate to extract generalizable recommendations. Furthermore, as per brief report guidelines, a conclusion section is optional, but we have included this to offer some future considerations for the reader. We hope that overall learnings can be taken from the study and that it may prompt funders to reflect on their own funding committee practices. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The full paper provides a more detailed conclusion based on your suggestions. However, in relation to this brief paper, we felt that it is not relevant to include these aspects. Thank you for suggesting them.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.
We have responded to your comments individually.

Suggested readings of relevant papers that have to be read and used in the text and references to improve theoretical framework, discussion and related implications for research policy.
Thank you for your suggestions.

Where appropriate we have added relevant references to explain how previous research has focused on specific areas, although there is limited evidence directly from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Peer review is not the feature of the article however some of the references have been added to further demonstrate the gap in the evidence. Thank you.

View more View less

Competing Interests

None

Back to all reports

Reviewer Report

8 Views

03 Jan 2025 | for Version 2

Santiago Vasco-Morales, Neonatología, Hospital Gineco Obstétrico Isidro Ayora, Quito, Pichincha, Ecuador; Universidad Central del Ecuador, Quito, Pichincha, Ecuador

8 Views Cite this report Responses(0)

Approved

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Pediatrics, neonatology, medical education

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

48 Views

25 Nov 2024 | for Version 1

Santiago Vasco-Morales, Neonatología, Hospital Gineco Obstétrico Isidro Ayora, Quito, Pichincha, Ecuador; Universidad Central del Ecuador, Quito, Pichincha, Ecuador

48 Views Cite this report Responses(1)

Approved With Reservations

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.

Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.

Clarify Methods and Analysis:
- Include detailed descriptions of the survey design and coding process.
- Provide more illustrative examples of data categorization.
Improve Literature Integration:
- Expand the discussion to connect findings with existing studies on peer review and funding committees.
Acknowledge Limitations:
- Discuss the impact of the low response rate on findings.
- Clearly state the limitations in generalizing results beyond the NIHR context.
Provide Additional Data:
- Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

porque Pediatrics, neonatology, medical education

Respond to this report

Responses (1)

Author Response

02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

Thank you for your feedback. Please see the comments below:

Integration with Literature: While the article cites relevant studies, it does not adequately connect its findings to the broader body of literature on peer review and funding committees.
Thank you for your comment. We have revised the literature and added additional references.

Inconsistent Claims: The claim that there is no published evidence on the roles and expectations of funding committees is too broad. Relevant studies (e.g., Langfeldt, 2004; Lee, 2015) are mentioned in the peer review comments but not included in the discussion.
We have revised the content of the article to address this point. Due to the limitations of brief reports (2500 words) we have briefly mentioned the relevant literature and have made it clearer that this article only focuses on the responses from the survey about how funding committee members answered two questions on the purpose and expectation and role of funding committee meetings. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Recommendations: Incorporate additional references from the literature on peer review and funding committee practices to contextualize findings and substantiate claims. Expand the discussion section to critically compare findings with existing evidence, highlighting similarities and differences.
Thank you. We have addressed your comment (and others) in relation to the wider literature and added in some additional text to the discussion. Due to the nature of brief papers, we did not go in depth as the survey complemented the wider netnographic study. As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Survey Design: The survey questions are not fully described, limiting understanding of how data was collected.
Only two survey questions are related to this article (both reported under data analysis). Both questions are provided in the article. Links to the main study are provided. Data collection is included in the article, including timeframe and process of the complete survey.

Role Distinction: The study does not clearly distinguish between the roles of individual committee members and the collective purpose of the committee.
In this brief paper, the two survey questions that are reported explore this distinction. Q1 relates to the funding committee and Q2 to the individual, and Table 1 and 2 respectively show the findings for how the roles of the committee and the individuals are perceived by the funding committee members. We have made this distinction clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

In addition, due to the sensitivity of the data and being able to identify members of committees and funding programmes, as reported in the article a high cloaking level was used to ensure anonymity of all committee members. However, this does not impact the results due to being inclusive of all members and all members having equal contribution to the analysis and results.

Low Response Rate: A 22.5% response rate may introduce bias and limit generalizability.
This is covered under strengths and limitations section.

Recommendations: Provide a detailed breakdown of the survey design, including sample questions and their alignment with study objectives. Discuss the potential impact of the low response rate and how it may affect the representativeness of findings. Clearly separate analysis and discussion of individual versus collective roles.
This is reported in the brief paper but due to the 2500-word limit, providing the full details of the methods is not feasible, and it is reported elsewhere. We have reported the low response rate under the discussion section. It is not possible to report findings at an individual level, as reported above. However, we have made the distinction of the findings clearer in the discussion and added additional text comparing the role of committee members to the role of committees.

Lack of Transparency in Coding: Details on the coding process, inter-coder reliability, and how themes were finalized are not provided.
A summary is provided in the brief paper. As stated, full details are reported elsewhere along with the wider themes of the main study. The purpose of the brief paper is to highlight the findings from two questions, as the current literature lacks evidence from real-world, first-hand understanding of what funding committee members perceive their role and function to be as a committee member and what the function and expectation is of a funding committee panel. As detailed, most of the literature is based on modelling, data, peer review, grant proposals, scoring etc. These studies do not address the two questions asked in the survey. By understanding the perceptions from committee members directly can help funding organisations enhance their guidance and consider where improvements are needed.

Limited Survey Details: The full set of survey questions is not included, and the categorization of responses lacks illustrative examples.
These are reported in the main study and in the OSF repository as stated under the data availability statement. All materials are provided but F1000Research policy does not allow supplementary materials.

Recommendations: Describe the coding process in detail, including how consistency was ensured (e.g., inter-coder reliability metrics). Provide an appendix or supplementary materials with the full survey questionnaire. Include more examples of raw data (e.g., anonymized quotes) to demonstrate how responses were categorized. Supplementary Data: Additional resources, such as detailed coding frameworks or categorized data, are not provided.
All relevant materials that comply with ethical procedures are provided in the OSF repository and linked to the main article as well.

Recommendations: Share anonymized or synthesized data to the extent possible, ensuring confidentiality. Provide coding frameworks and categorized themes as supplementary materials to support reproducibility.
This is available on request as stated under data availability, to comply with ethical procedures, and to protect the identity of all participants. This is not a requirement for publication due to the sensitive nature of the data. However, we clearly state it is available on request.

Superficial Interpretation: While the conclusions are consistent with the results, they do not sufficiently explore broader implications or practical applications.
We have revised sections of the brief paper to reflect your comments and feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Generality of Claims: Conclusions could overgeneralize findings from a single funding organization to the broader context of research funding.
Thank you for your comment. This is noted under strengths and limitations. However, as with previous literature, most of the research is with one funder (several references listed demonstrate this point), and the context of each funding organisation will always be a limitation for all research studies. We have already stated this within the article.

Recommendations: Deepen the discussion of how findings apply to funding committee practices and align with existing research. Acknowledge limitations in generalizability and provide recommendations for future studies across diverse funding contexts.
The netnographic study goes into depth on the aspects you mention. However, as noted this is a brief paper and is limited to 2500 words. Also, as per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader. The generalisability aspect is already noted in the article but we have extended a sentence to cover your comment.

Points That Must Be Addressed: To make the article scientifically sound, the following points must be addressed:

Clarify Methods and Analysis:
- Include detailed descriptions of the survey design and coding process.
- Provide more illustrative examples of data categorization.
Improve Literature Integration:
- Expand the discussion to connect findings with existing studies on peer review and funding committees.
Acknowledge Limitations:
- Discuss the impact of the low response rate on findings.
- Clearly state the limitations in generalizing results beyond the NIHR context.
Provide Additional Data:
- Share supplementary materials (e.g., survey questions, coding frameworks) to enhance transparency and reproducibility.

Please also see more detailed comments above that relate to these points and how we have addressed them. As described in the paper, please refer to the main study and the OSF repository, for a more comprehensive description of the whole survey design and analysis.

We have included additional references throughout the paper. However, due to the nature of brief papers, we did not go in depth on these.
Low response rate and generalisability are now covered under the strengths and limitations section.
All details and supplementary material are provided in the main study and the OSF repository. F1000Research policy does not allow supplementary materials.

Thank you for your suggestions and detailed feedback.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

13 Views

23 Oct 2024 | for Version 1

Junwen Luo, Economics, University College Cork Business School, Cork, County Cork, Ireland

13 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

research evaluation, impact assessment, peer review

Respond to this report

Responses (1)

Author Response

02 Jan 2025

Amanda Blatch-Jones, National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, School of Healthcare Enterprise and Innovation, Southampton, S)16 7NS, UK

In response to your feedback please see the comments below:

First, the quality and contribution of the paper will be improved if the authors could address how this paper’s conclusion is connected with the other aspects of the larger project as well as with the broad literature. For instance, are those reviewers’ internal expected roles and functions consistent with your observations of the committee meetings and especially the reviewer’s discussions and interactions on how to deal with conflicting views?
Response: Thank you. We have extended the discussion to take account of your feedback.

By the way, some of the literature addressing such topics should be considered especially when you stated ‘there is no published evidence exploring their integral role, purpose, or function as committee members’, see below:
Langfeldt, L. (2004). Ref 1
Lee, C.J. (2015) Ref 2
Langfeldt, L. (2001). Ref 3
Luo, J., Feliciani, T., Reinhart, M.,.., & Shankar, K. (2021). Ref 4
Van den Besselaar, P., and Leydesdorff, L. (2009) Ref 5
Holbrook, J.B., and Hrotic, S. (2013) Ref 6
Obrecht, M., Tibelius, K., & D’Aloisio, G. (2007).Ref 7
Pier, E.L., et al. (2018) Ref 8
Feliciani T, Luo J, Shankar, K (2022). Ref 9
Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022)Ref 10
Van den Besselaar, P., Sandström, U. & Schiffbaenker, H. (2018). Ref 11
Response: Thank you for these additional references. Where appropriate we have added these references to explain how previous research has focused on specific areas, although there is limited evidence from committee members about what they perceive their role to be and the function of committee members. This particular focus is different to the assessment and process of peer review, review scores and decision-making through external peer review or panel/grant meetings.

Second, for the analysis and results of content analysis, I would have expected more illustrations on your categorizations of the purposes and expectations which are your key findings, like how you derive those categorizations from the quotes and particular how DIFFERENT are those categorizations from each other? It’s also not very clear how you separated the expectations of the whole committee and of the individual reviewers in the study design, maybe through your survey questions settings?
Response: Thank you for your feedback. The survey data was at an individual level, and it is only this data presented in the brief report, as the full survey as part of the netnographic study is reported elsewhere.
Under data analysis we explain the two questions used in this article, and the process we used.

Third, an important aspect of the funding committee study, in my eyes, should be discussed which is the programme objectives and target groups. This is found highly important for the committee discussion regarding how to align evaluation criteria into specific programme reviews and how to make fair and transparent grant reviews. A brief introduction of the programme that you studied should be provided in the Methods section. Discussion section could address more clearly how your findings can be connected to the existing literature (see above suggested ones) and make contributions to the broad area of research evaluation and peer review.
Response: All NIHR programmes were invited to participate and six NIHR programmes participated in the netnographic study. To make this clearer all six NIHR programmes have been added in the results section for transparency purposes. Given the six NIHR programmes have different health focuses and aims, we have not included additional details or analysis on this. However, alignment to overall NIHR aims is discussed in paragraph 2 of the discussion. We have revised the discussion section based on your feedback.
As per brief report guidelines, a conclusion section is optional, but we have included this to provide context and future considerations for the reader.

Last but not least, I appreciate very much that the authors shared their research methods and guidelines including the survey questions that this paper made use of. For your reference, I am sharing another grant reviewer survey that you might find relevant and useful
https://doi.org/10.6084/m9.figshare.13651058.v1 (Shankar et al., 2021).
Response: Thank you for sharing this survey on peer review.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Abdoul H, Perrey C, Amiel P, et al. et al.: Peer review of grant applications: criteria used and qualitative study of reviewer practices.2012.

[2] 2. Steiner Davis ML, Conner TR, Miller-Bains K, et al.: What makes an effective grants peer reviewer? An exploratory study of the necessary skills. PLoS One. 2020; 15(5): e0232327. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Blatch-Jones AJ, Boxall C, Asante E, et al.: Exploring virtual funding committee practices in the allocation of National Institute for Health and Care Research funding: A netnographic study. medRxiv. 2023. 2023.11.17.23298707.

[4] 4. Coveney J, Herbert DL, Hill K, et al.: ‘Are you siding with a personality or the grant proposal?’: observations on how peer review panels function. Res. Integr. Peer Rev. 2017; 2(1): 19. Publisher Full Text

[5] 5. Herbert DL, Graves N, Clarke P, et al.: Using simplified peer review processes to fund research: a prospective study. BMJ Open. 2015; 5(7): e008380. PubMed Abstract | Publisher Full Text

[6] 6. Giannelos K, Wiarda M, Doorn N: Challenges to ethical public engagement in research funding: a perspective from practice. Open Res Eur. 2024; 4: 179. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Gallo SA, Carpenter AS, Glisson SR: Teleconference versus face-to-face scientific peer review of grant application: effects on review outcomes. PLoS One. 2013; 8(8): e71693. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Gallo SA, Schmaling KB, Thompson LA, et al.: Grant reviewer perceptions of the quality, effectiveness, and influence of panel.2020.

[9] 9. Vallée-Tourangeau G, Wheelock A, Vandrevala T, et al.: Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences. Humanit. Soc. Sci. Commun. 2022; 9(1): 1–11.

[10] 10. Gallo SA, Schmaling KB, Thompson LA, et al.: Grant review feedback: Appropriateness and usefulness. Sci. Eng. Ethics. 2021; 27: 1–20.

[11] 11. Abma-Schouten R, Gijbels J, Reijmerink W, et al.: Evaluation of research proposals by peer review panels: broader panels for broader assessments? Sci. Public Policy. 2023; 50(4): 619–632. Publisher Full Text

[12] 12. Gallo SA, Thompson LA, Schmaling KB, et al.: The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey. Sci. Eng. Ethics. 2020; 26(2): 761–782. PubMed Abstract | Publisher Full Text

[13] 13. Bol T, de Vaan M , van de Rijt A : The Matthew effect in science funding. Proc. Natl. Acad. Sci. 2018; 115(19): 4887–4890. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Shankar K, Luo J, Ma L, et al.: SPRING 2020 survey: peer review of grant proposals.Figshare.2021. [Last accessed 16/12/2024]]. Publisher Full Text

[15] 15. Feliciani T, Luo J, Shankar K: Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study. Quant. Sci. Stud. 2022; 3(3): 832–856. Publisher Full Text

[16] 16. Holbrook JB, Hrotic S: Blue skies, impacts, and peer review. RT A J. Res. Policy Eval. 2013; 1(1). Publisher Full Text

[17] 17. Obrecht M, Tibelius K, D'Aloisio G: Examining the value added by committee discussion in the review of applications for research awards. Res. Eval. 2007; 16(2): 70–91. Publisher Full Text

[18] 18. Schmaling KB, Evenson GR, Marble BK, et al.: Perceptions of grant peer reviewers: a mixed methods study. Res. Eval. 2024; 33. Publisher Full Text

[19] 19. Gallo S, Thompson L, Schmaling K, et al.: Risk evaluation in peer review of grant applications. Environ. Syst. Decis. 2018; 38(2): 216–229. Publisher Full Text

[20] 20. Guthrie S, Ghiga I, Wooding S: What do we know about grant peer review in the health sciences? F1000Res. 2017; 6: 6. Publisher Full Text

[21] 21. Feliciani T, Morreau M, Luo J, et al.: Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model. Res. Policy. 2022; 51(4): 104467. Publisher Full Text

[22] 22. Cristall N, Drozdowska BA, Fladt J, et al.: Priorities and expectations of researchers, funders, patients and the public regarding the funding of medical research: results from the PERSPECT qualitative study. BMJ open. 2024; 14(11): e084655. PubMed Abstract | Publisher Full Text | Free Full Text

[23] 23. Langfeldt L: Expert panels evaluating research: decision-making and sources of bias. Res. Eval. 2004; 13(1): 51–62. Publisher Full Text

[24] 24. Lee CJ: Commensuration Bias in Peer Review. Philos. Sci. 2015; 82(5): 1272–1283. Publisher Full Text

[25] 25. Langfeldt L: The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome. Soc. Stud. Sci. 2001; 31(6): 820–841. Publisher Full Text

[26] 26. Luo J, Feliciani T, Reinhart M, et al.: Analyzing sentiments in peer review reports: Evidence from two science funding agencies. Quant. Sci. Stud. 2021; 2(4): 1271–1295. Publisher Full Text

[27] 27. Pier EL, Brauer M, Filut A, et al.: Low agreement among reviewers evaluating the same NIH grant applications. Proc. Natl. Acad. Sci. 2018; 115(12): 2952–2957. Publisher Full Text

[28] 28. van den Besselaar P , Leydesdorff L: Past performance, peer review and project selection: a case study in the social and behavioral sciences. Res. Eval. 2009; 18(4): 273–288. Publisher Full Text

[29] 29. van den Besselaar P , Sandström U, Schiffbaenker H: Studying grant decision-making: a linguistic analysis of review reports. Scientometrics. 2018; 117(1): 313–329. PubMed Abstract | Publisher Full Text | Free Full Text

[30] 30. Fackrell K, Meadmore K, Recio-Saucedo A, et al.: Identification and comparison of key criteria of feedback of funding decisions: mixed-methods analysis of funder and applicant perspectives. BMJ Open. 2021; 11: e048979. PubMed Abstract | Publisher Full Text | Free Full Text

[31] 31. National Institute for Health and Care Research (NIHR): Guidance for NIHR Committee Member Development Scheme (CMDS) Participant applicants.2020. Reference Source

[32] 32. Bouter LM: Fostering responsible research practices is a shared responsibility of multiple stakeholders. J. Clin. Epidemiol. 2018; 96: 143–146. PubMed Abstract | Publisher Full Text

[33] 33. Munafò MR, Nosek BA, Bishop DVM, et al.: A manifesto for reproducible science. Nat. Hum. Behav. 2017; 1(1): 0021. PubMed Abstract | Publisher Full Text | Free Full Text

[34] 34. Guthrie S, Rincon DR, McInroy G, et al.: Measuring bias, burden and conservatism in research funding processes. F1000Research. 2019; 8(851): 851. Publisher Full Text

[35] 35. Wilsdon J: Are we all metascientists now? The shifting landscape for research on research.2024. [Last accessed: 16/12/2024]. Publisher Full Text

[36] 36. Ioannidis JPA: Meta-research: Why research on research matters. PLoS Biol. 2018; 16(3): e2005468. PubMed Abstract | Publisher Full Text | Free Full Text

[37] 37. Blatch-Jones A; Katie Meadmore, Cherish BoxallAsante E: Exploring Virtual Funding Committee Practices in the Allocation of National Institute for Health and Care Research Funding: A Netnographic Study. OSF. July 31 2024. Publisher Full Text

Role, function, and expectations of research funding committees: Perspectives from committee members

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Methods

Study design

Data collection

Data analysis

Results

Purpose of a funding committee

Table 1. Qualitative content analysis of responses on the purpose of a funding committee.

Expectation and role of a funding committee member

Table 2. Qualitative content analysis of responses on the expectation and role of a funding committee member.

Discussion

Strengths and limitations

Conclusion

Ethical considerations and consent

Data availability

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated