Strategic priorities and challenges in research software funding: Results from an international survey

Eric A. Jensen; Daniel S. Katz

doi:10.12688/f1000research.155879.2

Home Browse Strategic priorities and challenges in research software funding:...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

Strategic priorities and challenges in research software funding: Results from an international survey

[version 2; peer review: 2 approved, 2 approved with reservations, 1 not approved]

Eric A. Jensen ¹, Daniel S. Katz¹

PUBLISHED 06 Jan 2025

Author details Author details

¹ National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, IL, USA

Eric A. Jensen
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Daniel S. Katz
Roles: Conceptualization, Funding Acquisition, Investigation, Project Administration, Resources, Supervision, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Background

Research software is increasingly recognized as critical infrastructure in contemporary science. It spans a broad spectrum, including source code files, algorithms, scripts, computational workflows, and executables, all created for or during research. While research funders have developed programs, initiatives, and policies to bolster research software’s role, there has been no empirical study of how these funders prioritize support for research software. Understanding their priorities is essential to clarify where current support is concentrated and to identify strategic gaps.

Methods

We conducted an online mixed methods survey of international research funders (n=36) to explore their priorities in supporting research software. The survey gathered data on the specific outcomes funders emphasize in their programs and initiatives for research software.

Results

The survey revealed that funders place strong emphasis on developing skills, promoting software sustainability, embedding open science practices, building community and collaboration, advancing research software funding mechanisms, increasing software visibility and use, fostering innovation, and ensuring security.

Conclusions

The findings highlight opportunities to enhance research software’s role through increased funder attention on professional recognition for software contributions and the non-technical, social aspects of research software sustainability. Addressing these areas could lead to more effective support and development of research software, ultimately benefitting the entire research ecosystem.

Keywords

Research software, research policy, research funding, science systems, FAIR, open science

Corresponding author: Eric A. Jensen

Competing interests: No competing interests were disclosed.

Grant information: The Alfred P. Sloan Foundation funded this research via a grant to the National Center for Supercomputing Applications at the University of Illinois Urbana-Champaign for a project entitled “Charting the Course: Policy and Planning for Sustainable Research Software” (G-2022-19344).

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2025 Jensen EA and Katz DS. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Jensen EA and Katz DS. Strategic priorities and challenges in research software funding: Results from an international survey [version 2; peer review: 2 approved, 2 approved with reservations, 1 not approved]. F1000Research 2025, 13:1447 (https://doi.org/10.12688/f1000research.155879.2) First published: 29 Nov 2024, 13:1447 (https://doi.org/10.12688/f1000research.155879.1) Latest published: 06 Jan 2025, 13:1447 (https://doi.org/10.12688/f1000research.155879.2)

Revised Amendments from Version 1

Changes have been made primarily to the Methods section to add further detail on the study's limitations and context.

See the authors' detailed response to the review by Claire Jean-Quartier
See the authors' detailed response to the review by Mario COCCIA
See the authors' detailed response to the review by Adrian Barnett

Introduction

Research software is increasingly recognized as critical infrastructure in the contemporary research ecosystem (Barker et al., 2022; Jensen & Katz, 2023b; Knowles et al., 2021; Hocquet et al., 2024). Research software spans a broad spectrum, including source code files, algorithms, scripts, computational workflows, and executables, all created for or during research (Gruenpeter et al., 2021). With the growing reliance on computational methods across the sciences, supporting software development, maintenance, and impact is more essential than ever (e.g., Barker et al., 2020; Jay et al., 2021). For this reason, some research funders have developed programs, initiatives and policies to bolster research software’s role and impact (Hertweck et al., 2024; Strasser et al., 2022). The research software landscape includes government and philanthropic funding sources (Barker & Katz, 2022) that are seeking impact within the scientific ecosystem and beyond from their investments (e.g., Jensen et al., 2022). Research software is often open source and is sometimes tied to the larger open science movement that aims to improve transparency and reproducibility in research.

To date, there has been no empirical study of how software is supported by research funders, the aims of such support and the priorities that underpin it. This information is needed to clarify where current funder support is concentrated and highlight significant gaps. This study investigates the range of efforts by research funders to support the sustainability and impact of research software. The aim is for these findings to inform improvements in funder strategies that would bolster research software’s role and value for academic research and the wider society.

This paper draws on part of the dataset from this survey to address the following research question: What are the top priorities of international research funding organizations related to research software?

Methods

This research was carried out using a survey combining qualitative and quantitative items. The survey was designed to investigate how research software funders support research software’s sustainability and impact.

Survey design

The survey designed for this study began by collecting profile information, including institutional affiliation and job title. The survey gathered information about respondents’ organization’s initiatives, policies, or programs to support research software. The range of questions yielded too much data for one article. In this article, we focus exclusively on the results generated via an open-ended question asking about the top priorities for the respondents’ organizations’ support for research software: “What are your organization’s top priorities related to research software?” (see Jensen, 2024 for the extended data underpinning this article). Four open-response text boxes were provided for respondents to indicate and list these priorities.

Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.

Sampling

This survey was aimed at international research funders, including governmental and non-governmental (e.g., philanthropic) funders. A list of contacts to invite to participate in this survey was created based on participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors. ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding. This initial list of people was refined, with removals based on individuals having moved to unrelated professional roles or being unavailable long-term, for example, due to personal issues.

The final, refined contact list comprised 71 people. After removing individuals when a member of their organization already provided a complete answer or when the person turned out to no longer be working on a relevant topic or to be otherwise unavailable (total of n=30), 41 people remained. Five of these individuals did not complete the survey, while 36 people (representing 30 research funding organizations) did, yielding a response rate of 88%. Fully completed survey responses were not required for individuals to be retained in the sample, resulting in varied sample bases across survey questions.

The sample includes research funders in North and South America, Europe, Oceania and Asia, but over-represents North America and European funder representatives. Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.

Continent	Count
North America	15
South America	4
Europe	12
Oceania	3
Asia	1

The respondents represented research funders supported by governmental (n=26), philanthropic (n=6) and corporate (n=1) resources.

Respondents’ job titles span the following categories: Senior Leadership and Executive, such as a Vice President of Strategy; Program and Project Management, such as Senior Program Manager; Planning and Business Development; Scientific, Technical and IT, such as Scientific Information Lead.

Most respondents 72.7% (n=24) answered ‘Yes’ to the question, “Has your organization established any policies, initiatives or programs aimed at supporting research software?”, while 18.2% (n=6) said ‘No’ and 9.1% (n=3) ‘Unsure’.

Data collection, management and analysis

Data collection took place from December 2023 to May 2024. The mean completion time for the detailed survey was 28 minutes and 13 seconds.

The data were cleaned and prepared for analysis by removing any identifiable respondent details. The data analysis process followed a standard thematic qualitative analysis approach (e.g., Jensen & Laurie, 2016). This involved first identifying themes and organizing the data accordingly. Dimensions of each theme were identified where relevant. Then data extracts were selected from the survey responses associated with each theme and theme dimension. In line with qualitative research methodology, the focus is on presence or absence of ideas in the survey responses, not quantification. However, prevalence has been used to organize the results section, as the more prevalent themes also come with a more extensive set of data extracts and potential dimensions. The results section below presents each theme in turn, with data extracts providing evidence to support the description of the themes.

Results

The survey reveals several priorities driving research funders’ support for research software. These themes are presented below, ordered from the most to least prevalent categories. Many of these categories overlap, so the boundaries between themes are porous. After each point is explained below, a verbatim data extract in the form of a block quotation from the survey responses is presented to substantiate the explanation.

Skills

The first identified theme, Skills, shows funders are prioritizing equipping researchers and other research professionals with competencies in software development, maintenance, and sustainability best practices.

Supporting the training and retention of qualified staff to ensure human expertise and support within and between institutions.

Funders were keen to ensure sufficient skilled personnel capable of writing and maintaining software.

Skills - ensuring that there are enough skilled people writing software.

Such skill development was viewed as foundational for research software as a field.

Building the necessary skills, training, infrastructure, and incentives to support the growth of the [research software] field.

Respondents noted a priority focus on researchers’ software skills, seeking to support “train [ing] researchers to cope with research software on different levels.”

Training: We have an active initiative to train researchers to build software and digital research infrastructure.

However, international funder priorities also included software-related skills for other research professionals.

Invest in the training of both professional research and support staff to be able to reuse, develop, and maintain sustainable research software.

Overall, the funder representatives indicated that training and skills development support was an important pathway to advancing research software.

Software sustainability

Software Sustainability emerged as a significant priority for research software funders. Sustainability means “the software will continue to be available in the future, on new platforms, meeting new needs” (Katz, 2024). Indeed, the term sustainability was frequently used to answer this question about top priorities for research software funding, such as “supporting research software sustainability” or “identifying pathways to sustaining the research software.” In this section, we focus on non-financial aspects of software sustainability that respondents highlighted. Indeed, this priority was expressed in many ways.

Sustainability and reuse of software. How can we reduce legacy debt and ensure that research software is continuously improved and reused.

Reducing waste by maximizing the use of existing software was a notable focus evident in funders’ responses.

Fund research software sustainability, and in particular incentivize the reuse and improvement of existing research software by providing funding to improve existing software.

Some responses evinced the view that sustainability needed to be supported because it was part of ‘best practice’ in research software: “Promote the best practices for the production of sustainable research software.” Others focused on keeping software relevant and useful as a pathway to sustainability: “Support improvements to essential open source tools in [a specific field of] research”. Another response highlighted the importance of having standardized measures and practices to ensure sustainability.

Establish metrics and best practices for software sustainability and integrate these into [the] software development lifecycle.

In sum, the responses signaled funders’ full-throated commitment to ensuring research software’s long-term viability and continuous improvement (also see Barker et al., 2022).

Open science

The theme of Open Science highlights respondents’ emphasis on promoting open sharing, reuse, and accessibility of research software, adhering to the principles of open science. This theme is evident in the responses of research software funder representatives, who highlighted the importance of “open source practices” in fostering a collaborative and transparent research environment.

One respondent articulates the goal of promoting open sharing and reuse through significant initiatives, stating, “To promote the open sharing and reuse of research software, mainly through the European Open Science Cloud.” Many funder representatives highlighted maintaining, improving, and “making research software reusable” in their survey responses. This indicates a strategic effort to leverage established platforms to facilitate research software’s open distribution and reuse, thereby enhancing its accessibility and impact.

One respondent took an expansive view of what was included in efforts to integrate open science principles within research software: “Open Science including Accessibility, Inclusion, Reproducibility, Recognition, and security.” Another respondent called for a more explicit integration of research software into open science policies, saying, “Include research software more explicitly in our open science policies.” This points to the need for clear policy frameworks that explicitly address the role of research software in open science, ensuring that it is systematically included and supported (e.g., see Jiménez et al., 2017). This set of responses reflects a commitment to fostering an environment where a wide range of users and developers can contribute to and benefit from open-source research software.

Building community and collaboration

The theme of building community and collaboration among research software funders encompassed three main dimensions: Enhancing community engagement, fostering international collaboration, and sharing best practices through professional networks. These community-related priorities were ultimately aimed at spreading and embedding good practices in research software.

The emphasis on community engagement highlights the importance of involving diverse groups in developing and maintaining scientific software. Funders prioritize approaches that ensure high-quality software through community involvement. They seek to promote opportunities that engage new communities in software development, making these resources accessible to under-represented communities and broadening participation in data science.

Promote opportunities to engage new communities in software development and make these resources accessible to under-represented communities interested in data science.

Another respondent emphasizes the importance of inclusivity in open source communities of practice:

Supporting diverse participants (users and developers) in scientific open source (research) software.

Additionally, there is a focus on community-focused software development and dissemination, ensuring that the processes and outcomes of software projects align with the community’s needs and contributions.

Enhance community-focused software development and dissemination.

International collaboration was identified as another critical dimension, focusing on creating a global network for software production. Funders aimed to facilitate work across institutions to prevent duplication of efforts and ensure that lesser-resourced institutions can equally benefit from open-source software, whether it be code, databases, or extensive infrastructure. This collaborative approach leveraged collective expertise and resources, promoting efficiency and innovation.

Enabling work across institutions so that there is no duplication of effort and lesser-resourced institutions can benefit from OSS (whether bits of code, databases, or huge infrastructure) as much as the larger institutions.

Moreover, fostering international collaboration is seen as a way to enhance the production of research software by tapping into a diverse pool of global expertise.

Foster international collaboration in the production of research software.

Funders prioritized disseminating best practices as a means of building robust communities around software development. Respondents emphasized the need for networks to share successful strategies and methodologies, ensuring that effective practices are widely adopted. This was aimed at contributing to improving research software quality and community development.

Best practice and community - ensuring that there are networks to disseminate best practice and build communities.

In addition to promoting best practices, there is a focus on developing community-based approaches for ensuring and improving the quality of scientific software.

Development of community-based approaches for ensuring and improving the quality of scientific software and code.

This response shows that funders linked community-building goals with practical concerns about continuous improvement and quality assurance of research software.

Advancing research software funding

The theme of advancing research software funding encompasses “funding new research software”, strategic planning of funding support, and fostering collaborative efforts to improve software impact.

Funders noted the need for financial resources to support research software, for example, by “creating instruments to fund the production of high-quality research software.” They also emphasized the importance of developing plans to guide funding decisions over the medium term.

Our primary priority is developing a strategy for how to fund [research software for the] next [several years].

Such a forward-looking approach is undoubtedly challenging in the constantly evolving research software landscape.

Collaboration also emerged as a funder priority, with a systemic focus.

Fund joint developments of research software useful for the [research] ecosystem in collaboration with other institutions.

Overall, the theme of advancing research software funding highlighted funders’ commitment to creating strategic funding for research software.

Increasing software visibility and use

Funders highlighted the need to make existing research software more visible to a broader range of researchers.

Provide visibility to successful computational projects from users of our platform, not only about the science but also about the software being developed.

Additionally, there was a focus on enabling existing software to facilitate wider adoption and make it more accessible and usable.

Make existing research software more visible [and] shape some of that software for wider adoption. [emphasis added]

One funder noted the need for a centralized research software portal where researchers could access information on supported platforms, tools, and services.

Developing a comprehensive catalog of currently supported [research software] platforms, tools and services that researchers can access via [a] single portal, including the cloud resources needed to run those services.

Another funder’s related suggestion also focused on bolstering the visibility of research software through a shared portal or database.

Develop a profile of funding and support programs for [research software], with a focus on those infrastructures that have a national and international impact, but also including emerging RS platforms that have not yet established their role/impact.

The aim of increasing the visibility of research software in this way was “encouraging the [scientific] field to use research software.” Ultimately, these efforts aim to enhance the overall impact of research software.

Innovation

A small number of funders noted prioritizing integrating advanced technologies in software development. One funder noted the importance of “transitioning to accelerated compute” and maintaining pace with advancements in large-scale computing resources.

Cutting edge - ensuring that software is developed to keep pace with the next generation of large-scale compute and supports the best research.

A specific ‘innovation’ example mentioned was the “use of Generative AI in software engineering.” Such priorities reflected a forward-looking approach from a small subset of funders aiming to ensure that research software continues to support research effectively.

Security

Funders raised the theme of “security” in the development and use of research software, including ensuring “data security,” enhancing software security, and conducting “Security Awareness Training.”

Security: Research software will not be able to be reused if IT departments are not confident that it is secure[,] even for Open science projects.

While infrequently mentioned compared to other themes and with limited detail in the responses, security was seen as a fundamental issue to be addressed for a small number of research software funders.

Discussion

In this paper, we have described the priorities guiding research software funders. Here, we consider the implications of these priorities for advancing research software’s role in the scientific ecosystem.

Within the skills development priorities highlighted by respondents, several technical training topics to support the design, development and deployment of research software were mentioned. This funder priority aligns with an identified need for training in research software skills (e.g., Carver et al., 2022; Cosden et al., 2022). Moreover, this skills theme suggests that there is recognition among funders that there is a need to invest in the people who underpin research software (e.g., Hartley & Barker, 2023; Katz, 2021). However, the more social aspects of research software sustainability and impact were not explicitly mentioned by any respondents. Addressing such social aspects and non-technical skills is an essential need in the field. For example, research software projects need to be able to identify relevant parties to engage with, maintain effective communication with current and potential users, establish a robust framework for monitoring and addressing evolving needs in a user community, and ensure good user experiences with the software. A recently published report highlighted “UX design, product management, and community management” as examples of key roles that often go unfunded in open source research software projects (Iacovou, 2024). Iacovou (2024) noted that this averred gap in available support was part of a broader need to address the critical social and community aspects of research software sustainability. While respondents in our survey explicitly noted community building and management, supporting the development of the skills underpinning this kind of work was not highlighted as a priority for research software funders.

Software sustainability emerged as a prominent theme in responses by research software funders. Software sustainability is a complex and, in some ways, an irresolvable challenge due to ever-shifting contexts on the broader research ecosystem (e.g., Howison, 2020; Katz, 2024). However, there is clearly a need for funders to address not only the technical aspects of software sustainability but also the social and professional infrastructure that bolsters long-term sustainability (e.g., Carver et al., 2021).

The theme of Open Science in the survey responses emphasized promoting open sharing, reuse, and accessibility of research software. This is achieved through strategic initiatives, including open science principles, explicit policy integration, support for diverse participants, and promoting open source practices. For example, funding open-source tools has been a major focus in the Chan Zuckerberg Initiative’s Essential Open Source Software for Science (EOSS) program, which has focused on software for biomedical research. A recent report on the EOSS program highlighted wide-ranging benefits from this investment in terms of scientific progress and diversifying scientific open source software teams and communities (Hertweck et al., 2024). This suggests good alignment between funder priorities and the evidence base about what is yielding practical scientific and broader impacts.

Different practical ways of ensuring that research software contributes to such a healthy and open ecosystem are being discussed in the research software field (e.g., McKiernan et al., 2023; Sonabend et al., 2024; Sellanga et al., 2024). For example, applying FAIR (Findable, Accessible, Interoperable and Reusable) principles to research software has been highlighted as an essential step in the literature (Barker et al., 2022). While our respondents mentioned concepts such as ‘accessibility’ and ‘reuse’, the term FAIR was not explicitly used. Overall, the survey responses evince a commitment to fostering an open, inclusive, and transparent research ecosystem. However, it is worth acknowledging that the focus on open source may be exaggerated in our purposive sample of funders already active in the research software arena, which is strongly oriented towards open source.

The theme of building community and collaboration was defined by efforts to enhance community engagement, foster international partnerships, and disseminate best practices. Advancing research software funding was also highlighted as a theme, with priorities including backing new software, strategic financial planning, and fostering collaborations. In these responses, funders emphasized the need for resources, strategic guidance, and joint efforts to enhance software impact. Promoting the visibility and use of existing software also emerged as a priority for research software funders. This included an interest in bolstering the awareness of available software by making it more prominent, cataloging and profiling available software resources, and encouraging software adoption. It has previously been noted that research software visibility can be bolstered through institutions that help to build community and collaboration, such as software sustainability institutes (Katz et al., 2021).

A notable gap in the priorities identified by research software funders is the thorny challenge of bolstering recognition of research software contributions in academic hiring, promotion, and tenure (Jensen & Katz, 2023a). This may appear to be outside the purview of research software funders, but such ‘people’ aspects are a clear priority for those who produce and maintain research software (e.g., Cohen et al., 2021; Lamprecht et al., 2022). A recent article summarized ‘ten simple rules’ for addressing this challenge, including ensuring that software contributions are recognized as key scholarly contributions, consistent with open science principles (Puebla et al., 2024). While all of these rules/recommendations focus on the level of research institutions (Barker et al., 2024), research funders can play a pivotal role in nudging the priorities and actions of such institutions. Therefore, research software funders should consider how they could help encourage research institutions to adopt a software-inclusive stance within hiring, promotion and tenure.

Much less frequently mentioned themes were innovation in research software, stressing the need to stay ahead of rapidly shifting trends in computing and the need to ensure that research software is secure from threats. Adapting research software good practices to advancements in AI and related developments is a need that has also been highlighted in the literature (e.g., Duarte et al., 2023), and it is an ongoing challenge facing the research software field. The theme of security in research software development and use was defined by a strong focus on ensuring data security, enhancing software security, and promoting security awareness training.

Conclusion

The insights from research software funders reveal a commitment to enhancing the scientific landscape through targeted investments in software sustainability, open science, and community collaboration. Yet, the relative underemphasis on non-technical skills and social infrastructure suggests a need for a more holistic approach. To fully realize the potential of research software, funders must broaden their focus, integrating social and technical priorities to foster a resilient and dynamic research environment. By addressing such gaps, funders can help ensure that research software plays a more efficient, inclusive, sustainable, and impactful role within the scientific ecosystem.

Ethics and consent

The University of Illinois Urbana-Champaign Institutional Review Board reviewed this research and determined it to be exempt (no. 24374) on September 11, 2023. All participants provided explicit informed written consent to participate in this study at the start of the online form used to gather the survey data. This included consent for data to be used for research purposes and to be published as open data after deidentification.

Data availability

Underlying data

Repository name: Zenodo

Title of project: Strategic priorities and challenges in research software funding: Results from an international survey [Dataset]. DOI: https://doi.org/10.5281/zenodo.13852004 (Jensen, 2024)

This project contains the following underlying data:

• Survey data file: Anonymized responses from the international survey on strategic priorities and challenges in research software funding gathered from representatives of research funding organizations.

Details of license: The anonymized survey data are available under the Creative Commons Attribution Non-Commercial No Derivatives 4.0 International license.

Extended data

Repository name: Zenodo

Title of project: Strategic priorities and challenges in research software funding: Results from an international survey. DOI: https://doi.org/10.5281/zenodo.13852004 (Jensen, 2024)

This project contains the following extended data:

• Consent block from survey: The information and questions used to gather informed consent through the online survey have been uploaded as a document file.
• SURVEY DESIGN_Research_software_funder_survey_UIUC (upload to Zenodo).pdf

Details of license: The anonymized survey data are available under the Creative Commons Attribution Non-Commercial No Derivatives 4.0 International license.

Acknowledgements

The Alfred P. Sloan Foundation funded this research via a grant to the National Center for Supercomputing Applications at the University of Illinois Urbana-Champaign for a project entitled “Charting the Course: Policy and Planning for Sustainable Research Software” (G-2022-19344). This project aims to devise strategies for enhancing the durability and sustainability of research software by bolstering the role of policy at various levels within the research ecosystem. The authors would also like to express appreciation for feedback from Professor Caroline Jay (University of Manchester).

References

Barker M, Chue Hong NP, Katz DS, et al.: Introducing the FAIR principles for research software. Sci. Data. 2022; 9: Article 622. PubMed Abstract | Publisher Full Text | Free Full Text
Barker M, Katz DS: Overview of research software funding landscape. Zenodo. 2022. Publisher Full Text
Barker M, Katz DS, Gonzalez-Beltran A: Evidence for the importance of research software. Zenodo. 2020. Publisher Full Text
Barker M, Carlin D, Cohen J, et al.: Resources for supporting policy change in research institutions in practice: A report from Subgroup 2 of the ReSA & RDA Policies in Research Organisations for Research Software (PRO4RS) Working Group. Zenodo. 2024. Publisher Full Text
Carver JC, Cosden IA, Hill C, et al.: Sustaining research software via research software engineers and professional associations. Proceedings of the 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS’21). IEEE; 2021; pp. 69–72. Publisher Full Text
Carver JC, Weber N, Ram K, et al.: A survey of the state of the practice for research software in the United States. PeerJ. Comput. Sci. 2022; 8: e963. PubMed Abstract | Publisher Full Text | Free Full Text
Cohen J, Katz DS, Barker M, et al.: The four pillars of research software engineering. IEEE Softw. 2021; 38(1): 97–105. Publisher Full Text
Cosden IA, McHenry K, Katz DS: Research software engineers: Career entry points and training gaps. Comput. Sci. Eng. 2022; 24(6): 14–21. Publisher Full Text
Duarte J, Li H, Roy A, et al.: FAIR AI models in high energy physics. Mach. Learn.: Sci. Technol. 2023; 4(4): 045062. Publisher Full Text
Gruenpeter M, Katz DS, Lamprecht A-L, et al.: Defining research software: A controversial discussion. Zenodo. 2021. Publisher Full Text
Hartley K, Barker M: Investing in people: Anticipating the future of research software. Research Software Alliance Blog. 2023, October. Reference Source
Hertweck K, Strasser C, Taraborelli D: Insights and impact from five cycles of Essential Open Source Software for Science. Zenodo. 2024. Publisher Full Text
Hocquet A, Wieber F, Gramelsberger G, et al.: Software in science is ubiquitous yet overlooked. Nat. Comput. Sci. 2024; 4: 465–468. PubMed Abstract | Publisher Full Text
Howison J: Sustainability in scientific software: Ecosystem complexity and software visibility. Zenodo. 2020. Publisher Full Text
Iacovou G: Funding open source science software. Notion. 2024, June. Retrieved November 26, 2024. Reference Source
Jay C, Haines R, Katz DS: Software must be recognised as an important output of scholarly research. Int. J. Digit. Curation. 2021; 16(1). Publisher Full Text
Jensen EA: Strategic priorities and challenges in research software funding: Results from an international survey. (Dataset). Zenodo. 2024. Publisher Full Text
Jensen EA, Katz DS: From code to tenure: Valuing research software in academia. Commonplace. 2023a. Publisher Full Text
Jensen EA, Katz DS: From backstage to spotlight: A call to highlight the critical role of research software. LSE Impact Blog. 2023b. Publisher Full Text
Jensen EA, Laurie C: Doing real research: A practical guide to social research. SAGE; 2016.
Jensen EA, Wong P, Reed MS: How research data deliver non-academic impacts: A secondary analysis of UK Research Excellence Framework impact case studies. PLoS One. 2022; 17(3): e0264914. PubMed Abstract | Publisher Full Text | Free Full Text
Jiménez RC, Kuzak M, Alhamdoosh M, et al.: Four simple recommendations to encourage best practices in research software. F1000Res. 2017; 6: 876. PubMed Abstract | Publisher Full Text | Free Full Text
Katz DS: Valuing people vs hardware. Daniel S. Katz’s Blog. 2021. Publisher Full Text
Katz DS: There’s no such thing as sustainable research software. Daniel S. Katz’s Blog. 2024. Publisher Full Text
Katz DS, Carver J, Chue Hong N, et al.: Addressing research software sustainability via institutes. Proceedings of the 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS’21). IEEE; 2021; pp. 13–16. Publisher Full Text
Knowles R, Mateen BA, Yehudi Y: We need to talk about the lack of investment in digital research infrastructure. Nat. Comput. Sci. 2021; 1: 169–171. PubMed Abstract | Publisher Full Text
Lamprecht A-L, Martinez-Ortiz C, Barker M, et al.: What do we (not) know about research software engineering? Journal of Open Research Software. 2022; 10(1): Article 11. Publisher Full Text
McKiernan EC, Barba LA, Bourne PE, et al.: Policy recommendations to ensure that research software is openly accessible and reusable. PLoS Biol. 2023; 21(5): e3002204. PubMed Abstract | Publisher Full Text | Free Full Text
Puebla I, Ascoli GA, Blume J, et al.: Ten simple rules for recognizing data and software contributions in hiring, promotion, and tenure. PLoS Comput. Biol. 2024; 20(8): e1012296. PubMed Abstract | Publisher Full Text | Free Full Text
Sellanga G, Steinhart E, Tsang N, et al.: 2024 survey of recent open science policy developments. Invest in Open Infrastructure. 2024. Publisher Full Text
Sonabend R, Gruson H, Wolansky L, et al.: FAIR-USE4OS: Guidelines for creating impactful open-source software. PLoS Comput. Biol. 2024; 20(5): e1012045. PubMed Abstract | Publisher Full Text | Free Full Text
Strasser C, Hertweck K, Greenberg J, et al.: Ten simple rules for funding scientific open source software. PLoS Comput. Biol. 2022; 18(11): e1010627. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 29 Nov 2024

Author details Author details

¹ National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, IL, USA

Eric A. Jensen
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Daniel S. Katz
Roles: Conceptualization, Funding Acquisition, Investigation, Project Administration, Resources, Supervision, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

Article Versions (2)

version 2

Revised

Published: 06 Jan 2025, 13:1447

https://doi.org/10.12688/f1000research.155879.2

version 1

Published: 29 Nov 2024, 13:1447

https://doi.org/10.12688/f1000research.155879.1

© 2025 Jensen EA and Katz DS. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Jensen EA and Katz DS. Strategic priorities and challenges in research software funding: Results from an international survey [version 2; peer review: 2 approved, 2 approved with reservations, 1 not approved]. F1000Research 2025, 13:1447 (https://doi.org/10.12688/f1000research.155879.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 2

VERSION 2

PUBLISHED 06 Jan 2025

Revised

Views

Reviewer Report 06 Feb 2025

Rena Bakhshi, Netherlands eScience Center, Amsterdam, The Netherlands

Approved with Reservations

https://doi.org/10.5256/f1000research.176306.r358849

The current paper presents the results of a survey among research funders focusing on research software. In particular, the focus of the paper is around funders responses to the question "What are your organization’s top priorities related to research software?” The authors employ a qualitative research methodology for their analysis, and extract topics/themes from the survey results.

While the topic and this empirical study are interesting, the paper is somewhat underwhelming as a "research article". Beyond reporting of the dataset (survey results), it is unclear to me how this paper is addressing the research question posed in the introduction. The paper simply reports on the opinions of the respondents. What is missing is a deeper analysis, for instance, how these opinions relate to the policies of these organizations. For example, the authors write that "in sum, the responses signaled funders’ full-throated commitment to ensuring research software’s long-term viability and continuous improvement " but do not elaborate on relevant programs of those funders.

The results section can be summarized in the form of a table to help readability.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: research software engineering, project management, program management, computer science (formal methods)

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 05 Feb 2025

Fabio Kon, University of São Paulo, São Paulo, Brazil

Approved

https://doi.org/10.5256/f1000research.176306.r356589

This is an interesting paper showing the priorities of research funders concerning funding research software. The method is sound, and the anonymized data is presented in a public repository. The contribution is small but relevant and can help both funders engage in RS funding. It can also guide researchers when seeking funding to develop RS.

The set of survey respondents is biased because it started with a list from the Research Software Alliance, but this is clearly stated in the paper.

Thus, I believe the paper should be indexed in F1000research, but I suggest the following changes before indexing.

In the abstract, Results paragraph, the view is overly optimistic. Most funders probably don't value all these things; you got those skewed results because you used the ReSA list of contacts, as you correctly explain in Methods-Sampling. Somehow, this explanation should be reflected in the abstract, too.

In the Methods-Survey design, it would be important to emphasize graphically the question "What are your organization.... software?"; e.g., by putting it in a box.

In the Results section, you wrote that verbatim data extract in the form of a block quotation is shown. But, in fact, the reader can't see any block quotation, instead, a red text is displayed. I guess you might be referring to the latex command quotation that's obviously not seen by the reader. Probably, a \fbox command would work better.

In the Results section, it would be interesting to know how often the topic was mentioned and/or how many respondents mentioned it.

"the responses signaled funders' full-throated commitment to...". In fact, this doesn't show they're committed. It shows that they say this is something they look for in projects to be funded.

In the discussion, I suggest revising the paragraph about Open Science as there's little connection between the beginning of the paragraph ("the theme of open science") and the example concerning EOSS. Funding OSS is a small part of Open Science in general, which that paragraph does not capture well.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Software Engineering, Data Science, Distributed Systems

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 25 Jan 2025

Claire Jean-Quartier, Research Data Management, Graz University of Technology, Graz, Austria

Approved with Reservations

https://doi.org/10.5256/f1000research.176306.r358851

Authors present qualitative survey results on research software funding focusing on related priorities by the funders. Research software has and is often neglected in publications, and one would guess that funders do not focus on or require researchers to plan for software produced and used during research projects to be sustainable. Still, this key aspect remains an open question and is addressed by the herein presented results from a qualitative survey.

Though, it would be highly interesting to read about all results from the survey, it makes sense to split the more extensive survey and make use of the subsection regarding funders’ priorities in regard to research software, as has been done by the authors.

The presented manuscript could be improved in terms of scientific completeness and comprehensibility, therefore suggesting the following major points:

(1) The manuscript seems to lack content in regard to the scientific background. It is recommended to extend the introduction section towards a more wholistic setting describing and reviewing software sustainability factors in general on the one side and general funders' priorities on the other side.

(2) The results section could be made more comprehensible, since the so far used structural layout of text only, divided by numerous short paragraphs, does hinder readability. Results could be summarized in a table presenting key aspects, or something similar could be introduced to increase structural clarity and readability!

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: research data management, open science, bioinformatics, cancer science

CITE

Report a concern

Author Response 14 Apr 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

14 Apr 2025

Author Response
- We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.
- The
... Continue reading
We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.

The short paragraph structure is standard for qualitative research results write-ups. However, we appreciate the suggestion of adding a table that presents key aspects of the findings. This has now been added in Version 3 (Table 1).
We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.

The short paragraph structure is standard for qualitative research results write-ups. However, we appreciate the suggestion of adding a table that presents key aspects of the findings. This has now been added in Version 3 (Table 1).
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 14 Apr 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

14 Apr 2025

Author Response
- We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.
- The
... Continue reading
We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.

The short paragraph structure is standard for qualitative research results write-ups. However, we appreciate the suggestion of adding a table that presents key aspects of the findings. This has now been added in Version 3 (Table 1).
We appreciate your suggestion of extending the introduction to provide a more expansive account of the relevant background information. This section has now been significantly expanded.

The short paragraph structure is standard for qualitative research results write-ups. However, we appreciate the suggestion of adding a table that presents key aspects of the findings. This has now been added in Version 3 (Table 1).
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 22 Jan 2025

Mario COCCIA, DSU, National Research Council Department of Engineering ICT and Technology for Energy and Transport, Rome, Lazio, Italy

Not Approved

https://doi.org/10.5256/f1000research.176306.r356373

Authors have partially clarified ... Continue reading

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 07 Jan 2025

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

Approved

https://doi.org/10.5256/f1000research.176306.r356372

No ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 29 Nov 2024

Views

Reviewer Report 13 Dec 2024

Mario COCCIA, DSU, National Research Council Department of Engineering ICT and Technology for Energy and Transport, Rome, Lazio, Italy

Approved with Reservations

https://doi.org/10.5256/f1000research.171109.r344621

Strategic priorities and challenges in research software funding: Results from an international survey

The topics of this paper are interesting, though well known. The structure and content must be revised, and results have to be better explained by authors before to be reconsidered.

Title has to be shorter and more specific of the fields concerning funding research software.

Abstract has to clarify the goal, methods, empirical research and research policies to support funding of software.

Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science, etc. After that they can focus on the topics of this study to provide a correct analysis for fruitful discussion (See suggested readings that must be all read and used in the text).

Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections:
•   Sample and data. The sample is not balanced between continents and also respondents and authors have to clarify how this is representative.
•   Measures of variables under study
•   Data analysis procedure.
Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.... now the analysis is rather vague.

Results.
The sample can be represented with some figures to be clearer for readers.
Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.

Discussion.
First, authors have to synthesize the main results in a simple table to be clear for readers and then show what this study adds compared to other studies sone on these topics.
To reiterate, avoiding in the just mentioned sections, sub-headings that create fragmentation of the paper.
The results should be better discussed considering the research fields because the sources of research software, their goal, their adopters change according to the research fields, such as biomedical research, environmental sciences, economic sciences, etc....role of research software, funding and citations for diffusion of open science should be useful to decide specific policies. A SWOT matrix with pros and cons can support managerial implications.

Conclusion has not to be a summary, but authors have to focus on manifold limitations of this study and based on results, provide best practices of research policies to support the development of research software having practical implications for the diffusion of knowledge (also with citations) and development of science.

Overall, then, the paper is interesting, but theoretical framework is weak, and some results, too descriptive, create confusion… structure of the paper has to be improved; study design, discussion and presentation of results have to be clarified using suggested comments.

I strongly suggest improving the paper by using all comments (suggested papers included reading and using all) and maybe it can be considered.

Suggested readings.

Afiaz, A., Ivanov, A.A., Chamberlin, J., ... Leek, J.T., Wright, C. 2024 (Ref -1). Best practices to evaluate the impact of biomedical research software-metric collection beyond citations, Bioinformatics, 40(8), btae469

Sochat, V., May, N., Cosden, I., Martinez-Ortiz, C., Bartholomew, S.   2022 (Ref-2). The Research Software Encyclopedia: A Community Framework to Define Research Software.Journal of Open Research Software, 10(1)

Coccia M. 2017. Sources of technological innovation: Radical and incremental innovation problem-driven to support competitive advantage of firms. Technology Analysis & Strategic Management, vol. 29, n. 9, pp. 1048-1061, https://doi.org/10.1080/09537325.2016.1268682\

Cosden, I.A. 2022 (Ref-3). ,The Princeton University Research Software Engineering Group Model: Operational and Organizational Approaches.Computing in Science and Engineering, 24(5), pp. 24–31

Coccia M., Roshani S. 2024 (Ref-4). General laws of funding for scientific citations: how citations change in funded and unfunded research between basic and applied sciences. Journal of Data and Information Science, 9(1), 1–18. https://doi.org/10.2478/jdis-2024-0005

Gomez-Diaz, T., Recio, T. 2022 (Ref-5). Research Software vs. Research Data II: Protocols for Research Data dissemination and evaluation in the Open Science context.F1000Research, 11, 117

Roshani S., Bagheri R., Mosleh M., Coccia M. 2021 (Ref-6). What is the relationship between research funding and citation-based performance? A comparative analysis between critical disciplines. Scientometrics 126, 7859–7874. https://doi.org/10.1007/s11192-021-04077-9

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Afiaz A, Ivanov AA, Chamberlin J, Hanauer D, et al.: Best practices to evaluate the impact of biomedical research software-metric collection beyond citations.Bioinformatics. 2024; 40 (8). PubMed Abstract | Publisher Full Text
2. Sochat V, May N, Cosden I, Martinez-Ortiz C, et al.: The Research Software Encyclopedia: A Community Framework to Define Research Software. Journal of Open Research Software. 2022; 10 (1). Publisher Full Text
3. Cosden I: The Princeton University Research Software Engineering Group Model: Operational and Organizational Approaches. Computing in Science & Engineering. 2022; 24 (5): 24-31 Publisher Full Text
4. Coccia M, Roshani S: General laws of funding for scientific citations: how citations change in funded and unfunded research between basic and applied sciences. Journal of Data and Information Science. 2024; 9 (4): 71-89 Publisher Full Text
5. Gomez-Diaz T, Recio T: Research Software vs. Research Data II: Protocols for Research Data dissemination and evaluation in the Open Science context.F1000Res. 2022; 11: 117 PubMed Abstract | Publisher Full Text
6. Roshani S, Bagherylooieh M, Mosleh M, Coccia M: What is the relationship between research funding and citation-based performance? A comparative analysis between critical disciplines. Scientometrics. 2021; 126 (9): 7859-7874 Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: science development and policies

CITE

Report a concern

Author Response 06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

06 Jan 2025

Author Response

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with ... Continue reading We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with all points presented in this reviewer report, we have taken on board several points and made updates to the manuscript.

Reviewer 2: “Title has to be shorter and more specific of the fields concerning funding research software.”

Response: The relevant research fields for the participating research funders was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” Because this includes all research fields, it wouldn’t be appropriate to claim a narrower scope in the title.

Reviewer 2: The reviewer makes several requests for a wider paper scope, e.g.: “Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science.”

Response: These issues are beyond the scope of the current paper, which focuses on open-ended survey data from research funders. Imposing focus areas that did not emerge from these open-ended responses would be methodologically problematic. The research question currently stated in the manuscript is deliberately broad in line with the qualitative nature of the data that is being presented in the paper: “What are the top priorities of international research funding organizations related to research software?”

Reviewer 2: “Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections: […] Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.”
Response: The current Methods section structure follows the journal guidelines and is appropriate for a qualitative survey study like this. A purely qualitative analysis and reporting of results is permitted by this journal; adding quantitative methods is not required.

Reviewer 2: “Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.”

Response: We appreciate the reviewers ideas about how to present the results. However, the current structure is in line with accepted practices in qualitative research. Each subheading represents a distinct theme, so it would be inappropriate to force them together.

Reviewer 2: Raises concerns about the Discussion section structure

Response: While the reviewer’s feedback about the Discussion section is noted, it is presented in a manner consistent with the journal guidelines and normal social science practices. Adding a table, for example, would be out of line with typical Discussion section approaches for this kind of paper.

Suggested readings: Because none of the suggested readings directly relate the research software funding, these papers have not been added to the citation list despite the reviewer’s insistence on adding citations to the reviewer’s published work.
We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with all points presented in this reviewer report, we have taken on board several points and made updates to the manuscript.

Reviewer 2: “Title has to be shorter and more specific of the fields concerning funding research software.”

Response: The relevant research fields for the participating research funders was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” Because this includes all research fields, it wouldn’t be appropriate to claim a narrower scope in the title.

Reviewer 2: The reviewer makes several requests for a wider paper scope, e.g.: “Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science.”

Response: These issues are beyond the scope of the current paper, which focuses on open-ended survey data from research funders. Imposing focus areas that did not emerge from these open-ended responses would be methodologically problematic. The research question currently stated in the manuscript is deliberately broad in line with the qualitative nature of the data that is being presented in the paper: “What are the top priorities of international research funding organizations related to research software?”

Reviewer 2: “Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections: […] Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.”
Response: The current Methods section structure follows the journal guidelines and is appropriate for a qualitative survey study like this. A purely qualitative analysis and reporting of results is permitted by this journal; adding quantitative methods is not required.

Reviewer 2: “Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.”

Response: We appreciate the reviewers ideas about how to present the results. However, the current structure is in line with accepted practices in qualitative research. Each subheading represents a distinct theme, so it would be inappropriate to force them together.

Reviewer 2: Raises concerns about the Discussion section structure

Response: While the reviewer’s feedback about the Discussion section is noted, it is presented in a manner consistent with the journal guidelines and normal social science practices. Adding a table, for example, would be out of line with typical Discussion section approaches for this kind of paper.

Suggested readings: Because none of the suggested readings directly relate the research software funding, these papers have not been added to the citation list despite the reviewer’s insistence on adding citations to the reviewer’s published work.
Competing Interests: I am the first author of the manuscript that this review report addresses. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

06 Jan 2025

Author Response

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with ... Continue reading We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with all points presented in this reviewer report, we have taken on board several points and made updates to the manuscript.

Reviewer 2: “Title has to be shorter and more specific of the fields concerning funding research software.”

Response: The relevant research fields for the participating research funders was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” Because this includes all research fields, it wouldn’t be appropriate to claim a narrower scope in the title.

Reviewer 2: The reviewer makes several requests for a wider paper scope, e.g.: “Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science.”

Response: These issues are beyond the scope of the current paper, which focuses on open-ended survey data from research funders. Imposing focus areas that did not emerge from these open-ended responses would be methodologically problematic. The research question currently stated in the manuscript is deliberately broad in line with the qualitative nature of the data that is being presented in the paper: “What are the top priorities of international research funding organizations related to research software?”

Reviewer 2: “Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections: […] Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.”
Response: The current Methods section structure follows the journal guidelines and is appropriate for a qualitative survey study like this. A purely qualitative analysis and reporting of results is permitted by this journal; adding quantitative methods is not required.

Reviewer 2: “Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.”

Response: We appreciate the reviewers ideas about how to present the results. However, the current structure is in line with accepted practices in qualitative research. Each subheading represents a distinct theme, so it would be inappropriate to force them together.

Reviewer 2: Raises concerns about the Discussion section structure

Response: While the reviewer’s feedback about the Discussion section is noted, it is presented in a manner consistent with the journal guidelines and normal social science practices. Adding a table, for example, would be out of line with typical Discussion section approaches for this kind of paper.

Suggested readings: Because none of the suggested readings directly relate the research software funding, these papers have not been added to the citation list despite the reviewer’s insistence on adding citations to the reviewer’s published work.
We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with all points presented in this reviewer report, we have taken on board several points and made updates to the manuscript.

Reviewer 2: “Title has to be shorter and more specific of the fields concerning funding research software.”

Response: The relevant research fields for the participating research funders was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” Because this includes all research fields, it wouldn’t be appropriate to claim a narrower scope in the title.

Reviewer 2: The reviewer makes several requests for a wider paper scope, e.g.: “Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science.”

Response: These issues are beyond the scope of the current paper, which focuses on open-ended survey data from research funders. Imposing focus areas that did not emerge from these open-ended responses would be methodologically problematic. The research question currently stated in the manuscript is deliberately broad in line with the qualitative nature of the data that is being presented in the paper: “What are the top priorities of international research funding organizations related to research software?”

Reviewer 2: “Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections: […] Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.”
Response: The current Methods section structure follows the journal guidelines and is appropriate for a qualitative survey study like this. A purely qualitative analysis and reporting of results is permitted by this journal; adding quantitative methods is not required.

Reviewer 2: “Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.”

Response: We appreciate the reviewers ideas about how to present the results. However, the current structure is in line with accepted practices in qualitative research. Each subheading represents a distinct theme, so it would be inappropriate to force them together.

Reviewer 2: Raises concerns about the Discussion section structure

Response: While the reviewer’s feedback about the Discussion section is noted, it is presented in a manner consistent with the journal guidelines and normal social science practices. Adding a table, for example, would be out of line with typical Discussion section approaches for this kind of paper.

Suggested readings: Because none of the suggested readings directly relate the research software funding, these papers have not been added to the citation list despite the reviewer’s insistence on adding citations to the reviewer’s published work.
Competing Interests: I am the first author of the manuscript that this review report addresses. Close
Report a concern

Views

Reviewer Report 13 Dec 2024

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

Approved with Reservations

https://doi.org/10.5256/f1000research.171109.r344615

The paper examines funding for software. Software must compete against other priorities, e.g., equipment. It also must compete against other more “exciting” ideas, and hence may be neglected despite its importance to research. As an example, here’s a relevant and recent skeet from the software goliath Hadley Wickham (27 November 2024):

“I had zero luck with grants. People really didn’t get why what I did was important and never figured out how to explain it to them.” https://bsky.app/profile/hadleywickham.bsky.social/post/3lbumdqcmcc2r

And an interesting reply to Hadley’s skeet: “lol thinking about you burying the tidyverse into a corner of aim 3 for an NIH grant on some very specific cardiovascular outcome, which is how most infrastructure work gets funded” https://bsky.app/profile/travisgerke.bsky.social/post/3lca7rkqv222z

I don’t usually quote skeets or tweets in peer reviews, but these are good anecdotal examples of how software can miss out to other priorities. It also highlights how software is often funded as an add-on rather than the main idea.

Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. What’s the total funding awarded per year? How many scientists per year do they support?

The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.

The questions asked the funders about priorities, but did not ask for actual financial support. Funders can have many priorities, but with strong competition for funding they might not all get funded. Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.

The quotes from funders were generally very positive and I expected to see something in the results on the challenges of funding software. This prompted my previous question on wanting to know more about the funders, as I wondered if they were an atypical group of funders with a particular interest in software. Perhaps the funders were not prompted for their negative experiences?

I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.

Minor comments:

Sampling section (and elsewhere): “87.8%” – no need for a decimal place, see: Cole T. Too many digits: the presentation of numerical data, DOI: 10.1136/archdischild-2014-307149
The key question “What are your organization’s top priorities related to research software?” has no time frame. So I presume it was current priorities only? Although the question: “Has your organization established any policies, initiatives or programs aimed at supporting research software?” covers all times. A consistent time frame may have been better, for example, “in the last 5 years”.
Discussion: I agree that funders have a part to play in increasing the recognition of software. Funders can be influential in how science is recognised and rewarded. The COARA initiative is likely relevant here, as it discusses needing to recognise the “diversity of contributions” which includes those scientists who create valuable software (https://coara.eu/agreement/the-commitments/).

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Cole TJ: Too many digits: the presentation of numerical data.Arch Dis Child. 2015; 100 (7): 608-9 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Statistics, research funding, meta-research

CITE

Report a concern

Author Response 06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

06 Jan 2025

Author Response

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points ... Continue reading We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points have been addressed in the new version of the manuscript are presented below:
Reviewer 1: “Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. [a] Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. [b] What’s the total funding awarded per year? How many scientists per year do they support?”

Response: [a] This information was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” [b] this information was not collected in the survey.

Reviewer 1: “The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.”

Response: This is already discussed as a limitation. However, more limitations content has now been added to the methods section: “ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding.”

Reviewer 1: “The questions asked the funders about priorities, but did not ask for actual financial support. […] Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.”

Response: This point is now addressed in the survey design subsection of the Methods section: “Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.”

Reviewer 1: “I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.”

Response: There is an important distinction between research software and software for more general use. Research software, per se, often attracts less industry interest than some other software domains due to its tendency to have a narrower market/commercial potential.
We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points have been addressed in the new version of the manuscript are presented below:
Reviewer 1: “Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. [a] Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. [b] What’s the total funding awarded per year? How many scientists per year do they support?”

Response: [a] This information was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” [b] this information was not collected in the survey.

Reviewer 1: “The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.”

Response: This is already discussed as a limitation. However, more limitations content has now been added to the methods section: “ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding.”

Reviewer 1: “The questions asked the funders about priorities, but did not ask for actual financial support. […] Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.”

Response: This point is now addressed in the survey design subsection of the Methods section: “Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.”

Reviewer 1: “I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.”

Response: There is an important distinction between research software and software for more general use. Research software, per se, often attracts less industry interest than some other software domains due to its tendency to have a narrower market/commercial potential.
Competing Interests: I am the first author for the original manuscript submission. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

06 Jan 2025

Author Response

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points ... Continue reading We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points have been addressed in the new version of the manuscript are presented below:
Reviewer 1: “Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. [a] Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. [b] What’s the total funding awarded per year? How many scientists per year do they support?”

Response: [a] This information was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” [b] this information was not collected in the survey.

Reviewer 1: “The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.”

Response: This is already discussed as a limitation. However, more limitations content has now been added to the methods section: “ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding.”

Reviewer 1: “The questions asked the funders about priorities, but did not ask for actual financial support. […] Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.”

Response: This point is now addressed in the survey design subsection of the Methods section: “Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.”

Reviewer 1: “I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.”

Response: There is an important distinction between research software and software for more general use. Research software, per se, often attracts less industry interest than some other software domains due to its tendency to have a narrower market/commercial potential.
We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points have been addressed in the new version of the manuscript are presented below:
Reviewer 1: “Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. [a] Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. [b] What’s the total funding awarded per year? How many scientists per year do they support?”

Response: [a] This information was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” [b] this information was not collected in the survey.

Reviewer 1: “The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.”

Response: This is already discussed as a limitation. However, more limitations content has now been added to the methods section: “ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding.”

Reviewer 1: “The questions asked the funders about priorities, but did not ask for actual financial support. […] Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.”

Response: This point is now addressed in the survey design subsection of the Methods section: “Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.”

Reviewer 1: “I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.”

Response: There is an important distinction between research software and software for more general use. Research software, per se, often attracts less industry interest than some other software domains due to its tendency to have a narrower market/commercial potential.
Competing Interests: I am the first author for the original manuscript submission. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 29 Nov 2024

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3	4	5
Version 2 (revision) 06 Jan 25	read	read	read	read	read
Version 1 29 Nov 24	read	read

Adrian Barnett, Queensland University of Technology, Brisbane, Australia
Mario COCCIA, National Research Council Department of Engineering ICT and Technology for Energy and Transport, Rome, Italy
Claire Jean-Quartier, Graz University of Technology, Graz, Austria
Fabio Kon, University of São Paulo, São Paulo, Brazil
Rena Bakhshi, Netherlands eScience Center, Amsterdam, The Netherlands

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

12 Views

06 Feb 2025 | for Version 2

Rena Bakhshi, Netherlands eScience Center, Amsterdam, The Netherlands

12 Views Cite this report Responses(0)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

research software engineering, project management, program management, computer science (formal methods)

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

13 Views

05 Feb 2025 | for Version 2

Fabio Kon, University of São Paulo, São Paulo, Brazil

13 Views Cite this report Responses(0)

Approved

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Software Engineering, Data Science, Distributed Systems

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

14 Views

25 Jan 2025 | for Version 2

Claire Jean-Quartier, Research Data Management, Graz University of Technology, Graz, Austria

14 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

research data management, open science, bioinformatics, cancer science

Respond to this report

Responses (1)

Back to all reports

Reviewer Report

20 Views

22 Jan 2025 | for Version 2

Mario COCCIA, DSU, National Research Council Department of Engineering ICT and Technology for Energy and Transport, Rome, Lazio, Italy

20 Views Cite this report Responses(0)

Not Approved

Authors have partially clarified my questions presented in my first report.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

science development and policies

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

11 Views

07 Jan 2025 | for Version 2

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

11 Views Cite this report Responses(0)

Approved

No further comments.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Statistics, research funding, meta-research

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

35 Views

13 Dec 2024 | for Version 1

Mario COCCIA, DSU, National Research Council Department of Engineering ICT and Technology for Energy and Transport, Rome, Lazio, Italy

35 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

science development and policies

Respond to this report

Responses (1)

Author Response

06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. While we don’t entirely agree with all points presented in this reviewer report, we have taken on board several points and made updates to the manuscript.

Reviewer 2: “Title has to be shorter and more specific of the fields concerning funding research software.”

Response: The relevant research fields for the participating research funders was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” Because this includes all research fields, it wouldn’t be appropriate to claim a narrower scope in the title.

Reviewer 2: The reviewer makes several requests for a wider paper scope, e.g.: “Introduction has to better clarify the research questions of this study, considering that software is an intangible innovation, clarifying the sources that are problem driven in specific contexts, such as medicine, environmental sciences, etc. Role of metrics that can assess the effectiveness of research software as an open science, such as citations and their association with funding in the diffusion of science.”

Response: These issues are beyond the scope of the current paper, which focuses on open-ended survey data from research funders. Imposing focus areas that did not emerge from these open-ended responses would be methodologically problematic. The research question currently stated in the manuscript is deliberately broad in line with the qualitative nature of the data that is being presented in the paper: “What are the top priorities of international research funding organizations related to research software?”

Reviewer 2: “Methods of this study is not clear. The section of Materials and methods must be re-structured with the following three sections: […] Insert a flow chart to specify the study design, with inclusion and exclusion criteria of organizations and respondents under study.
The identifications of themes and key aspects can be done with specific quantitative approaches, and not only qualitative, such also as entity linking, etc.”
Response: The current Methods section structure follows the journal guidelines and is appropriate for a qualitative survey study like this. A purely qualitative analysis and reporting of results is permitted by this journal; adding quantitative methods is not required.

Reviewer 2: “Authors have to avoid subheadings that create fragmentation and confusion. If necessary, you can use bullet points (same comments for section of results and all sections).
Results can be also systematized in some tables to be more effective in communicating results.
I suggest using different levels of description results from general to more specific themes, using tables to be more effective.”

Response: We appreciate the reviewers ideas about how to present the results. However, the current structure is in line with accepted practices in qualitative research. Each subheading represents a distinct theme, so it would be inappropriate to force them together.

Reviewer 2: Raises concerns about the Discussion section structure

Response: While the reviewer’s feedback about the Discussion section is noted, it is presented in a manner consistent with the journal guidelines and normal social science practices. Adding a table, for example, would be out of line with typical Discussion section approaches for this kind of paper.

Suggested readings: Because none of the suggested readings directly relate the research software funding, these papers have not been added to the citation list despite the reviewer’s insistence on adding citations to the reviewer’s published work.

View more View less

Competing Interests

I am the first author of the manuscript that this review report addresses.

Back to all reports

Reviewer Report

19 Views

13 Dec 2024 | for Version 1

Adrian Barnett, Queensland University of Technology, Brisbane, Queensland, Australia

19 Views Cite this report Responses(1)

Approved With Reservations

Sampling section (and elsewhere): “87.8%” – no need for a decimal place, see: Cole T. Too many digits: the presentation of numerical data, DOI: 10.1136/archdischild-2014-307149
The key question “What are your organization’s top priorities related to research software?” has no time frame. So I presume it was current priorities only? Although the question: “Has your organization established any policies, initiatives or programs aimed at supporting research software?” covers all times. A consistent time frame may have been better, for example, “in the last 5 years”.
Discussion: I agree that funders have a part to play in increasing the recognition of software. Funders can be influential in how science is recognised and rewarded. The COARA initiative is likely relevant here, as it discusses needing to recognise the “diversity of contributions” which includes those scientists who create valuable software (https://coara.eu/agreement/the-commitments/).

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Cole TJ: Too many digits: the presentation of numerical data.Arch Dis Child. 2015; 100 (7): 608-9 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Statistics, research funding, meta-research

Respond to this report

Responses (1)

Author Response

06 Jan 2025

Eric Jensen, National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, Urbana, USA

We are grateful for this report's helpful and constructive reviewer comments. The reviewer report requests some additional information and raises some points for discussion. The details for how these points have been addressed in the new version of the manuscript are presented below:
Reviewer 1: “Without breaking confidentiality, some more summary details on the funders would be useful to contextualise their answers and assess the generalisability of the results. [a] Are the funders predominantly in science, engineering, or health and medicine? Or do they cover all these areas. [b] What’s the total funding awarded per year? How many scientists per year do they support?”

Response: [a] This information was already presented in the ‘sampling subsection’ of the methods section: “Some participating funders cover a broad spectrum of disciplines, while others focus on a particular domain such as social science, health, environment, physical sciences or humanities.” [b] this information was not collected in the survey.

Reviewer 1: “The sampling frame could be biased towards funders with a greater interest in software as: “participation in the Research Software Association (ReSA) and responsibility for research software funding known to the authors.” This should be discussed as a limitation. It would also be useful to describe who ReSA are.”

Response: This is already discussed as a limitation. However, more limitations content has now been added to the methods section: “ReSA is an international organization involving research policymakers, funders and organizations that recognize software’s importance in their work. This means that a ReSA-focused sample is likely to over-represent research funders who already view research software as important. For this reason, this paper does not claim that its findings generalize to the global population of research funders; rather, the claims focus on current patterns in research software funding.”

Reviewer 1: “The questions asked the funders about priorities, but did not ask for actual financial support. […] Money spent on software would be the best measure of support. I realise this can’t be collected now, but some discussion of this would be useful.”

Response: This point is now addressed in the survey design subsection of the Methods section: “Clearly, potentially relevant variables were not included in the survey design for this exploratory study. These include details on the exact amount of funding spent on research software and other details about which many research funders would be unable to readily report either because the information is not actively tracked or relevant data do not currently exist.”

Reviewer 1: “I was surprised that there was nothing on partnering with industry. In Australia the funding schemes that require partnerships with industry (or government) have a much higher success rate than standard schemes. Given the potential commercial benefits of software, partnerships would be an option for many software applications.”

Response: There is an important distinction between research software and software for more general use. Research software, per se, often attracts less industry interest than some other software domains due to its tendency to have a narrower market/commercial potential.

View more View less

Competing Interests

I am the first author for the original manuscript submission.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] Barker M, Chue Hong NP, Katz DS, et al.: Introducing the FAIR principles for research software. Sci. Data. 2022; 9: Article 622. PubMed Abstract | Publisher Full Text | Free Full Text

[2] Barker M, Katz DS: Overview of research software funding landscape. Zenodo. 2022. Publisher Full Text

[3] Barker M, Katz DS, Gonzalez-Beltran A: Evidence for the importance of research software. Zenodo. 2020. Publisher Full Text

[4] Barker M, Carlin D, Cohen J, et al.: Resources for supporting policy change in research institutions in practice: A report from Subgroup 2 of the ReSA & RDA Policies in Research Organisations for Research Software (PRO4RS) Working Group. Zenodo. 2024. Publisher Full Text

[5] Carver JC, Cosden IA, Hill C, et al.: Sustaining research software via research software engineers and professional associations. Proceedings of the 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS’21). IEEE; 2021; pp. 69–72. Publisher Full Text

[6] Carver JC, Weber N, Ram K, et al.: A survey of the state of the practice for research software in the United States. PeerJ. Comput. Sci. 2022; 8: e963. PubMed Abstract | Publisher Full Text | Free Full Text

[7] Cohen J, Katz DS, Barker M, et al.: The four pillars of research software engineering. IEEE Softw. 2021; 38(1): 97–105. Publisher Full Text

[8] Cosden IA, McHenry K, Katz DS: Research software engineers: Career entry points and training gaps. Comput. Sci. Eng. 2022; 24(6): 14–21. Publisher Full Text

[9] Duarte J, Li H, Roy A, et al.: FAIR AI models in high energy physics. Mach. Learn.: Sci. Technol. 2023; 4(4): 045062. Publisher Full Text

[10] Gruenpeter M, Katz DS, Lamprecht A-L, et al.: Defining research software: A controversial discussion. Zenodo. 2021. Publisher Full Text

[11] Hartley K, Barker M: Investing in people: Anticipating the future of research software. Research Software Alliance Blog. 2023, October. Reference Source

[12] Hertweck K, Strasser C, Taraborelli D: Insights and impact from five cycles of Essential Open Source Software for Science. Zenodo. 2024. Publisher Full Text

[13] Hocquet A, Wieber F, Gramelsberger G, et al.: Software in science is ubiquitous yet overlooked. Nat. Comput. Sci. 2024; 4: 465–468. PubMed Abstract | Publisher Full Text

[14] Howison J: Sustainability in scientific software: Ecosystem complexity and software visibility. Zenodo. 2020. Publisher Full Text

[15] Iacovou G: Funding open source science software. Notion. 2024, June. Retrieved November 26, 2024. Reference Source

[16] Jay C, Haines R, Katz DS: Software must be recognised as an important output of scholarly research. Int. J. Digit. Curation. 2021; 16(1). Publisher Full Text

[17] Jensen EA: Strategic priorities and challenges in research software funding: Results from an international survey. (Dataset). Zenodo. 2024. Publisher Full Text

[18] Jensen EA, Katz DS: From code to tenure: Valuing research software in academia. Commonplace. 2023a. Publisher Full Text

[19] Jensen EA, Katz DS: From backstage to spotlight: A call to highlight the critical role of research software. LSE Impact Blog. 2023b. Publisher Full Text

[20] Jensen EA, Laurie C: Doing real research: A practical guide to social research. SAGE; 2016.

[21] Jensen EA, Wong P, Reed MS: How research data deliver non-academic impacts: A secondary analysis of UK Research Excellence Framework impact case studies. PLoS One. 2022; 17(3): e0264914. PubMed Abstract | Publisher Full Text | Free Full Text

[22] Jiménez RC, Kuzak M, Alhamdoosh M, et al.: Four simple recommendations to encourage best practices in research software. F1000Res. 2017; 6: 876. PubMed Abstract | Publisher Full Text | Free Full Text

[23] Katz DS: Valuing people vs hardware. Daniel S. Katz’s Blog. 2021. Publisher Full Text

[24] Katz DS: There’s no such thing as sustainable research software. Daniel S. Katz’s Blog. 2024. Publisher Full Text

[25] Katz DS, Carver J, Chue Hong N, et al.: Addressing research software sustainability via institutes. Proceedings of the 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS’21). IEEE; 2021; pp. 13–16. Publisher Full Text

[26] Knowles R, Mateen BA, Yehudi Y: We need to talk about the lack of investment in digital research infrastructure. Nat. Comput. Sci. 2021; 1: 169–171. PubMed Abstract | Publisher Full Text

[27] Lamprecht A-L, Martinez-Ortiz C, Barker M, et al.: What do we (not) know about research software engineering? Journal of Open Research Software. 2022; 10(1): Article 11. Publisher Full Text

[28] McKiernan EC, Barba LA, Bourne PE, et al.: Policy recommendations to ensure that research software is openly accessible and reusable. PLoS Biol. 2023; 21(5): e3002204. PubMed Abstract | Publisher Full Text | Free Full Text

[29] Puebla I, Ascoli GA, Blume J, et al.: Ten simple rules for recognizing data and software contributions in hiring, promotion, and tenure. PLoS Comput. Biol. 2024; 20(8): e1012296. PubMed Abstract | Publisher Full Text | Free Full Text

[30] Sellanga G, Steinhart E, Tsang N, et al.: 2024 survey of recent open science policy developments. Invest in Open Infrastructure. 2024. Publisher Full Text

[31] Sonabend R, Gruson H, Wolansky L, et al.: FAIR-USE4OS: Guidelines for creating impactful open-source software. PLoS Comput. Biol. 2024; 20(5): e1012045. PubMed Abstract | Publisher Full Text | Free Full Text

[32] Strasser C, Hertweck K, Greenberg J, et al.: Ten simple rules for funding scientific open source software. PLoS Comput. Biol. 2022; 18(11): e1010627. PubMed Abstract | Publisher Full Text | Free Full Text

Strategic priorities and challenges in research software funding: Results from an international survey

Abstract

Background

Methods

Results

Conclusions

Keywords

Revised Amendments from Version 1

Introduction

Methods

Survey design

Sampling

Data collection, management and analysis

Results

Skills

Software sustainability

Open science

Building community and collaboration

Advancing research software funding

Increasing software visibility and use

Innovation

Security

Discussion

Conclusion

Ethics and consent

Data availability

Underlying data

Extended data

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated