Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme [version 1; peer review: 3 approved with reservations]

Background: This paper reports the results of an evaluation of the Software Sustainability Institute’s Fellowship Programme, which focused on identifying and categorising the benefits that the fellowship has afforded its recipients, via a series of open questions. Methods: The evaluation took the form of a survey open to people awarded Fellowships between 2012 and 2016, which asked people to report the effect that the programme had had on them, their institutions, their research domains and their careers. Results: The results show that the Fellowship plays a wide-ranging role in supporting communities of best practice and skills transfer, and that a significant benefit is the way it has raised the profile of software in research, and those people who develop and advocate for it. Conclusions: The evaluation of the programme has shown the need to support research software in situ and credit the engineers and researchers who are working in this important area that supports reproducibility, reuse and the integrity of research investments.


Introduction
The Fellowship programme 1 run by the UK Software Sustainability Institute (SSI) is a unique package of financial support, networking and advice, which is competitively awarded to members of the research software community. The main goals of the programme are to encourage Fellows to develop their interests in the area of software sustainability and help them to become ambassadors of good software practice in their domains. The programme offers £3000 to support event attendance, workshops, training and other activities to help build awareness, capability and capacity in computational techniques, reproducible research and open science in diverse research domains.
Fellows are selected via an open competition, where candidates are judged by a panel of experts (former Fellows and Institute staff members) in terms of their track record in practising and promoting software sustainability, and the activities they plan to run with the Fellowship award. To promote diversity, funding is allocated to people at different career stages (from PhD student to research leader) and a variety of domains (e.g. Glaciology, Research Software Engineering, Humanities and Astrophysics). The overarching aim of the Fellowship programme is to provide support and recognition to those people promoting sustainable software practices, and advocating for and producing more verifiable, shareable and useful research outputs. This paper reports the results of a recent survey evaluation of the programme's effects on its recipients and their wider communities. A thematic analysis of the results shows that the award of a Fellowship had substantial and wide-ranging benefits both for the Fellows themselves, and for their institutions and research domains. The theme that emerged most strongly and consistently was that the Fellowship provided status to both the Fellows themselves and the role of software within research. Respondents reported that current academic culture does not always afford recognition to research software and research software engineers, and that the Fellowship has played a key role in improving the visibility of this ubiquitous yet undervalued component of research methodology.
An earlier version of this article is available on PeerJ as a preprint https://doi.org/10.7287/peerj.preprints.26849v1.

Related work
A number of other fellowship providers have published evaluations of their programmes, including the Humboldt Foundation 2 , the Erwin Schrodinger Fellowships 3 and the Union for International Cancer Control (UICC) 4 . These evaluations used a combination of surveys, data held about the fellows (e.g. demographics, subject areas), and in the case of UICC, case studies. The reports are openly available, but do not constitute peer-reviewed research. Here, we take a different approach, treating the evaluation as a research project (for which ethical approval was obtained), asking primarily open questions, and only including data that were obtained via the study.
By conducting the work in this way, we aim to contribute empirically to the software sustainability literature, as well as gaining a local understanding of the Fellowship programme's impact.

Methods
The survey was conducted using the University of Manchester SelectSurvey.NET instance to ensure the data was collected and stored securely. Participants were contacted via email using the all-fellows mailing list; all current and previous Fellows who are still in contact with the Institute are subscribed to this list. The survey was conducted from the 12 th July 2017 to 31 st August 2017. After the initial email there were two reminder emails and we chased two individual Fellows who had only made partial survey entries to see if they would offer complete entries (which the subsequently did).
The initial part of the survey explained what the purpose of this research was and asked for consent from participants. Participants were asked to confirm that they agreed to participate, that they understood that participation was voluntary, that they understood their data would remain confidential, and that they permitted anonymous quotes to be published. They were able to say 'Yes' or 'No' to any of these questions. All participants included in the analysis answered 'Yes' to all of these questions. There was a further question around retention, "I agree to my data being retained indefinitely for further research related to the Fellowship Programme." All participants bar one answered 'Yes' to this question.
The survey then asked Fellows to comment on the benefits of the programme in a number of categories, and to report any negative consequences and suggested improvements (see Table 1). The survey was sent to the entire population of the 2012-2016 Fellows (78 in total). The study received approval from the Computer Science School Panel (ref: 2017-2308-3295) on the delegated authority of the University Research Ethics Committee (UREC), University of Manchester .
Fellows were asked to provide information about gender, year in which their Fellowship was awarded, which funding bodies supported their work and their research area. It also asked about their current job role, job role at the time the Fellowship was awarded, and specific research area, but this information is not reported here as the small number of participants means it may be possible to identify individuals from this data.
The free text answers were thematically analyzed in an open coding fashion following established analysis methods 5 : 1) familiarization with data, 2) generating the initial codes, 3) searching for themes, and 4) iteratively reviewing themes. The generated codebook was agreed between the authors.

Results
There was a response rate of 33% (N = 26). Seven fellows from 2016 responded, 8 from 2015, 6 from 2014, 4 from 2013 and 1 from 2012. One of the respondents (Caroline Jay) is an author of this paper, and her results have thus been excluded from the analysis, leaving a total of 25 respondents.

Demographic information
Five respondents were female and 21 were male. Table 2 shows the funding bodies that supported the respondents' research.

Forced choice analysis
In answer to the question, 'Do you think being awarded a Software Sustainability Institute Fellowship has benefitted you?' 96% (n = 24) answered 'yes'. One person answered 'unsure' and zero people answered 'no.' In answer to the question, 'Do you think being a Fellow has helped to advance your career?' 72% (n = 18) answered 'yes,' 16% (n = 4) answered 'no', and 12% (n = 3) answered 'unsure.'

Free text analysis
The first author coded the dataset into a number of initial themes. These were grouped into overarching themes by the second author, which were then used as a codebook for the answers to the questions 'How has the Fellowship benefitted you/your institution(s)/your domain/others?'. The results were checked by the first author for agreement. The emergent themes are described in the bulleted list below.
• Status: giving status and recognition to individuals and organisations for their role in sustaining software, and to sustainable software practices themselves.
• Community/network: organizing/attending events; building professional and personal networks.
• Professional development: improving one's own skills through undertaking training and improving the skills of others by providing training.
• Resources: obtaining resources for travel and other professional activities. Table 3 shows the number of respondents who reported a benefit under each theme for the categories that the questions asked about: self, institution, domain and others. In the following sections we explore each of these themes in turn.

Status
Across the questions, 31 comments were made in relation to the Fellowship leading to an improvement in "profile and prestige" (R5). The majority of these (twenty responses) were in relation to improving the status of the individual Fellow.
The impact on the Fellows' status manifested itself in a number of ways, including: giving them recognition as someone who knew about software sustainability and good coding practices; providing a badge which opened doors and allowed them to market themselves; and becoming more appealing as collaborators at the institutional, domain and interdisciplinary level. Four respondents reported that having a Fellow raised the profile of a department or institution.    Respondents reported that the Fellowship gave them the mandate to collaborate with different organisations and institutions, as well improving the local networking of those involved with research software. Three Fellows at one institution were able to work together.
Fellows from a single domain expressed that a number of them working with each other across years had had a cumulative effect over time, in effect seeding a hub of researchers/ fellows who took sustainability seriously. There was a platform for them to then influence domain specific groups at different institutions increasing the impact and reach of promoting better sustainability practices. Fellows felt motivated to collaborate, form online communities, and contribute to the open source community.
The Fellowship ultimately provided community, friendship and motivation for new ways of doing things. The Fellowship also helped them become better scientists and ambassadors for sustainability issues in their community and thus better recognised. Table 5 illustrates the impact of the Fellowship on community and network with quotations.

Professional development
Respondents stated that the Programme had helped them to progress in their careers, either by way of a new job, promotion, or change in direction: "I can map my entire career trajectory from the opportunity that the fellowship gave me. One meeting led to another..." (R11).
In answer to the question, 'If not already specified, how has being a Fellow helped your career progression?' three respondents The Fellowship awards had an even greater impact on the professional development of others, with 26 comments relating to this altogether. Fellows ran training courses, such as Software Carpentry 6 , spread best practice via workshops, and supported data sharing and reproducibility initiatives. Table 6 illustrates the impact of the Fellowship on professional development with quotations.

Resources
Fellows used the £3000 award for attending conferences and workshops that they normally would not be able to; organising events; running training; kick-starting an initiative (such as a product, service or approach); and inviting visitors. Although not everyone used the funds: "My position is probably different to many fellows in that I mostly wanted to be a fellow to show support for the SSI and the fellows network/community and to highlight the importance of this area in my institution. Access to funds wasn't a consideration" (R3), across the respondents they supported a wide range of activities, summarised in Table 7.
Negative consequences In answer to the question, 'Have there been any negative consequences of your fellowship?' 14 people said there had not been anything negative, and 7 people did not give an answer. One person commented that they sometimes had to explain that software sustainability was not the same as digital preservation, and that this disappointed the person they were talking to. Three respondents gave lighthearted answers: "I definitely spend more time on Twitter because of you guys!" (R10); spending time "struggling with installing and implementing open source software (just kidding, though it takes time, I thoroughly enjoy learning new things, and it's an investment in the future)" (R11) and "a lack of time to take advantage of all the opportunities -not a bad problem to have!" (R23) Although the programme itself did not appear to result in negative consequences, R17 commented that their institution "was not interested in [the Fellowship] at all."

Improvements
In answer to the question, "How would you improve the Fellowship Programme?" six respondents did not make any suggestions. Nine respondents recommended increasing the number/length of events, and one raised an issue around the distance that they were required to travel for an event. One respondent suggested making more significant funds available to Fellows, including providing salary, and two commented that administration of funds could be improved. Three people had suggestions for improving mentoring, including having non-academic mentors, and using existing Fellows as mentors. Two respondents, who had both moved away from the UK, thought it would be good for the Institute to build stronger links internationally.  "Data sharing in *** is still in its infancy, but thanks to support from the Software Sustainability Institute colleagues in *** were able to host a datathon to promote reproducibility and data sharing in ***, as well as to share a *** dataset." (R1) Three respondents suggested having more explicit roles/activities for Fellows over the longer term. The free text questions and answers for the survey. CSV file. *** indicates removed to preserve anonymity

Limitations
The study focused on the benefits of the Fellowship Programme. We chose to use the word 'benefit', rather than 'impact', because we wanted people to reflect on the potential positives that came from the Fellowship in the broadest terms. Whilst the authors did not anticipate that the Fellowship would result in negative consequences, and a question checked for these explicitly, the phrasing of the questions could have biased respondents towards seeing the programme in a positive light. The survey only captured the responses of a third of Fellowship holders, so we do not know the experiences of the remaining two thirds.

Conclusion
The survey evaluation provided evidence that the Fellowship programme has played a significant role in supporting and galvanising engaged people in contributing to the domain of research software engineering. The gains in community building, networking, individual status, individual learning and the development of others, leading to long term benefits, initiatives and communities of practice are significant given the modest investment. Seed corn funding approaches are noted as being particularly effective mechanisms of support 7 . The evaluation of the programme has shown the need to support research software in situ and credit the engineers and researchers who are working in this important area that supports reproducibility, reuse and the integrity of research investments.

Lois Curfman McInnes
Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, USA 18th February 2019: This report was briefly published as an Approved report, and has now been updated to an Approved with Reservations at the request of the reviewer.
This paper evaluates the Fellowship Programme of the Software Sustainability Institute (SSI), which has the primary goals of encouraging Fellows to develop their interests in software sustainability and to become ambassadors of good software practice in their communities.
The paper analyses a survey of people who were Fellows during 2012-2016 (26 respondents of 78 Fellows contacted). The paper's goals are contributing to literature on software sustainability and understanding the program's impact. The results of the study include only data obtained from the study itself, which featured open questions. The paper explains the methods of the survey and discusses results, around four themes that emerged from analysis of free-text answers: status, community/network, professional development, and resources.
This paper and the SSI Fellowship Programme overall are of strong interest to international communities who are working to advance software practices as a key element of increasing overall scientific productivity. Overall, the paper is well written and clearly explains the approach and analysis of the survey. The analysis concludes that the Fellowship promotes the status of the role of research software and of the Fellows themselves. A key observation is that the Fellowship promotes community and provides a platform for Fellows to influence their domain-specific communities in advancing practices of research software. The Fellowship also contributes strongly to professional development.
My main criticism of the paper is the implicit assumption that the reader understands the importance of software sustainability and the scope of software practices addressed by the Fellowship Programme. I recommend adding background information about software sustainability and the SSI, including references, in order for the paper to be more effective as a stand-alone document.
While I understand the reason for the authors to exclude information about employment during and after the Programme (preserving anonymity), it would be interesting to explore the changes in employers and roles over time (of all Fellows), and whether the SSI Fellowship Programme influenced that. Also, it would be interesting to explore changes in the software practices and culture of domain-specific communities, to help understand the longer-term impact of the SSI Fellowship Programme.

If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Yes
Berkeley, CA, USA The paper presents the results of a survey investigating the impact of the UK Software Sustainability Institute's (SSI) Fellowship Programme on recipients' research, institutions, and careers. The authors note that their approach to Programme evaluation is novel in that, unlike other reports on programme outcomes (e.g., the Humboldt Foundation), it is a peer-reviewed study using only the data collected via survey. Therefore, the results contribute empirical findings to the software sustainability literature. The study found that the Programme elevated the perceived importance of software development for research as well as the status of the fellowship awardees. The authors state that the survey's findings indicate the importance and value of research software and the people who develop it.
The paper also provides a sound overview of the SSI Fellowship's goals: Supporting research software developers in their own work and in championing research software's value (e.g., promoting reproducible research and open science). The introduction notes how awardees are selected and how the Programme fosters diversity in career stages and disciplines.
Overall, I support the indexing of this study with one substantial addition (a discussion section) and several minor changes. I provide some details on what might be included in the discussion in Comment 3 below. The authors clearly state that they wish to contribute to the literature on software sustainability. However, they do not provide any definitions of software sustainability, generally, and the types of work and workers needed to achieve sustainability. These additions are necessary to contextualize the findings within the broader discussion about software sustainability. Although I am sure the authors are familiar with this emerging literature because they have made substantial contributions to it, I provide some references to start with (Crouch et  1.
The authors could reflect more on why there were far more respondents who identified as male than female. For example, does this set of responses reflect the overall makeup of the Fellowship Programme? If not, why might that be? 2.
The paper lacks a discussion that integrates and synthesizes the discrete findings sections. There are several possible ways forward to develop such a discussion question. The first suggestion is to contextualize the free text themes within the forced choice responses. For example, did the 4 respondents who answered "no" to the question about career advancement thematically respond to free text questions, particularly the one about negative impacts? Because the sample size is small, it may be possible to point out the threads in the responses of the "no" participants vs. the "yes" participants. Another way forward is to put all of the sections into conversation with one another. For example, in the discussion, the authors could discuss why (in relation to the existing literature or public discourse) fellowship awards had a greater impact on professional development of others than on the individuals. The "how" is already present and appreciated-e.g., Software Carpentry workshops. Likewise, the authors might return to statements such as R17's institution not being interested in the fellowship, and how that relates to the reported benefits to the institution. In sum, some more reflection on the responses and how they relate to the broader discourse around software sustainability and support for research 3.
software development would be very much appreciated.
I commend the authors for explicitly stating the themes in the findings section. I have one issue with how the authors collaborated for the qualitative coding of the free text responses: Did the authors use any "test data" or something similar to ensure inter-rater reliability? For a study like this one, I do not think it is important to report a quantitative measure of IRR, but a bit more detail about how authors reached agreement would be helpful and contribute to a perception of validity for the reader. One way of doing this is to briefly describe an example of disagreement between the two authors on a particular theme or instance of a theme and describe how the authors reached a resolution.

4.
What software, if any, was used for qualitative coding of the data? 5.
I thank the authors for providing Table 4 with some illustrative examples of responses. I also appreciate how the authors presented some examples in the text and place them in conversation with one another as you might see in an interview-based study (indeed, the prevalence of free-text questions lends itself nicely to presenting the results in this way).

6.
With regard to the statement "Across the other questions, 17 comments related to professional benefits for the Fellows themselves that included: improving personal knowledge and practices; understanding how much of research is software driven; developing a habit for research related blogging; identifying new areas in their own research fields; and thinking about research software engineering as a career": Can the authors provide some information about the prevalence of each category of professional benefits? Counts are not necessarily the only way to do this; the authors might add phrasing indicating whether one or more of the categories was more prominent than the others.

7.
In the limitations section, the authors might also note the bias in who would respond to such a survey. In other words, awardees who had a positive experience might be more inclined to respond to a survey about the program's benefits.

If applicable, is the statistical analysis and its interpretation appropriate?
Not applicable

Colin C. Venters
School of Computing and Engineering , University of Huddersfield, Huddersfield, UK This paper reports the results of a qualitative study into the perceived effectiveness of the Software Sustainability Institute's Fellowship program between 2012-2016; a programme that provides fellows with financial support to attend a range of events to help build awareness in sustainable software engineering "best" practice and processes relevant to research software. The authors suggest that the results of the study reveal that the fellowship programme supports communities of best practice and knowledge transfer in the development of sustainable research software. However, this position is not sufficiently supported by the anecdotal evidence reported in the paper.
This was an extremely interesting paper to read; hopefully the first of many by the authors or the Institute that examine the efficacy of the programme and the impact of the Software Sustainability Institute in transforming the development of research software as a whole. The topic is of significant interest to those working in the field of research software engineering -especially potential fellows -and of general interest to the broader software engineering community as a whole as it pertains to having identified individuals that practice and promote sustainable software engineering "best" practice with a particular focus on research software as a first-class research object. The paper attempts to address a pertinent question regarding the efficacy of the fellowship programme as a mechanism for gathering intelligence about research and software from all disciplines and identifying and communicating good software best practice in a range of different domains.
The paper would benefit enormously from a background section that included the motivation for the creation of the Software Sustainability Institute in the first instance i.e. unsustainability of academic software, and linking this to the study by Hettrick et al. 2014 1 who demonstrated the importance of software in research. In addition, the paper would also benefit from introducing and defining key terms and concepts for readers unfamiliar with the topic of software sustainability and linking these to emerging themes being driven forward by leading researchers and groups in the fields of requirements engineering, software architectures, HCI, and software engineering in general that address software sustainability.
The overall research methodology is described in sufficient detail. It is somewhat surprising and disappointing that critical demographic data has not been reported in the paper including the recipient's job role and their institutions at the time the fellowships were awarded, as this information is already publicly available and free to harvest should anyone be interested in doing so; this would seem to be a major oversight and the rationale in the paper for its exclusion would appear to be rather weak. In addition, the paper would have benefited enormously by including data related to recipient's track record and how have they promoted software sustainability prior to the fellowship programme as a comparison; this would have provided a baseline to establish how the programme transformed the recipients into ambassadors of good software practice in their domains. However, without having established what good or best practice means in the first instance, any claims are challenging at best to determine.
It is worth noting that not all fellows are research software engineers. As such, some analysis on the range of the recipient's job role would have been interesting since the programme is designed to promote diversity to people at different stages in their career.
The results of their analysis revealed a number of interesting findings. However, the main result revealed a worrying trend in that the primary outcome is simply an improvement in the status of beneficiaries of the fellowship programme. This raises serious questions as to whether the programme has failed to achieve its primary aim as a key driver of the programme is the promotion of sustainable software engineering practice to produce verifiable, shareable and useful research output.
The paper also contains several unsubstantiated statements to support a number of claims regarding the perceived benefits of the programme. The claim by one recipient that the collective fellowship in their domain has had a global impact cannot be substantiated and is at best anecdotal. Similarly, it is unclear how the programme helped recipients become "better" scientists. Against what baseline is this improvement being measured? Without any control mechanism in the study, it is impossible to demonstrate with any certainty that the perceived benefits of the programme could not have been achieved in another way.
It would have been interesting for the study to have considered what better sustainability practice means in practice and identify the range of sustainability issues that the variety of communities face. The use of a distributed version control system and commenting code does not in itself make software sustainable.
Based on the results of this study the primary benefits of the fellowship programme are the sole recipients themselves. The benefits to the broader community outside the echo chamber have yet to be established and provide a fruitful area for further research. Overall this was an interesting study of an exciting fellowship programme that lays the foundation for future assessment of the programme as a whole in order to assess the wider benefits to the broader community at large.

Partly
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Sustainable Software Engineering I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 21 Nov 2018 Shoaib Sufi, The Software Sustainability Institute, UK We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer's comments and suggestions below: It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.
We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent's public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.
The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.
The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.
We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.
A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk ), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.