The case for openness in engineering research

Introduction

Working openly should be the default mode of science—after all, how can we advance knowledge “by standing on the shoulders of giants”^a if we cannot access or see those shoulders? This paper operates on the following definition of open science:

The field of engineering provides an interesting case study for examining the impacts of open practices since engineering touches every aspect of human life. Engineering research is inherent to the development of goods and products such as medical devices and pharmaceuticals, so issues around the protection of intellectual property and innovation draw stark contrasts, for some, with the tenets of open science. On the other hand, work being done in open-source software and open-source hardware can enable us to engineer the tools of modern scientific discovery, greatly reducing the costs of scientific research¹.

In a time of constrained university budgets, which are not expected to improve as long as most public universities rely heavily on state funding, many universities are being forced to evaluate their institutional priorities². For some, particularly state universities subject to the whims of state legislation, this could mean abandoning the pursuit of fundamental or basic knowledge generation in favor of more marketable vocational training models that cater more directly to industry needs. While this model is in line with the Morrill Act of 1862, which underpins the missions of many US institutions, the university has evolved since that time to encompass a much greater proportion of the economic development of the country². Despite the challenges faced by institutions today, it is critical for the university to continue to position itself as a center of societal development—economically, technologically, and socially. Additionally, the university should push this model further towards positioning itself as the main driver of social and technological innovation. To achieve this, it is necessary to position and market the business of the university, as clearly as possible, as a service provider to many relevant stakeholders. This can be best accomplished by disseminating and distributing the products of university activities as widely as possible through open access publishing, open research, and open innovation, and further demonstrating the impact that these products have on local, state, national, and international populations. As stated by Ashley Farley of the Gates Foundation, “Open research should be the norm. Knowledge should be a public good”³.

Importance of open science

Conducting open research is an act of assigning value to the work to which you are passionately committed. This includes all of the final, polished products of that work—including papers, software, and data—as well as the half-baked ideas, the napkin sketches, the first drafts, and the failures. The dissemination of these artifacts may on occasion comprise an act of humility, but ultimately it recognizes that each of these items is a piece of the research process and that even your failures have value in the lessons you learned and can be passed on to others. Transforming research communities from traditional, closed environments to open ones is important for a number of reasons, including (but not necessarily limited to) the following six. McKiernan et al.⁴ discuss these and additional benefits for researchers working openly. Tennant et al.⁵ review in detail the benefits of open-access publications to academics and society.

Accessibility: Openness in research ensures that research products, particularly written output, remain accessible to all. This includes the research community, funders, policy makers, and the general public. Accessibility of research products is particularly important for publicly funded research—since the public paid for the research, the public should have access to, and be able to benefit from, it. (This does not prevent innovators or other parties from developing commercial intellectual property based on the findings, but ensures that the original discovery, when funded by the public, remains accessible to all.) While the body of published work available freely through open access or online social networks has approached 50%, this percentage is notably lower in engineering at approximately 35%⁶. However, Piwowar et al.⁷ found that this percentage drops below 20% when not considering articles self-archived on author websites, which can lack assurance of long-term availability.

Reproducibility: Releasing products of research, including software and data, helps enable reproducibility. This is particularly true for computational science, where a written description of methods can never describe an approach as completely as the source code⁸. In general, access to research software used to perform a computational study, or the data from an experimental study, should enable others to reproduce the findings of the original researchers. However, open science is a necessary but not sufficient aspect of reproducibility, as it can be challenging to reproduce or replicate results even with available research software and data^9,10.

Impact: As a selfish motivation, performing research openly helps increase the impact of the work. Studies have shown that open-access papers are cited more in most research fields. In engineering, open-access papers are cited around 1.5 times more often than non-open-access papers^6,11. Similarly, papers with associated open data were cited 9–50% more than those without^4,12. Vandewalle¹³ showed that papers in the image-processing field receive up to three times the number of citations when source code is made available. We must note, however, that the concept of impact should not be solely regarded through the measure of citations. The true societal impact of the work is likely more important but also more difficult to quantify¹⁴.

Establish priority: Some researchers hesitate to embrace open science out of a fear of being “scooped,” where competitors will use some findings, software tools, or data made available and then publish first. However, contrary to this belief, practicing open science can actually prevent being scooped: releasing preprints can establish priority of discoveries or techniques prior to the publication of a traditional peer-reviewed journal article^15,16.

The peer-review and editorial process of such papers can take many months or years, but journal articles are still necessary for research findings to be considered valid (and for researchers to receive credit). Publishing a preprint of an article publicly time-stamps the work, even as it undergoes peer review and possible revision.

Encourages trust: Embracing openness in scientific research can help encourage other researchers to trust published results, by giving the ability to inspect data or software. Soergel¹⁷ estimated that 5–100% of computational results given by software may be incorrect or inaccurate. While simply releasing source code openly will not solve this problem, this is a necessary step towards verification and reproducibility.

It’s nice: In addition to the above benefits, sharing products of research openly is kind to colleagues and the greater research community, as it prevents people from wasting time by unnecessarily repeating work. For example, many graduate students begin working on their dissertation research by attempting to reimplement another group’s methods and reproduce some of their published results. However, lacking access to software source code or datasets can hinder this work. As a result, significant time can be wasted guessing about minor implementation details or inputs not discussed in the corresponding published papers. This can be avoided by sharing the source code and data, which would allow these junior researchers to more quickly move on to new work. Graduate students and other researchers constantly face similar challenges that could be avoided by greater openness in research.

Openness increases societal impact of research

Many published journal articles go unread, even in their topical domains. One study of citation rates found that 27% of papers published in the natural sciences and engineering go uncited^{18 b}. Those who do read most papers likely come from research institutions similar to those of the authors, even if the findings could be impactful beyond these confines, for example by leading to policy changes or technological solutions for humanitarian purposes. In part, this is due to the challenging technical content, jargon, and niche topics—but it is also due to a lack of access to the journals where most research findings reside. (Making the content of these papers actually understandable or digestible by most potential readers is another challenge.) Considering the high and ever-increasing cost of scholarly journal subscriptions, research results should not be limited to those with the means to purchase access. By self-archiving (i.e., green open access) or publishing articles in open-access journals, researchers can ensure access for all members of society, including policymakers, funders, members of the media, entrepreneurs, and the general public—as well as scientists and engineers in the Global South.

Furthermore, being more open with all of the outputs of research (e.g., papers, software, data) could help improve the general public’s perception and trust in scientific research. Simply making research products available will not solve all of these problems—for one, it will not sway those who strongly believe ideas contrary to fact. However, ensuring everyone has access to the data researchers generate and analyze, and the software tools on which we rely, could eliminate one major barrier to trust in our findings¹⁹.

Looking specifically at the field of engineering, we can also find examples of the positive effects of open knowledge dissemination. According to Chris Ategeka, founder of Health Access Corps, “Patenting a social-impact product hinders scale, ultimately obstructing the maximum impact that particular product would have in the world if it was open source”²⁰. Thus, the clear benefit of using open research and development practices is achieving greater impact with your research products. The counter argument to this is that, through patenting, the entrepreneur can more easily market and sell their product in developed markets, which could then increase their ability to affect change by subsidizing their efforts in developing nations. This situation may hold true for products with broad appeal and therefore it is necessary for the inventor to assess which path will produce the greatest impact. Assuming, also, that we encourage and reward impact. It could be argued that in the majority of scenarios, open dissemination will yield greater impact through simplified adoption and adaptation by others—especially if the front-end development activities are incentivized in other ways.

Open science and the scholar’s research agenda

For the new researcher looking to build their profile and develop their research agenda, we present a vision and plan for performing research openly, synthesized from literature practices and advice. The ideas presented here are heavily inspired by examples from others active in this work, such as Lorena Barba’s Reproducibility PI Manifesto²¹, the Peer Reviewers’ Openness Initiative²², and others. While these exemplars provide useful case studies, it is important to emphasize that each individual must define for themselves a workflow that works for them. Sometimes it is enough to simply be more open than the current norms in their field.

Many fields within engineering lack reputable open-access journals; indeed, only 17% of published manuscripts in engineering can be accessed by the public for free legally, while some sub-disciplines, such as chemical engineering, are lower at 9% available⁷. Thus, the engineering researcher looking to publish in open-access venues can quickly become discouraged. For the early career researcher looking to make their work available while operating within this research environment can take simple steps such as submitting preprints of any publications to the engrXiv^c, and deposit (otherwise non-accessible) conference papers or slide decks on Figshare^d.

For the researcher looking to develop their open workflow further, we recommend the following steps:

As a means of supporting these efforts, researchers should take care to implement these policy statements by incorporating them into funding proposals, for example in Data Management Plans. Note that policies where data and code are made “available upon request” are generally not sufficient for reproducibility²⁴.

Several community efforts have developed in recent years with the goal of defining and supporting open science practices. Some examples include the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE)²⁵ and the FORCE11 Software Citation Working Group which developed the Software Citation Principles²⁶ with the goal of standardizing software citation to help ensure authors/developers receive academic credit for their work in releasing open research software. On the publishing side, community driven research journals have been built to promote open publishing practices. Some examples include the Journal of Open Source Software^{g 27} and the Journal of Open Engineering^{h 28}. Similarly, engrXiv, an open archive for engineering publications, has been developed to serve the engineering community, inspired by the success of the arXiv.

Challenges to performing open science

The primary challenges facing those individuals interested in conducting open research generally involve incentives (or the lack thereof) and restrictive policies maintained by traditional publishers, in addition to the lack of a culture of sharing within the researcher’s disciplinary field. First, researchers are often pressured to carefully consider the venue in which they publish their work and to select only those that are “well established” and “high impact.” However, if these venues are not amenable to open research activities such as the posting of preprints, these challenges disincentivize those activities. To remedy this, the research community must continue to pressure publishers to modify their copyright transfer policies. Some progress has already been made in this effort through policies from funding sources such as the National Institutes of Healthⁱ, the National Science Foundation^j, the Bill & Melinda Gates Foundation^k, and the Wellcome Trust^l or from research institutions who require deposition in a repository. More information on these policies can be found on the Registry of Open Access Repository Mandates and Policies^m. Additionally, authors themselves can in some cases work with publishers to modify the standard publisher copyright transfer agreements allowing the author to retain more rightsⁿ.

Additionally, promotion and tenure requirements typically focus exclusively on the final published manuscript and associated metrics, neglecting other research outputs such as code, data, solid models, etc., and their associated impacts. Some institutions actively discourage making these alternative research products available due to idealistic dreams of future income generation from licensing revenues. However, in reality, the majority of universities lose money through their technology commercialization offices, since translation of university intellectual property to commercial success is generally poorly realized^29,30. Some institutions are instead pursuing alternatives such as EasyAccessIP^o which promotes universal knowledge dissemination as a mechanism “to create impact from university research outcomes as opposed to monetary aims.” Ultimately it is likely that societal pressure is necessary to push more institutions to participate in such initiatives. For that to happen, the public first needs to be aware of the possible benefits of broad knowledge dissemination and needs to experience those benefits first hand. Researchers may even see benefits in terms of their scholarly productivity, as Frankenhuis and Nettle³¹ argue that open-science practices may actually increase creativity and researcher output.

It seems the challenges impeding greater adoption of open-science practices are mainly institutional and cultural, rather than technical. General venues for sharing and developing the products of research openly abound these days, with the availability of services like arXiv, engrXiv, and PeerJ Preprints for ensuring open access of publications; repositories like GitHub for developing (and version-controlling) research software openly; and data and software archives like Zenodo and Figshare, which practically have no file size limitations^p. Of course, some technical problems remain: How do we make results of computational science, particularly when it involves demanding high-performance computing resources, truly reproducible? How can we cite software and data consistently, when the version might change regularly? How can open practices be integrated into a researcher’s workflow without further straining the researcher’s already overburdened time?

As cultural inertia and lack of institutional recognition/ rewards pose significant challenges to increased openness in science, the biggest barrier to greater openness in research may be ambivalence or outright hostility in many research communities. Many academic researchers either disagree on or are unaware of the importance (and benefits) of working openly. Since they were not trained in doing this, e.g., during graduate school or during postdoctoral training, they also may simply be unaware of how to research openly, or the resources that are available to do so. Furthermore, since most of their colleagues, collaborators, and competitors do not practice open science, no pressure comes from the research community to change. In addition, some communities do not support or actively oppose activities such as submitting preprints.

This lack of pressure is related to the other major issue: lack of institutional recognition and reward for open practices. In general, academic researchers will work on what gets them credit for promotion and tenure—anything beyond that requires strong intrinsic motivation, or external motivators from the research community. At most institutions, promotion and tenure review includes some judgment (whether explicit or implicit) of where faculty publish their work, but many, “high-impact”’ traditional publication venues—particularly domain journals—may not support, e.g., the posting of preprints.

Recommendations for university leaders

As already discussed, there are real career advantages to open-access publishing and open dissemination of data, code, or other research products and therefore, for some, the incentives to conduct open research may already be in place. However, for many, citation metrics alone are not enough to ensure success in promotion and tenure, and therefore they must play to the norms of their field, department, and institution. Therefore, the institution (and the department) should be looking to institute policy that redefines how we measure success in academic engineering research. Some suggestions include focusing less on journal-level metrics and lending greater credibility to article-level metrics. For article-level metrics, go beyond the citation count and look for other evidence of research impact such as alternative metrics (tweets, blog posts, media coverage) and replication by others. Lastly, look for evidence of broader implications such as economic development, student development, or even lives saved. Encourage your researchers to aim for those broader impacts and value them greater than the publishing of one more paper.

Thinking about what institutions can do to promote open engineering research and create support structures around open dissemination, we provide the following recommendations:

Thinking specifically about the perspective of the researcher within an institution, the following list of recommendations for departments are mostly targeted at changing criteria for promotion and tenure, and performance reviews, to encourage faculty to practice more open science:

Research communities that impede openness cannot be forced to change from the outside. Instead, by making changes to institutional reward systems, researchers will be encouraged to improve their open practices, and thus evolve communities from the inside.

Conclusions

In this paper we have reviewed the existing state of knowledge on the benefits and challenges of practicing openness in engineering research. We have further briefly outlined our thoughts on how open research practices in the sciences, engineering, and other fields can and should be employed by public universities to position themselves as centers for the creation and broad dissemination of knowledge as a public resource. The opposition to this proposal is immense, particularly in a political climate that devalues an educated populace and with systemic practices and policies that exclusively reward the monetization of any form of intellectual property. Change likely needs to be driven with grassroots initiatives that demonstrate the possible benefits and make it clear that tax dollars could fund these efforts if distributed properly and with accountability.

Data availability

No data are associated with this article.

Notes

^a“If I have seen further, it is by standing on the shoulders of giants.” Isaac Newton (1676), although similar statements can be found as far back as the 12th century.

^bOf course, papers that are read may not be cited, and papers that are cited may not actually be read.

^chttps://engrxiv.org

^dhttps://figshare.com

^ehttps://creativecommons.org/licenses/by/4.0/legalcode

^fhttps://zenodo.org

^ghttp://joss.theoj.org/

^hhttps://www.tjoe.org/

ⁱNIH Public Access Policy https://publicaccess.nih.gov/index.htm

^jNSF Public Access Plan https://www.nsf.gov/pubs/2015/nsf15052/nsf15052.pdf

^kBill & Melinda Gates Foundation Open Access Policy http://www.gatesfoundation.org/How-We-Work/General-Information/Open-Access-Policy

^lWellcome Trust Open Access Policy https://wellcome.ac.uk/funding/managing-grant/open-access-policy

^mROARMAP http://roarmap.eprints.org/

ⁿSPARC Author Addendum https://sparcopen.org/our-work/author-rights/#addendum

^ohttp://easyaccessip.com/

^pZenodo currently accepts datasets up to 50GB, but stores data in the CERN Data Center, along with 100PB of physics data from the Large Hadron Collider (https://zenodo.org/faq).