Fostering open science and responsible research practices: A pre-post study

Trial design

A single-arm pre-post study was designed following the CONSORT (Consolidated Standards of Reporting Trials) statement.¹² The study protocol was approved by the Research Ethics Committee of the Faculty of Dentistry, Universidade Federal de Pelotas, Brazil, (https://wp.ufpel.edu.br/cepesef/) on March 03, 2022, under tracking number 54276221.2.0000.5318. This study also adheres to the Declaration of Helsinki statement.¹³ The protocol, questionnaire, analysis plan, Informed Consent Form (ICF), and the complete anonymized dataset are openly available in the project folder on the OSF platform (https://osf.io/nz7va/) without restrictions. No significant methodological changes occurred after the study began, and it was conducted according to the registered protocol.

Participants

The study was carried out at the Universidade Federal de Pelotas (UFPel) in southern Brazil. The graduate programs at UFPel comprised at the time of the study of 1265 Master’s and 1374 PhD candidates. In Brazil, academic and professional masters are part of graduate studies, and professional habilitation is received after the bachelor studies. At UFPel, no course about RRPs and OS was available to all candidates before the implementation of the course reported here.

The inclusion criteria were MSc and PhD candidates, registered in the graduate programs at UFPel. The exclusion criteria were students not fully enrolled in the programs, i.e, participating in specific classes but not fully enrolled, trainees, visiting scholars and mobility students.

Recruitment and timeline

The inclusion criteria were applied to all MSc and PhD candidates at UFPel registered in the Brazilian national system and all of them were invited to participate. The list of all eligible participants and their e-mail addresses were obtained from the responsible department at UFPel (Pró-Reitoria de Pesquisa, Pós-Graduação e Inovação - PRPPGI). Before starting the course, the students who accepted the invitation were asked to answer a questionnaire (survey tool developed for this study and explained below) for the first time. In the same way, the number of OS practices outputs from each participant was collected at the Open Science Framework (OSF.io) before they started the course. Once all the enrolled participants answered the questionnaire, the course started. At the end of the course, all participants were invited to answer the questionnaire again ( Figure 1).

Figure 1. Participant timeline with the study steps described.

Intervention

The intervention was an online interactive course focused on topics such as OS, including open data, RI, RRPs, and transparency in science. During the course, the main problems resulting from research misconduct and questionable practices were discussed, and possible solutions based on adoption of OS practices were promoted. The course also showed how the whole research lifecycle could be carried out using OS practices ( Figure 2).

Figure 2. The steps and details of the OS life cycle (Adapted from https://www.cos.io/).

The course was made available in video format with the participation of instructors and researchers on the topics, and included 85 hours of lectures, workshops, and supporting learning material (self-study). The box below provides a summary of the course structure and its contents.

Students were able to access the videos through login into the platform. The intervention was applied just once to each participant, and at the end of the course, each participant was invited to answer the questionnaire again.

The course syllabus, all course materials (in Brazilian Portuguese), and content are openly available on a specific project hosted at the Open Science Framework (https://osf.io/7rz5n/).

Data collection

OSF data collection

According to the primary objective of our study, data related to the participants’ OS practices outputs was collected at the OSF website (osf.io). This platform was chosen because the participants had no prior knowledge of any OS tools, and the OSF was demonstrated during the course. The procedure for data collection was based on typing the participants’ names directly on the OSF searching tool. If the participant had an OSF account, this was registered, as well as the number of registered studies, protocols published, analysis plans related to their projects, data sets shared, preprint publications, and OSF projects for each participant.

Questionnaire

Following an extensive literature review conducted by JC (https://osf.io/hjtwn), only one relevant but unvalidated questionnaire was identified.¹⁴ This questionnaire was then adapted to the Brazilian context, incorporating elements from the Brazilian research assessment system and Open Science (OS) practices based on indicators proposed in the Declaration on Research Assessment (DORA),¹⁵ the Leiden Manifesto,¹⁶ and the Hong Kong Principles.¹⁷ The final survey comprised 50 questions distributed across five sections.

The entire questionnaire, including the introduction, informed consent form, questions, and explanation of the concepts, is available in the project folder on the OSF platform (https://osf.io/nz7va/). In brief, the questionnaire consisted of: Part 1 - demographic questions; Part 2-3 questions on researchers evaluation in Brazil; Part 3-27 typical academic activities or characteristics, for which respondents were asked to rate the perceived impact on (I) advancing career, (II) advancing science, (III) personal satisfaction, and (IV) social impact; Part 4-4 questions about their perception about the freedom they sense to adopt RRPs and OS practices, and their willingness to adopt those practices it in the future; and Part 5 - questions related to their current scientific production. Only the 3 first parts of the questionnaire are reported here, because they answer the stated secondary objectives of our protocol.

The questionnaire was hosted online at REDCap (REDCap – Research Electronic Data Capture, Vanderbilt, Nashville, Tennessee, USA, www.projectredcap.org/). REDCap was used to collect, store, and anonymize the data.

Piloting the questionnaire

This questionnaire was tested with 93 different master’s and PhD students from UFPel (who did not participate in the final study) to assess its comprehensibility in generating answers about the research hypotheses. For each question, participants were asked to answer whether the question was clear about their understanding when answering it. The questions evaluated with a low level of clarity and understanding were edited until they reached a consensus among the participating researchers. The total time needed was about 20 minutes. The pilot results are available at OSF (https://osf.io/kag4w).

Outcomes

Primary outcome

The primary outcome was the number of outputs per participant on the OSF platform (preregistrations, protocols, analysis plans/data management plans (DMP), data sets and preprints), before and after the intervention. The primary outcome was collected by JC before the course, just after, and after 6 and 12 months.

Secondary outcomes

Blinding and consent

The participants were unaware of being enrolled in our study during the course. At the end of the intervention, and after they had answered the questionnaire for the second time, they were informed about the intention of reporting on the data collected, and they were invited to participate and sign the informed consent form (ICF). We used only data from participants who gave permission. There were no negative consequences for students who refused, as the students’ evaluations and grades were conducted and finished before we asked for their ICF signatures.

Statistical methods

Descriptive statistics was used to report the number of OS practices and summarize questionnaire answer’s part 1. In order to confirm the results for the primary outcome, the Friedman test with Wilcoxon signed-rank with Bonferroni correction post hoc was applied

To answer the secondary objectives, the questionnaire answers were evaluated as follows:

Descriptive analysis was performed to get the frequencies of the participant’s answers to the 3 questions addressing their perceptions of the current researchers’ assessment system (Charts 2, 3, and 4). No statistical test was previous planned in the protocol, but to confirm the differences in the results, the Wilcoxon signed rank test with continuity correction was applied for each question in part 2. To make evaluation possible using the Wilcoxon test, considering alternatives A, B, C and D as an ordinal scale, alternative E was treated as missing for question 3 of part 2.

To compare the participants’ perceptions about the 27 common researchers’ activities related to the third part of the questionnaire, those activities were separated into activities related to traditional and non-traditional practices, as indicated in Table 1. The main objective was to identify what activities resulted in essential, important, irrelevant, unfavorable, or detrimental perceived impact in each one of the 4 domains (advancing career, advancing science, personal satisfaction, and social impact), and what changes related to it we can identify comparing before and after the intervention. To enable statistical analysis, the responses related to the Likert scale were transformed numerically as follows: detrimental = 1, unfavorable = 2, irrelevant = 3, important = 4, essential = 5. A score (average) for each group of activities was calculated and compared ( Table 2). The nomenclature and characterization of activities into traditional and non-traditional were adapted from previous studies.^18,19

Although no statistical test was planned to this part in the protocol, a Paired T-test was performed for each domain (advancing career, advancing science, personal satisfaction, and social impact) in each group (traditional and non-traditional) to aid data interpretation.

Despite not being listed in the protocol as one of the objectives of this study, the assessment of the intention to adopt OS practices (protocol registration; protocol publication; opening the analysis plans; opening the research notebooks; data sharing; and sharing codes), assessed by a question in part 4 of the questionnaire, presented interesting results and was therefore included in this article. Descriptive analysis was done to analyze the answers regarding the participants’ intention to adhere to these practices in the future, and charts was presented to summarize the results ( Figure 7). The Wilcoxon test was also applied to each practice to confirm the results.

A confidence interval of 95% and a significance level of α = 0.05 were used for all quantitative analyses. All analyses were conducted using R statistical software (Version 4.3.0, RStudio Inc., Boston, USA, free available at https://www.r-project.org/).

Demographics

Of the 84 participants enrolled in the course, 83 finished all activities. Only the 80 participants who signed the ICF were included in the analysis. The mean age of the participants was 32.3, varying from 22 to 57 years. The gender distribution was 62 females (77.5%) and 18 males (22.5%).

Most participants were from STEM (Science, Technology, Engineering, and Math) graduate programs (62; 77.5%) and 18 (22.5%) were from the Arts and Humanity graduate programs. 37 (46.3%) participants were at the PhD level and 43 (53.8%) were at the MSc level. Most of the participants graduated from public universities (59; 73.8%), followed by private universities (20; 25.0%) and Community universities (1; 1.3%). About the duration since they acquired their last title (undergraduate or master’s graduation), 66 (82.5%) got it less than 5 years ago, 11 (13.8%) between 5 to 10 years, and 3 (3.8%) got it more than 10 years ago. Finally, most participants (73; 91%) had never taken a course addressing RI and OS practices.

OSF outputs

To answer the study’s primary objective, the data referring to the number of outputs in the OSF platform was shown in Figure 3. The number of OSF accounts increased from 7 to 78 after the end of the course. Two participants have not made an account during the course, and this situation remained after 6 and 12 months. Before the course, the number of projects was 7 and the number of public projects was 3. Just after the course, they increased to 9 and 4 respectively, and after 6 months the number of projects increased to 10. No changes happened after 12 months of the end of intervention. No registrations, protocols, analysis plans, data sets, or preprints were found before or after the course.

Figure 3. The number of OSF outputs before, just after, and six months after intervention.

The Friedman test with Wilcoxon signed-rank post hoc results shown that OSF account before the course is statistical different of the other 3 times evaluated (p<.01); the projects and public projects did not show any statistical difference between the evaluated time frames (p=0.6; p=0.3).

Participants’ perceptions of the current researchers’ assessment system

Overall, participants perceived before the intervention that the current evaluation system contributes to promote good practices in research ( Figure 4). However, this perception changed after the intervention, showing a higher proportion of scores 1 and 2 (assessment system does not contribute) and a lower proportion in scores 4 and 5 (assessment system contributes to good practices in research) ( Figure 4). The same trends were observed in answering the question on how much the current evaluation system contributes to the delivering value to society ( Figure 5).

Figure 4. Participants answer to the question 1, Part 2 of the questionnaire.

Figure 5. Participants answer to the question 2, Part 2 of the questionnaire.

When inquired about the relation between the researchers’ current evaluation model and the hiring processes/distribution of grants, no participant pointed out that ‘the process answers all requirements and must be maintained’ after the intervention compared to 3.2% in the first evaluation. The number of participants who answered that ‘The process answers the most important requirements but may be improved’ decreased from 43.8% to 27.5% after the intervention. The frequencies for the answer ‘The process needs to be reviewed because it partially answers the most important requirements’ increased from 25% to 60% and the answer ‘The process does not answer the most important requirements and needs to be completely reformulated’ increased from 5% to 11% after the intervention. The number of participants who chose the option ‘I don’t want to answer’ decreased from 22.5% to 1.3% ( Figure 6).

Figure 6. Participants answer to the question 3, Part 2 of the questionnaire.

Perceptions about the 27 common researchers’ activities

For both, traditional and non-traditional activities, there was a statistical difference in the importance given by the participants before and after the intervention only for ‘advancing science’ and ‘social impact’ domains ( Table 2).

Current adoption or intention to adopt OS practices by the participants

The answers to the question ‘What practices do you adopt or think about adopting for your projects?’ are shown in the Figure 7.

Figure 7. Participants’ intention to adhere to the open science practices in the future.

For the results of the Wilcoxon test, protocol registration (p<0.01), protocol publication (p<0.01), publishing and making available the analysis plans (p<0.01), opening the research notebooks (p<0.01), and sharing codes (p<0.01) showed statistical differences between before and after. Only the differences in data sharing (p=0.10) were not statistically significant.