Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals

The MSCA and ERC programme

The MSCA and ERC programmes are two flagships of H2020. Their initial earmarked budget amounted to 6.2 and 13.1 billion Euro, respectively, representing around 25% of the initial H2020 overall budget (European Commission, 2013).

The MSCA programme is composed of a set of funding schemes (actions), mostly dedicated to research training and career development of mobile researchers, especially those at an earlier career stage (Pina et al., 2015). Among these actions, the IF covers transnational postdoctoral research training, and ITN specifically targets doctoral training of early-stage researchers. These two actions represent around 95% of the total number of MSCA proposals submitted and funded.

The ERC grants are funding principal investigators and their teams on frontier research. The programme has remained quite stable since its inception in 2007. Taken together, the MSCA IF, ITN, and the ERC grants represent more than 42% of the overall eligible proposals submitted to the whole H2020 programme, and with success rates in the range of around 8–15%, they are among the most competitive funding scheme of the framework programme.

Both MSCA and ERC programmes cover all fields of research, in a so-called bottom-up approach. This means that applicants are free to apply for any type of research, with no ex-ante priority given to specific topics. This contrasts with most of the rest of the H2020 programmes. However, for evaluation purposes, MSCA applicants are asked to submit proposals in one of eight scientific panels: Chemistry (CHE), Social Sciences and Humanities (SOC), Economic Sciences (ECO), Information Science and Engineering (ENG), Environment and Geosciences (ENV), Life Sciences (LIF), Mathematics (MAT), and Physics (PHY). In the ERC, proposals are submitted to scientific panels belonging to one of the three following research domains: Life Sciences (LS), Physical Sciences and Engineering (PH), and Social Sciences and Humanities (SH). To facilitate the comparison of the two programmes, we grouped the ECO and SOC panels of MSCA as a proxy for the SH domain of ERC, and the CHE, ENG, ENV, MAT, PHY panels of MSCA as a proxy for the PH domain of ERC. Only the LIF panel in MSCA was considered as an equivalent of the LS domain of ERC.

Ethics review procedure

The ethics review procedure applies systematically for all proposals considered for funding (i.e. proposals ranked in the main list for funding) (Figure 1). It usually follows the scientific evaluation, where expert reviewers assess and score proposals based on a set of three criteria for the MSCA (Excellence, Impact, and Implementation) and the sole criterion of Excellence for ERC. All submitted proposals contain an ethics self-assessment checklist filled in by the applicants, where they list, in their views and opinion, the ethics issues raised by their proposed research (European Commission, 2019a). The applicants must submit an Ethics Annex only in the case if they declare any ethics issue in the checklist.

Figure 1. Ethics appraisal procedure in Horizon 2020.

PS – pre-screening, S – screening. Pre-screening in European Research Council (ERC) evaluation process is performed by internal ERC staff.

Proposals that do not raise immediate ethics issues are generally pre-screened by ERC dedicated internal staff, to assess if a full screening is necessary. In MSCA, main-listed proposals are in most of the cases sent automatically for screening. The following step in the ethics review process, the ethics screening, is performed by two or more external independent experts and, in some cases, it is followed by an ethics assessment when a more in-depth analysis is needed for serious or complex ethics issues raised by the proposed research. Grant proposals involving research with human embryonic stem cells or human embryos automatically go for ethics assessment (European Commission, 2019a). Ethics experts are involved in both the screening and the assessment phase. For ERC proposals, ethics assessments are managed internally, whereas MSCA ethics assessments are conducted by the Ethics Sector within the Directorate-General for Research and Innovation (DG RTD).

The final output of the ethics review is the Ethics Summary Report (EthSR), drafted by ethics experts, which includes an ethics opinion on the proposal. This opinion can have the following outcome: i) No ethics issues, when the ethics review confirms that the proposal does not raise any ethics issue; ii) Ethics clearance, when ethics issues are adequately addressed in the proposals, and there is no need for change upon signature of the grant agreement; and iii) Conditional ethics clearance, when the clearance is subject to conditions, usually in the form of “ethics requirements” listed in the EthSR. Ethics requirements must be implemented either before the grant signature, by updating the ethics section of the project description, or at a specific time after the grant signature in the form of ethics deliverables. When an ethics assessment is needed for proposals raising serious or complex ethics issues, the applicants are usually requested to submit additional information for an in-depth ethics analysis before the ethics experts formulate their opinion. Proposals conditionally cleared can therefore be the result of an ethics screening or an ethics assessment. In very rare cases, a project proposal can be rejected on ethics grounds (European Commission, 2019a)

Data sources and description of ethics issues categories

We analysed the data for main-listed proposals from MSCA IF, MSCA ITN calls in 2016-2019, and for ERC calls in 2017-2019, the year for which the data were available in the electronic Submission and Evaluation of Proposals (SEP) tool of the European Commission. SEP integrated the full ethics evaluation process in 2016 for MSCA and in 2017 for ERC. When this study was initiated in 2020, the ethics evaluation process was still not fully completed for IF and ERC, so we considered only the data up to the 2019 calls. As our datasets were based on the structured information available in the EthSR from SEP, they are partially incomplete. They did not include MSCA proposals for which an ethics assessment was performed by the Ethics Sector from DG RTD. These proposals include the cases with human embryonic stem cells or human embryos. The proposals that did not raise any ethics issues or were cleared by the ethics experts were not analysed further, as they correspond to the cases where the applicants either addressed correctly the ethics issues in their proposals or the proposed research did not raise any ethics issues. We focused on the main-listed proposals that were conditionally cleared, as they correspond to the cases where differences existed between what was declared by the applicants and what was later flagged by ethics experts.

The data from SEP were split into two datasets, the first containing the information about the ethics issues identified by applicants and ethics experts, and the second one containing the information about ethics requirements listed by ethics experts. For both datasets, we collected data about the proposal call year, scientific panel/domain, the host country (of the coordinating institution), and ethics issues/requirements categories. We also had the information related to ethics checks. The list of ethics issues as categorized by the European Commission is available in the Extended data (Buljan et al., 2021). The ethics issues are divided into 12 distinct categories, as listed in the Ethics Self-Assessment Checklist: Human embryos/foetuses; Humans; Human cells/tissues; Personal data; Animals; Third countries; Environment, Health & safety; Dual use; Exclusive focus on civil applications; Misuse; and Other ethics issues (European Commission, 2019a).

For ethics requirements dataset, there was no Exclusive focus on civil applications category, and the category General was merged with the Other category.

Ethical considerations

We worked on anonymized datasets, without insight about the actual content of the proposal, or the names of the applicants or ethics experts, so that issues of personal data protection were not applicable.

Data analysis

Ethics issues and requirements are presented as frequencies and percentages, by type of action (for MSCA) or grant (ERC), call year, and scientific domain. Ethics issues identified by the applicants and ethics experts are presented separately. The number of ethics issues/requirements per proposal is presented as median, with interquartile range (IQR). We calculated odds ratios for the probability that ethics issues in a specific category is identified by ethics experts vs. that it is identified by applicants.

We performed linear regression analysis, where the criterion was the number of ethics issues in a proposal identified by ethics experts, and with the type of action, year, host country, and number of issues in a specific category in a proposal identified by applicants as potential predictors. Host countries ware categorized into research low- and high-performing categories, based on a previously reported composite indicator (Pina et al., 2019). Research high-performing countries were Austria, Belgium, Denmark, Finland, France, Germany, Israel, the Netherlands, Norway, Sweden, Switzerland, and the United Kingdom, and low-performing countries were Bulgaria, Croatia, Cyprus, Czech Republic, Estonia, Greece, Spain, Hungary, Ireland, Iceland, Italy, Lithuania, Luxemburg, Latvia, Malta, Poland, Portugal, Romania, Serbia, Slovenia, Slovakia, and Turkey.

To determine the characteristics of main-listed proposals subject to ethics review, we collected the information about call year, type of action, number of requirements per proposal and research domain in logistic regression. For this analysis, we recoded ethics category groups by omitting Human embryos/foetus category due to small sample size and collapsing Dual use; Misuse; Other, Exclusive use on civil applications (linear regression only) and General (logistic regression only) categories into Other category.

The results are presented as odds ratios with 95% confidence intervals (CI) and Nagelkerke R squared. All analyses were performed using the R software environment for statistical computing and graphics (R Core Team, 2020).

Ethics issues identified by applicants and ethics experts in conditionally cleared main-listed proposals

Applicants identified 707 ethics issues in 417 proposals from ITN (85.9% out of all main-listed proposals), 4469 issues in 3054 proposals from IF (56.9%), and 2826 issues in 1465 proposals from ERC (55.7%) (Figure 2, Table S1 in Extended data (Buljan et al., 2021)). Ethics experts identified more than two times more issues than applicants: 1676 for ITN, 9695 for IF, and 6912 for ERC proposals. This trend was stable across different types of grants and call years (Figure 2).

Figure 2. Distribution of the number of proposals, ethics issues and ethics requirements among different grant programmes from 2016 to 2019.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission.

The median (Md) number of ethics issues identified by an applicant per proposal was 1 (IQR=0-3) for ITN, 1 (IQR=0-2) for IF, and 2 (IQR=1-3) for ERC, whereas ethics experts identified a median of 4 ethics issues (IQR=2-5) for ITN, 3 (IQR=2-4) for IF, and 4 (3-6) for ERC. This was true for all call years (Figure 3). The most frequently identified ethics issues categories overall were Humans; Protection of personal data; Animals; and Non-EU countries (Table S1 in Extended data (Buljan et al., 2021)). Applicants had higher odds to identify ethics issues related to Humans, Human cells/tissues, Protection of personal data, and Animals categories, whereas ethics experts were more likely to identify ethics issues in categories Non-EU countries and Environment, health and safety (Table S1 in Extended data (Buljan et al., 2021)).

Figure 3. Median number of ethics issues per proposal identified by applicants and by ethics experts, 2016-2019.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission.

With regard to the research domain, applicants from Life Sciences identified the most issues, with 3719 ethics issues (n=3719), followed by Social Sciences and Humanities (n=2916) and Physical Sciences and Engineering (n=1367). On the other side, ethics experts identified two times more ethics issues for proposals in Life Sciences (n=7652) and Social Sciences and Humanities (n=6055), and three times more ethics issues in Physical Sciences and Engineering (n=4576) (Table S2 in Extended data (Buljan et al., 2021)). The same was found for the median number of ethics issues per proposal (Figure 4).

Figure 4. Median number of ethics issues per proposal identified by applicants and by ethics experts, by research domain.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission. Soc/Hum – Social Sciences and Humanities, Phys/Eng – Physical Sciences and Engineering, Life sci – Life Sciences.

For Social Sciences and Humanities, the most prevalent categories were Humans, Protection of personal data and Non-EU countries, whereas in Life Sciences and Physical Sciences and Engineering, Animals, Human cells/tissues, and Environment, health and safety categories were more often identified by both the applicants and ethics experts (Figure 5). When observing the total number of ethics issues by category, the above six categories comprised most of the identified ethics issues, regardless of the type of grant (Figure 6). The greatest discrepancy in absolute numbers between project applicants and ethics experts was for the Non-EU countries and Environment, health and safety categories (Figure 6). Proposals from the Physical Sciences and Engineering were characterized by the highest number of issues related to Dual (military) use, identified by both the applicants and the experts (Figure 6).

Figure 5. Distribution of ethics issues categories identified by proposal applicants and ethics experts, by research domain.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission. Soc/Hum – Social Sciences and Humanities, Phys/Eng – Physical Sciences and Engineering, Life sci – Life Sciences.

Figure 6. Comparison between ethics issues identified by proposal applicants (grey) and ethics experts (white), by ethics categories.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission.

Regression analysis, where we assessed which proposal characteristics predicted higher number of identified ethics issues per ethics category by ethics experts, showed that it was more likely that the proposal will have significantly more ethics issues in the same category, identified by ethics experts when it was an ERC proposal, coming either from Life Sciences or Social Sciences and Humanities, or a newer proposal, as well as if it had issues related to the Protection of personal data, Non-EU countries, and Environment, health and safety categories, and had multiple issues identified by applicants in the same ethics category (Table S3 in Extended data (Buljan et al., 2021)). On the other hand, lower number of ethics issues in a same category identified by ethics experts were related to the proposals from MSCA funding schemes (both ITN and IF), from Physical Sciences and Engineering, from an earlier call, and had issues related to the Human cells/tissues, Animals or Other categories, as well as fewer ethics issues by ethics category identified by applicants (Table S3 in Extended data (Buljan et al., 2021)). However, the best predictor for the number of ethics issues identified by ethics experts was the number of ethics issues by ethics category identified by applicants, and it explained 19% of the variance of the criteria when independently entered in linear regression. Proposals which had multiple identified ethics issues per ethics category were more likely to have more ethics issues in a specific category identified by ethics experts.

Ethics requirements

The number of ethics requirements was greater than the number of ethics issues identified by ethics experts (3625 for ITN, 14249 for IF and 13846 for ERC) (Figure 2). ERC had the highest median number of ethics requirements per proposal (Md=8, IQR=4-14), compared to ITN (Md=6, IQR=3-13) and IF (Md=3, IQR=2-6) (Figure 7). The greatest proportion of ethics requirements was in the Humans, Human Cells/Tissues, Non-EU countries, and Environment, health and safety categories (Figure 8; Table S4 in Extended data (Buljan et al., 2021)). Proposals from Life Sciences and Social Sciences and Humanities domains had the greatest number of requirements (n=12544 and n=12377, respectively). There were 6799 requirements for proposals in Physical Sciences and Engineering (Table S5 in Extended data (Buljan et al., 2021)). All three types of grants were similar in the number of ethics requirements per proposal across domains (Figure 9). Although proposals from Life Sciences and Social Sciences and Humanities had a similar total number of requirements, most of the ethics requirements for Social Sciences and Humanities were in the Humans and Protection of personal data categories. Requirements for proposals in Life Sciences were linked to all ethics categories (Figure 10).

Figure 7. Median number of ethics requirements per proposal, 2016-2019.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission.

Figure 8. Ethics requirements identified by ethics experts, by ethics category.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission.

Figure 9. Median number of ethics requirements identified by ethics experts, by research domain.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission. Soc/Hum – Social Sciences and Humanities, Phys/Eng – Physical Sciences and Engineering, Life sci – Life Sciences.

Figure 10. Distribution of ethics requirements identified by ethics experts, by research domain.

ITN – Innovative Training Networks, IF – Individual Fellowships, ERC – European Research Council grants. The data are from years available in the SEP tool of the European Commission. Soc/Hum – Social Sciences and Humanities, Phys/Eng – Physical Sciences and Engineering, Life sci – Life Sciences.

The majority of requirements’ deliverables were scheduled to be fulfilled after the grant agreement signature: 14162 (99.4%) for IF; 3606 (99.5%) for ITN, and 3600 (26.0%) for ERC. The median deliverable’s month of implementation, when a requirement had to be fulfilled, was month 3 for IF (IQR=1-5, range=1-36), month 12 for ITN (IQR=12-12, range=1-48), and month 12 (IQR=12-12, range=0-55) for ERC.

Ethics checks were required for 43 ITN proposals (10.2%), 141 IF proposals (4.6%), and 185 ERC proposals (12.6%). The predictors of whether a proposal will involve an ethics check were a higher number of ethics requirements, being an MSCA ITN proposal, coming from Social Sciences and Humanities, and being a proposal from an older call year (Table S6 in Extended data (Buljan et al., 2021)). It was also more likely that a proposal will go for an ethics check if it had multiple ethics requirements in the collapsed Other category, and less likely if the multiple requirements were for Animals or Environment, health and safety categories (Table S6 in Extended data (Buljan et al., 2021)).

Ethics experts can require the appointment of an independent ethics advisor or an ethics advisory board, for proposals with complex or serious ethics issues, or when they consider that the applicants did not demonstrate sufficient proficiency in identifying or managing ethics issues. 6.2% of all conditionally cleared proposals had such a requirement (5.3% for ethics advisor and 0.9% for ethics advisory boards). ERC proposals had more such requirements: 14.9% proposals needed ethics advisor vs 1.2% of MSCA proposals, and 2.5% needed ethics advisory boards vs 0.3% for MSCA proposals. For MSCA proposals, there was no clear difference, except for the predominance of proposals from Social Sciences and Humanities among proposals needing an ethics advisor (8 out of 11 proposals) and Life Science proposals among ITN (12 out of 32 proposals). Half (51.1%) of the proposals that required some kind of ethics monitoring, either by ethics advisors or boards, during the project lifetime also required an ethics check.

Underlying data

OSF: Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals. https://doi.org/10.17605/OSF.IO/T765V (Buljan et al., 2021).

This project contains the following underlying data:

Extended data

OSF: Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals. https://doi.org/10.17605/OSF.IO/T765V (Buljan et al., 2021).

This project contains the following extended data:

Reporting guidelines

OSF: STROBE checklist for ‘Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals’. https://doi.org/10.17605/OSF.IO/T765V (Buljan et al., 2021).

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Disclaimer

All views expressed in this description are strictly those of the authors and may in no circumstances be regarded as an official position of the European Research Executive Agency or the European Commission.