A review of the Cochrane COVID-19 Study Register reveals inconsistency in the choice and measurement of SARS-CoV-2 infection outcomes in prevention trials [version 1; peer review: 1 approved]

Background: Multiple studies are evaluating how to prevent SARS-CoV-2 infection. Interventions are wide ranging and include vaccines, prophylactic drugs, public health safety measures, and behavioural interventions. Heterogeneity in the outcomes measured and reported is leading to research waste and inefficiency, slowing worldwide identification and implementation of effective methods to prevent infection. A core outcome set (COS) for studies of interventions to prevent SARS-CoV-2 infection has recently been developed, identifying infection as a critical outcome to measure. This paper examines how SARS-CoV-2 infection outcomes are measured in registered COVID-19 prevention trials and considers how this can be improved. Methods: We searched the Cochrane COVID-19 Study Register to identify and review SARS-CoV-2 infection outcomes in prevention trials, including the rationale for choice of outcome measurement. We included phase 3 and 4 trials of COVID-19 prevention interventions. Early phase trials and studies relating to the transmission, treatment or management of COVID-19 were excluded. Results: We identified 430 entries in the register, of which 199 unique prevention trials were included across eight settings and 12 intervention types. Fifteen (8%) trials did


Introduction
Coronavirus disease 2019 (COVID- 19), resulting from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, has significant morbidity and mortality.Addressing human-to-human transmission of SARS-CoV-2 is a global research priority. 1Research to fully understand the modes of transmission of SARS-CoV-2 is ongoing.Studies to prevent the spread of SARS-CoV-2 are increasing, with more than 1000 studies registered between December 2020 and February 2021. 2 Interventions are wide ranging, including public health measures such as lockdown of countries/cities, closure of schools and public places, social distancing measures and handwashing, interventions in primary and secondary care practice such as the use of personal protective equipment (PPE) and increased time for appointments/interventions, alongside the development of effective vaccines and drugs to prevent disease.
4][5] Whilst studies primarily focus on mitigating the risk of contracting COVID-19, they may measure other outcomes relevant to the population of interest and type of intervention, for example, educational outcomes in studies of school closures 6 and tolerance/acceptability in studies of PPE use in secondary care. 5However, unnecessary heterogeneity in the outcomes measured and reported in COVID-19 prevention trials leads to research waste and inefficiency.For example, a recent commentary highlighted the rarity of randomised trials of public health interventions for COVID-19, which is likely due to the need for unattainably large sample sizes. 7Fretheim argues that underpowered trials in public health should be regarded as contributions to the larger body of evidence on COVID-19.However, for underpowered trials to be collectively assessed and synthesised in systematic reviews they need to be measuring the same outcomes.
A solution to this issue is to develop a core outcome set (COS), which is defined as an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical research studies in a specific area of health care to address inconsistencies in outcome measurement. 8The COS-COVID-P study, in collaboration with an international and multidisciplinary committee of experts and public contributors, has developed a COS for studies of interventions to prevent SARS-CoV-2 infection.During the COS development process, online workshops were held to consider the evidence on COVID-19 prevention and the issue of outcome heterogeneity.More than 70 key international stakeholders, including clinicians, researchers, public representatives, methodologists, systematic reviewers, guideline developers, and regulatory agency representatives shared their expert opinions.Agreement on the final COS was reached in August 2020.Two outcome domains were identified as being essential to all COVID-19 prevention studies: infection and interventionspecific harms. 9How to measure these outcomes now requires consideration and led to this paper.SARS-CoV-2 infection can be measured through laboratory testing, symptom assessment or a combination of both. 2 However, there is no single standalone diagnostic test nor agreement on diagnostic symptoms. 10,11Thus, there is a pressing need to consider how to assess SARS-CoV-2 infection uniformly across studies to optimise data reporting and synthesis.The aim of this study is to review SARS-CoV-2 infection outcomes in registered COVID-19 prevention trials and examine the reported rationales for choice of outcome measurement.Our intention is that this will inform optimal methods for the design of future studies.

Search strategy
For the purposes of the review presented here, we searched the Cochrane COVID-19 Study Register https://covid-19.cochrane.org/on 24 November 2020.The Cochrane COVID-19 Study Register is a freely available, annotated reference collection of primary research studies on COVID-19, including interventional, observational, diagnostic, prognostic, epidemiological and qualitative designs.The register is continually updated to support rapid evidence synthesis, and includes studies from multiple data sources including Clinicaltrials.gov, the Cochrane Central Register of Controlled Trials (CENTRAL), Embase, medRxiv, PubMed and the WHO International Clinical Trials Registry Platform (ICTRP).We restricted our Cochrane COVID-19 Study Register search results by study type ("interventional"), study design ("parallel/ crossover"), intervention assignment ("randomised") and study aim ("prevention").The Cochrane COVID-19 Study Register defines prevention studies as those aiming to assess one or more interventions for preventing the development of a specific disease or health condition.

Eligibility criteria
We included phase three and four trials of prevention interventions for COVID-19.We excluded phase one and two trials, including those that used other terms to imply that they were early phase (e.g."explorative"), because these trials focus on safety, rather than the effectiveness of an intervention.Studies relating to the transmission, treatment or management of COVID-19 were also excluded.Studies identified from the Cochrane COVID-19 Study Register search were split equally between five experienced systematic reviewers (KA, SD, SG, NH, RM), who assessed the eligibility of their cohort of studies according to these eligibility criteria.

Data extraction
Data from eligible trials were extracted using a piloted, standardised data extraction spreadsheet, with the work split equally between five experienced systematic reviewers (KA, SD, SG, NH, RM).In cases of any ambiguity or doubt, queries were assessed by SD.Any queries were discussed with the whole team who collectively reached agreement on how to progress.The following data were extracted for each study: intervention type (e.g.drug, vaccine, behavioural), intervention (as described verbatim), setting (i.e.population of interest), SARS-CoV-2 infection outcomes (as described verbatim), SARS-CoV-2 infection outcome definition (i.e.how SARS-CoV-2 infection would be measured), whether or not the SARS-CoV-2 infection outcome was specified as being a primary or secondary outcome, and the rationale provided for SARS-CoV-2 infection outcome measurement selection.In addition, the reviewers recorded whether the study record related to a registry entry, trial protocol or the main report.If the registry entry record provided a link to a study protocol, this protocol was also reviewed, and the relevant data extracted.

Data analysis
Extracted data were summarised according to the number of SARS-CoV-2 infection outcomes per trial.For trials that included one SARS-CoV-2 infection outcome, data were summarised according to the single outcome definition specified in each trial (i.e.testing, symptoms, testing or symptoms, testing and symptoms).For trials that included multiple SARS-CoV-2 infection outcomes, data were summarised according to each outcome definition combination.For example, some trials may have measured one SARS-CoV-2 infection outcome using testing alone, a second SARS-CoV-2 infection outcome by symptoms alone and a third SARS-CoV-2 infection outcome by testing and symptoms.In addition to analysing the data by the overarching outcome definition (e.g.testing), we also summarised the data according to the specific underlying definition.For example, a trial would be summarised as having two versions of testing if they specified that one SARS-CoV-2 infection outcome was determined by reverse-transcriptase polymerase chain reaction (RT-PCR), and another SARS-CoV-2 infection outcome was determined by serology (antibodies, seroconversion, seroprevalence).The categorisation of outcome definitions was based on the verbatim text available in the registry records.Analysis was carried out using Stata version 16.1.

Search results
Our searching of the Cochrane COVID-19 Study Register returned 430 records that were screened for eligibility, of which 231 were excluded.A total of 199 unique prevention trials were identified (Figure 1).

SARS-CoV-2 infection outcomes
Of the 199 trials, 15 (8%) did not include any SARS-CoV-2 infection outcomes.These 15 studies predominantly evaluated behavioural (n = 8, where interventions were 'STOP touching your face' mindfulness online training programme; Brief informational infographic; E-E video about COVID hygiene; Mental Imagery; Digital health literacy intervention; e-learning module; Telepsychoeducation with personalized videos; combined video display and live demonstration as training methods to healthcare providers for donning and doffing PPE) or PPE/physical barrier (n = 3) interventions, with the remaining four trials evaluating social distancing, dietary supplements, a medicinal plant and homeopathy.Examples of outcomes recorded in trials that did not include a SARS-CoV-2 infection outcome included frequency of face-touching behaviour in a behavioural intervention trial 12 and contamination of any part of the base clothing or exposed skin of the upper body in a PPE trial. 13total of 268 SARS-CoV-2 infection outcomes were identified across 184 (92%) trials, with infection measured as a primary outcome in 149 (81%) studies.Table 2 provides the outcome definitions recorded.We categorised outcome definitions using the verbatim text available in the registry records.In 118 (64%) trials, a single infection outcome was included.Fifty (27%) trials included two infection outcomes, 14 (8%) included three infection outcomes and two (1%) included four infection outcomes.Examples of multiple infection outcomes within the same trial are provided in Table 2. Table 3 lists the specific outcome definitions across the 184 trials.
Forty-five (25%) trials defined infection in non-specific terms (e.g."COVID", "SARS-CoV-2 infection", "infected", "confirmed"), without specifying how infection would be/was determined (i.e. via a specific test to be undertaken or symptoms to be recorded).In 32 of these trials (17% of all included trials), the non-specific infection term was the only infection outcome listed.Figure 2 displays the number of trials with non-specific COVID outcomes by month of registration.Of the 45 trials with non-specific COVID outcomes, 23 (51%) trials were registered within the first three months (March to May 2020) of the COVID-19 outbreak being declared a global pandemic.
Testing was the only means employed for determining infection in 57 (31%) trials, whereas relying on symptoms alone to define infection occurred in 16 (9%).All other trials included multiple infection outcomes, defined in different ways (Table 2).Where one of the infection outcomes was to be determined by testing alone (n = 96), this was most commonly by RT-PCR (n = 36, 20%) and serology (n = 22, 12%) tests.Where infection was defined on the basis of symptoms alone (e.g.symptomatic COVID-19) (n = 45), the number of symptoms and definitions varied (Tables 4 and 5).Some trials included outcomes defined by combinations of testing and/or symptoms: 13 (7%) using either testing or symptoms to determine infection; 46 (25%) specifying both testing and symptoms must be positive to define infection (e.g.RT-PCR and symptomatic).

SARS-CoV-2 infection outcomes by setting and intervention type
Tables 6 and 7 detail the use of various SARS-CoV-2 infection outcome definitions by setting and intervention type, respectively.

Rationale provided for SARS-CoV-2 infection outcome selection/definition
Of the 184 trials that included SARS-CoV-2 infection outcomes, seven provided a rationale for their choice of outcome selection and/or measurement (see Table 8).Of these seven studies, this was often to justify their specific choice of testing, with one drug prophylaxis trial explaining that "testing and criteria for diagnosis COVID-19 is likely to rapidly evolve over the course of this trial and will vary internationally" 14 and a school closure trial basing outcome measures on routinely collected testing data to ensure "feasibility, facilitate speed, and limit costs". 15A drug prophylaxis trial in  1 Seven of these studies also included others in close contact with COVID cases (two of which also included high risk groups); a further two of these studies included others with potential exposure to COVID (one of which also included police).
healthcare workers argued that relying on symptoms consistent with infection without laboratory confirmation raises "concerns of type II error from asymptomatic participants". 16Conversely, another drug prophylaxis trial justified the decision to use symptoms given issues of access to testing, "Given limited availability of outpatient PCR testing in many jurisdictions during our study period, particularly in April 2020, probable Covid 19 based on Covid-19-compatible symptoms was included in the composite primary endpoint". 17
The heterogeneity of outcome measurement identified in the current study is not uncommon.A recent methodological systematic review revealed significant heterogeneity in the outcome measurement instruments used in trials evaluating therapeutic interventions for COVID-19, thus highlighting the need for greater consistency. 18However, such heterogeneity is not restricted to COVID-19 research studies.A multiplicity of outcome measurement instruments has recently • serology AND asymptomatic 1 (0.5) 1 (0.4) • serology AND symptomatic 2 (1.1) 2 (0.7) • (serology AND symptomatic) OR RT-PCR 1 (0.5) 1 (0.4) • virologically-confirmed AND symptomatic 2 (1.1) 2 (0.7) • test AND symptomatic 4 (2.2) 4 (1.5) • molecular confirmation AND symptomatic 1 (0.5) 1 (0.4) 1 Includes one study that specified individual symptoms only. 2 Note that "symptomatic" includes references to clinical confirmation.infection outcomes used when registering and reporting their trials, to improve understanding of the context of their research.The trial registration should include both the rationale for outcome measurement, along with details on how infection will be analysed.In addition, we understand trialists will collect as much data as they consider relevant, but we recommend they develop a coherent statistical analysis plan (SAP) to ensure transparency and reproducibility. 23 the current review, we found two thirds of the studies evaluating behaviour interventions, such as those intended to increase handwashing or SMS text prompts to wear masks, did not include any SARS-CoV-2 infection outcomes.Trials of simple behavioural interventions may be unlikely to measure infection, as their purpose is to modify an individual's behaviour rather than to directly prevent an individual from acquiring COVID-19.This is in contrast with those of complex interventions that include a behavioural component (e.g.donning standardised PPE versus specialised PPE; or hand washing with soap versus using alcohol wipes) where the overarching aim is still likely to be prevention of SARS-CoV-2 infection.
Alongside SARS-CoV-2 infection, the COS-COVID-P study also led to the inclusion of intervention-specific harms in the COS for all COVID-19 prevention research studies. 9The current review did not examine this aspect.Determining these harm outcomes requires agreement amongst those with expertise in the specific interventions being tested.It may be that the harms are similar within certain intervention types, e.g.drug prophylaxis and vaccines.However, expert opinion will be necessary to determine the relevance of such an approach, but consistency will be important to avoid research waste and to facilitate evidence synthesis in which the results of studies of similar interventions are compared, contrasted and, where appropriate, combined.There may also be a need for COS for studies in specific settings.For example, a COS evaluating interventions for the prevention of COVID-19 transmission in care homes (COS-COVID-PCARE) is Whilst the authors describe how future trials can be improved through reporting rationale, there is little acknowledgement about how this research question/aim is not addressed in the results of this paper.Highlighting the limitations of using a systematic review method in this way would be beneficial and perhaps suggesting alternative methods in the discussion (i.e.stakeholder consultation) in similar research.
My other query would be in regards to the behavioural interventions, if you are looking at outcomes for interventions that prevent an individual from acquiring covid-19 (infection outcomes) would it not be best to remove the behaviour modification interventions from this review or present them separately, it seems to me this should have perhaps been an inclusion/exclusion criteria.I would assume this may reduce some of the heterogeneity in outcomes also.

Are sufficient details of methods and analysis provided to allow replication by others?
The methods are detailed and analysis thorough and replicable.

If applicable, is the statistical analysis and its interpretation appropriate?
Statistical analysis and interpretation are appropriate.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?
Conclusions are supported by the results.My only concern is that whilst the authors argue the limitations with the trials reviewed there is not much discussion about the limitations of this study, some I have mentioned above.The heterogeneity in reporting infection outcomes, could be related to studies being conducted before testing was established in many countries and acknowledging that the studies reviewed were conducted during crisis and that future studies may provide better rationale for outcomes as we move forward.Also, advice/suggestions about how trialists could improve outcome decisions in pandemic situations (when the development of adequate infection testing is developing) to improve homogeneity of outcomes would be a useful addition to the literature.

Is the study design appropriate and is the work technically sound? Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?Yes Are all the source data underlying the results available to ensure full reproducibility?Yes Are the conclusions drawn adequately supported by the results?Yes Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Core outcome set development I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Figure 2 .
Figure 2. Trials with ≥1 non-specific SARS-CoV-2 infection outcome by month of registration (out of 184 trials with at least one SARS-CoV-2 infection outcome).Left-hand axis -number of trials.Right-hand axis -% of total number of trials.

Table 2 .
SARS-CoV-2 infection outcome definitions, by number of infection outcomes per trial.

Table 8 .
24tionale provided for definition of SARS-CoV-2 infection outcomes.Rationale https://clinicaltrials.gov/show/NCT04346329 "For this, once the informed consent form is signed, the molecular test for the diagnosis of SARS-CoV-2 infection by RT-PCR will be carried out every 4 days in order to determine as closely as possible the moment the participant become positive."https://clinicaltrials.gov/ct2/show/NCT04519125 "To determine the occurrence of infection with the virus the study will use both molecular tests that detect the presence of viral genes in respiratory secretions, and serological tests that detect the response of the immune system to the virus."https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2020-001448-24 "Testing and criteria for diagnosis COVID-19 is likely to rapidly evolve over the course of this trial and will vary internationally.Therefore, the following pragmatic definitions will be applied:o Confirmed positive test for COVID-19: either a laboratory test (performed according to each participating hospital's local protocols) or a computed tomography (CT) thorax scan (based on individual radiologist interpretation and diagnosis) that confirms COVID-19 diagnosis.oConfirmednegativetestfor COVID-19: a laboratory test (performed according to each participating hospital's local protocols) that is negative for COVID-19 diagnosis."https://pubmed.ncbi.nlm.nih.gov/33001138"Arecentrandomizedtrial for postexposure COVID-19 prophylaxis with a 5-day course of hydroxychloroquine did not demonstrate clinical benefit.However, the composite primary outcome measure for this study included symptoms consistent with infection without laboratory confirmation; most patients did not have assessment of SARS-CoV-2 infection by reverse-transcriptase polymerase chain reaction (RT-PCR), raising concerns of type II error from asymptomatic participants.We sought to test the hypothesis that administering daily hydroxychloroquine would prevent SARS-CoV-2 infection in hospital-based HCWs over 8 weeks of exposure via RT-PCR testing of nasopharyngeal (NP) swabs and serologic antibody testing from participants at baseline, 4 weeks, and 8 weeks of treatment."https://clinicaltrials.gov/show/NCT04328441 "The aim of our trial is to ensure the continuity of health care, and this is the reason for this pragmatic primary endpoint.Moreover, BCG vaccine is expected to reduce all-cause infections and not only COVID-19.Moreover, this was a non-funded study, precluding intensive personal followup, and the measurement of absenteeism as this was considered relevant, and immediately obvious....One of the secondary endpoints in the HCW trial is documented COVID-19 infection.It is important to keep in mind that health workers in the Netherlands are structurally examined for the presence of COVID-19 in case of any respiratory symptoms or in case of fever.Therefore, we expect a high detection rate."https://pubmed.ncbi.nlm.nih.gov/33068425"Givenlimitedavailability of outpatient PCR testing in many jurisdictions during our study period, particularly in April 2020, probable Covid 19 based on Covid-19-compatible symptoms was included in the composite primary endpoint."https://pubmed.ncbi.nlm.nih.gov/33068425"Toensurefeasibility,facilitatespeed, and limit costs, we aim to base our outcome measures on routinely collected data, to the extent possible.Testing for COVID-19 takes place continuously and consistently across the country.All suspect cases have access to testing and laboratories use valid tests meeting quality assurance requirements."currentlyunderdevelopment.Building on the COS-COVID-P study, the COS-COVID-PCARE study is proposing to develop a supplementary specific module for COVID-19 prevention in care homes.24ConclusionWehave described the variation in the selection and measurement of SARS-CoV-2 infection outcomes, with illustrated examples from on-going trials across various settings and interventions.As knowledge about COVID-19 increases over time, this is leading to improvements in the design of studies and reporting.Nevertheless, our findings highlight the need for clarity, transparency and standardisation of measures of SARS-CoV-2 infection.It is now timely to highlight these issues, to improve the collection of infection data to inform key public health decisions.