An assessment of the extent to which the contents of PROSPERO records meet the systematic review protocol reporting items in PRISMA-P [version 1; peer review: 2 approved]

Background: PROSPERO is an international prospective register for systematic review protocols. Many of the registrations are the only available source of information about planned methods. This study investigated the extent to which records in PROSPERO contained the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P). Methods: A random sample of 439 single entry PROSPERO records of reviews of health interventions registered in 2018 was identified. Using a piloted list of 19 PRISMA-P items, divided into 63 elements, two researchers independently assessed the registration records. Where the information was present or not applicable to the review, a score of 1 was assigned. Overall scores were calculated and comparisons made by stage of review at registration, whether or not a meta-analysis was planned and whether or not funding/sponsorship was reported. Results: Some key methodological details, such as eligibility criteria, were relatively frequently reported, but much of the information recommended in PRISMA-P was not stated in PROSPERO registrations. Considering the 19 items, the mean score was 4.8 (SD 1.8; median 4; range 2-11) and across all the assessed records only 25% (2081/8227) of the items were scored as reported. Considering the 63 elements, the mean score was 33.4 (SD 5.8; median 33; range 18-47) and overall, 53% (14,469/27,279) of the elements were assessed as reported. Reporting was more frequent for items required in PROSPERO than optional items. The planned comparisons showed no meaningful differences between groups. Conclusions: PROSPERO provides reviewers with the opportunity to be transparent in their planned methods and demonstrate efforts to reduce bias. However, where the PROSPERO record is the only Open Peer Review


Introduction
Detailing the planned methods for conducting a systematic review in advance of commencing the review is essential in order to minimise a range of potential biases 1,2 . The plan, set out in a protocol, should ideally be made available in the public domain to facilitate transparency 3,4 . In addition, registration of key protocol details is encouraged as best practice in reporting guidelines 5,6 by publishers like the British Medical Journal (BMJ), Public Library of Science (PLoS), and BioMed Central (BMC), and is mandated in their instructions to authors by journals such as BMC Systematic Reviews, BMJ, BMJ Open, PLoS One, and National Institute for Health Research (NIHR) journals.
There are a number of options for putting systematic review protocols into the public domain, such as publication in open access journals like BMC Systematic Reviews and uploading to open data repositories like the Open Science Framework (OSF) (https://osf.io/registries/discover?q=protocols). PROSPERO (https://www.crd.york.ac.uk/prospero/) is a facility for registering key methodological details in advance of carrying out a review. Registration on PROSPERO requires completion of an internationally agreed minimum dataset for a systematic review protocol 7,8 . Registrants also have the option of uploading their protocol or providing a hyperlink to it.
PROSPERO remains the only free, open access registry of systematic review protocols, making it a single searchable source of the protocols of on-going and completed reviews. Uptake of registration has increased exponentially and by the end of 2019 there were over 60,000 registrations in PROSPERO. There is evidence that considerably more systematic reviews are registered in PROSPERO than have peer-reviewed protocols published. In 2016, 1058 records were accepted by PROS-PERO; in the same time period, only 404 published systematic review protocols were identified 3 . Another study reported identifying 20,814 non-Cochrane systematic review protocols from web scraping PROSPERO and bibliographic database searches. Of these, 924 were only published in journals, 807 were published in journals and registered in PROSPERO and 19,890 were only available as a record in PROSPERO 9 . There is further evidence from Ge et al (2018) that of the non-Cochrane reviews registered in PROSPERO, only 3% or 4% have a published protocol 9, 10 . This means that for a large number of reviews a PROSPERO record is likely to be the only source providing details of the planned methods.
Published protocols and registration records aim to provide transparency in the review process by allowing public access to the key pre-specified elements for the conduct of a review. One of the stated aims of PROSPERO is to facilitate comparison between planned review methods and reported results 8 . Such a comparison enables peer reviewers and other readers of the final review to assess for themselves the potential for bias in the findings. There is also a steadily growing body of research using PROSPERO records to assess the risk of biases in final review reports [10][11][12][13][14][15] . Given this reliance on the information provided in PROSPERO records, it is important to understand the level of detail provided in records. The focus of this study was on the stated aim of PROSPERO to reduce the opportunity for bias by enabling comparison of the completed review with what was planned in the protocol 8 .
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Protocols (PRISMA-P) were developed through expert consensus using internationally compiled datasets such as PROSPERO and SPIRIT 4,6 .
Key methodological aspects of a protocol are mandated for registration in PROSPERO; other items, mainly administrative fields, are optional 7,8 . Submissions for registration are not subject to any form of peer review or critical appraisal, they are simply checked for sense but not methodological rigor. Therefore, there is the possibility that PROSPERO records do not provide all the necessary information identified by the PRISMA-P guidelines to enable comparison with the completed systematic review. The registration record may be the only place where a priori methods are available for users, in particular peer reviewers, to check for potential issues such as selection, outcome reporting and publication biases. This study investigated the extent to which records in PROSPERO, where no protocol or other information was available, comply with each of the items for reporting of protocols set out in the PRISMA-P guidelines.

Methods
A random sample of PROSPERO registration records were assessed against the systematic review protocol reporting criteria set out in the PRISMA-P 2015 checklist 4 . Key methods are provided here with further details available in the protocol for this study, which was prepared and made publicly available on the OSF, 17 March 2020 (Extended data 16 ).

Study sample of PROSPERO records
A dataset of non-Cochrane PROSPERO records was provided by Metaxis, the software managers of PROSPERO. Records of reviews defined by the record holder as a health intervention registered on or between 1 January 2018 and 31 December 2018, were identified.
Cochrane reviews, reviews of animal studies, non-intervention reviews as identified in PROSPERO, i.e. Diagnostic accuracy, Prognostic factors, Prevention, Epidemiological reviews relevant to health and social care, Public health, Service delivery in health and social care, Methodological reviews, reviews of reviews, and synthesis of qualitative studies, were all excluded as PROSPERO and PRISMA-P were developed for reviews of interventions. Only records with no evidence from the registration record of other protocol related information, for example in a published protocol or other links in the PROS-PERO record, were included and we restricted the data set to those records with a single registry entry.
Records from the calendar year 2018 were used to allow time for dissemination and adoption of the PRISMA-P guidelines published in 2015. A sample of 20% of these records was randomly selected using simple random sampling for assessment against the PRISMA-P reporting criteria.

Assessment tool and scoring
The PRISMA-P checklist recommends 17 numbered items, with nine subdivisions, totalling 26 items be reported in a systematic review protocol 4 . Seven of the 26 items were excluded from the assessment as they would always or never meet registration requirements in PROSPERO. For example, registration is implicit for a record accepted in PROSPERO, and there is no field for author contributions or sponsor role so these would never be reported. The study assessment tool, developed specifically for this study as a Google Form, therefore contained 19 of the PRISMA-P items. Where the PRISMA-P description for an item specified more than one piece of information, the individual elements were listed as subsets of the items 4,6 . This list contained 63 elements to be reported.
Where an item was reported or not applicable, a score of 1 was assigned. Where the information was not reported this scored 0. The maximum possible overall score for the PRISMA-P listed items was 19 per record. Scores for the breakdown of individual elements within the items was also reported, the maximum possible score was 63 per record.

Assessment procedure
The researchers undertaking the assessments (AB, ASM, AM, SJ, SC, SG) familiarised themselves with both PRISMA-P papers 4,6 . All had previously received training in systematic review methods and/or authored at least one systematic review. The draft assessment form and accompanying guidance notes were revised and finalised during a training session and piloted with the aim of achieving greater than 90% agreement.
Two researchers independently compared the information provided in each PROSPERO record with the relevant items in the study assessment tool. Options for decisions were: Reported (information provided as per PRISMA-P requirements); Not reported (some or all information not provided); and, Not applicable (where an item was not relevant to an individual record, e.g. a meta-analysis was not planned).
Records were randomly assigned to assessors by first creating a list of the sampled record unique identification numbers and dividing the list into 14 blocks of approximately equal size, with each block being assigned a colour. A copy of this list together with the block configuration was then placed alongside the original list. Seven sub-lists were then created by randomly selecting a block from the first list and a block from the second list, such that blocks of the same colour were not in the same sub-list, and each colour appeared in two sub-lists. Each sub-list was then randomly assigned to an assessor.
It was not feasible to blind the researchers to the authors of registrations in PROSPERO. None of the assessors were authors of included registrations. On completion of the pilot assessments and the full set of records, disagreements were resolved through discussion or recourse to a third researcher.
The assessment form and the guidance notes are available on the OSF (Extended data 16 ).

Analysis
The primary outcome for this study was the compliance of PROS-PERO registration records to PRISMA-P reporting items. This was measured by the total mean score allocated by the two independent assessors to each of the 19 items assessed (maximum possible score 19) for each record and by the total mean score for the individual elements within items (maximum possible score 63). Overall scores for the assessed dataset, scores by the 19 PRISMA-P items and by the 63 elements were the planned outcome measures.
For the eligible 2018 records that were assessed and those not assessed, demographic data for month of registration, funding/sponsor, planned meta-analysis, number of authors, stage of review at registration, topic and country of review were to be reported. Comparisons to identify any association between records registered before or after screening started; whether a meta-analysis was planned or not; and whether a review was funded/sponsored or not and completeness of reporting of items were planned.

Deviations from protocol
During piloting of the assessment form, it became clear that it would not be possible to assess records for PRISMA-P item 5a Sources and 5b Sponsor. This would have required separating sources of financial support from sponsorship or any other form of support as reported in the single PROSPERO field, which was not possible. This item was therefore removed from the assessment form. Instead, a series of regular expression patterns was compared to the list of eligible records to identify those where the record contained any indication of funding/ sponsorship/support or indicated there was none. These data were used in the presentation of demographics and subgroup comparison.

Results
The PROSPERO dataset contained 5,313 records for reviews of health interventions first accepted in 2018 (excluding Cochrane and reviews of animal studies). Applying the other study inclusion/exclusion criteria resulted in 2,194 eligible registration records. The randomly selected sample of 20% for assessment included 439 records. During assessment, six records were excluded, for not meeting the inclusion criteria (4), being a duplicate (1) or no longer available on PROSPERO (1). Assessments were therefore carried out on 433 PROSPERO records. A flow chart of record selection is shown in Figure 1.
Agreement following initial piloting of the assessment form was 87%; after further discussions and revision of the assessment guidance notes and form a second pilot achieved 92% agreement. For all the records assessed, agreement between researchers was 90%, all differences were resolved through discussion or referral to a third researcher.
Demographic details of the sample of PROSPERO records selected for assessment and those not assessed are provided in Table 1. The number of authors listed ranged between one and 17, with the exception of a single record, included in the assessed sample, where 47 authors were listed. The eligible sample for 2018 included records from 67 different countries: 20 records listed two countries and 15 listed between three and nine countries involved in the review. There were no substantial differences between the data sets in the month of registration; whether any details of funding and/or sponsorship were provided; whether a meta-analysis was planned or not; the number of authors listed per record; stage of review at registration; topic of review or country involved in undertaking the review.
None of the PROSPERO records assessed against the eligibility criteria reported on all elements in each of the items recommended for a systematic review protocol in the PRISMA-P guidelines. The mean total score for individual PROSPERO records, where 1 point was gained for each of the 19 items in the PRISMA-P checklist, was 4.8, the standard deviation 1.8, the median 4, and range 2 to 11. Considering all items across all the assessed records, only 25% (2081/8227) of the items were scored as reported.
The mean total score for individual PROSPERO records where 1 point was gained for each of the 63 elements of the PRISMA-P reporting guidelines was 33.4, the standard deviation 5.8, the median 33 and the range 18-47. Overall, 53% (14,469/27,279) of the elements were considered as reported.

Scoring for 19 PRISMA-P items
The highest scoring item was PRISMA-P 1b which requires the protocol to be identified as to whether it is an update of a review; the high score was the result of this being a notapplicable item for 423 (98%) of the 433 records (Table 2). Eligibility criteria (study design, setting, population, intervention,
The scores by PRISMA-P item and by breakdown of items are presented in Table 2. The full dataset with assessment outcomes and scores for individual records, and the subgroup analyses scoring are available on the OSF (Underlying data 16 ).
Scoring for 63 elements of the PRISMA-P items The score for some of the 19 items was reduced as a result of just one or two of the constituent elements being omitted from reports while others were relatively regularly identified.
Although overall the review question (item 7) was not found to contain all the expected elements, most did specify the elements of population (397, 92%) and the intervention (416, 96%) and just over half included the outcomes (237, 55%). The comparator was less frequently included (142, 33%); this may have been because of the intention of the review but where this was clear, the item was scored as not applicable (6%).
Information sources (item 9) was scored as completed in only two records (1%) overall; however, for the individual elements 431 (99%) did name the electronic databases to be searched, 289 (67%) said whether they planned to search study registries, and 238 (55%) indicated search dates. In item 10, provision of a draft search strategy (91, 21%) or search terms (100, 23%) was poor; but restrictions such as to English language papers were reported in 332 (77%).

(1) 5 (0)
Health inequalities/health equity Infections and infestations 22 (5) 97 (6) International development 0 (0) 2 (0) Mental health and behavioural conditions 51 (12) 129 (7) Musculoskeletal  Reporting of item 13, outcomes, scored badly overall (3, 1%) as, although the outcomes were included in most records (Primary 418, 97%; Secondary 430, 99%) only 8 (2%) were assessed as having provided a rationale for their choice of outcomes. Similarly, in item 14, the absence of information on how the risk of bias would be used in the synthesis, detracted from the high rate of inclusion of risk of bias tools and use. Reporting of the details for a quantitative synthesis, item 15b, had one element with a very low score (handling missing data, 14, 3%), the other six elements scored between 89 (20%) and 204 (47%).
In three items, the overall score reflected the general picture from the included elements. In item 6, rationale, both the reason for undertaking the review and the context were infrequently identified. PRIMSA-P items 16, meta-bias(es) and 17, confidence in cumulative evidence, were rarely reported. Only context is classified as optional information in PROSPERO, the remainder of these elements are not explicitly requested.
There appears to be a trend towards higher frequency of reporting of elements that are mandatory in PROSPERO, for example, in the eligibility criteria (item 8) and risk of bias (item 14). The trend is also seen in item 13, the required specification of primary and secondary outcomes, both frequently reported, but with a drop in specifying measures, which was optional.

Subgroup comparisons
The subgroup comparisons investigated the stage of review at registration; whether or not information was reported on source of funding, sponsorship or support and where none was indicated; and whether or not the relevant box in the registration form had been ticked to indicate a meta-analysis was planned.
There were no differences in total scores for the 19 PRISMA-P items or the 63 elements, between those records registered before screening against eligibility criteria had started and those records registered after screening had commenced. This held true for the mean, standard deviation, median and range of scores.
A 6% difference was seen in the total score achieved for the meta-analysis (23%) vs no meta-analysis (29%) groups in the assessment of the 19 PRISMA-P items. The difference was reduced to 2% when considering the breakdown of 63 elements within the reported items (52% vs 54%). At both item and element level, the group of records with no planned meta-analysis scored slightly higher, but with a higher standard deviation from the mean and wider range of scores achieved.
Across all results for both the 19 items and 63 elements, the group with funding, sponsorship or support, scored slightly higher than those not receiving funding, sponsorship or support.
The results of the subgroups investigated are presented in Table 3. The subgroup scores by individual PRISMA-P reporting item are available on the OSF (Underlying data 16 ). We present the scores by the 19 PRISMA-P items and by the breakdown of 63 elements for the ten countries and topics with the highest number of assessed records, and for number of authors listed in Table 4. None of these factors appear to have a marked influence on the number of PRISMA-P items or elements reported in PROSPERO records.

Discussion
Publication and registration of a systematic review protocol provides transparency in the review process, allowing readers to see the efforts made to minimise biases and where biases may still have influenced the final review findings. There is empirical evidence that few of the protocol registrations in PROSPERO Using a random sample of 433 PROSPERO records from 2018, two researchers independently assessed the frequency of reporting of 19 PRISMA-P items, with 63 individual elements. The results show that while some key methodological details are relatively frequently reported, much of the information recommended in PRISMA-P is missing. Reporting was unsurprisingly more frequent for items that are mandatory in PROSPERO than those that are optional. Comparisons by stage of review at registration, whether meta-analysis was planned and whether funding or sponsorship was reported showed no meaningful differences between groups. The slight difference between groups with a planned meta-analysis or none may be because in PRISMA-P more details are specified for the reporting of a meta-analysis than for a descriptive, narrative or qualitative analysis.
Eligibility criteria and type of analysis planned were most frequently reported and are all separate required fields in PROS-PERO. However, study selection process, which is optional, was also a higher frequency reported item. This may be explained by considering that some elements of items, such as eligibility criteria, study selection and risk of bias have what might be considered a standard, recognisable format that facilitates reporting.
Other items need a more nuanced approach underpinned by a clear understanding of systematic review methods, and therefore may be associated with being less frequently reported due to a lack of confidence or experience with these aspects of review methods. For example, how risk of bias will be used in the synthesis, data handling in a meta-analysis, meta-biases and confidence in cumulative evidence, all had low scores. Part of the problem may be the uncertainty of what the searches will find when designing a systematic review but needing to know so the design is appropriate. For example, the intention may be to perform a meta-analysis, this may not be possible once the studies for inclusion have been identified. While, both PROS-PERO and PRISMA-P acknowledge that protocols are iterative documents and may need to be amended, changes should be documented, justified and the stage of review at the time of the amendment made clear. Therefore, it is better to record alternative options for activities such as how data will be analysed and the conditions for selection of option when finalising the protocol.
Differences in frequency of reporting may also reflect where researchers considered items to be less or more important than others. For example, naming the software used for data management may not be seen as crucial, whereas the eligibility criteria and approach to synthesis are.
There are strengths and limitations to this study. The assessed sample of 433 records was representative of all the eligible 2018 non-Cochrane intervention reviews registered in PROS-PERO. As a result, the findings may reasonably be generalised to other registrations of healthcare interventions, but not necessarily other types of registered reviews excluded from our sample.
PRISMA-P is a reporting guideline and not a rating scale, so judgements about whether sufficient information had been provided for some items carried a degree of subjectivity. The assessment guide and form developed for the study aimed to maximise objectivity but in accordance with PRISMA-P did not weight importance of items. Although two researchers independently carried out the assessments, achieving an overall agreement rate of 90%, subjectivity was minimised but not eliminated.
PROSPERO was developed in 2011 to record key protocol details and does not necessarily accord with everything subsequently recommended in the 2015 PRISMA-P reporting guidelines. Some registration items are mandatory and others optional. However, this study looked at records that had no other protocol output and arguably should therefore have provided PRISMA-P level detail. The evidence that protocol details are only available in PROSPERO for around 96% of non-Cochrane reviews makes the infrequency of reporting of items a concern 9,10 .
Based on the findings of other studies, promoting improved reporting of protocol details may help increase the quality of systematic reviews 17,18 .
Protocols are iterative documents and even after a review has started there may be legitimate reasons for amendments. Such changes should and can be reported in a registration record, with their justification and timing. Just over two thirds of PROSPERO records have more than one version (Figure 1). . While focussing on single entry records to be certain that any changes were not made after completion of the review this may have excluded records where more complete information was added to the record over time at key points in the review process.
This study simply looked at whether items were reported and not at the level of detail or suitability/appropriateness of the planned methods. The option of 'partially reported' could have been used at assessment but was avoided to minimise subjectivity. The focus was on simply establishing whether items were reported or not. The assessors focussed on whether the information was reported or could reasonably be inferred from what was reported. Assessing the quality of planned methods in protocol registrations needs to be the subject of further research.
This study shows that there is work to be done to promote the complete reporting of items recommended in the guidelines for systematic review protocols when the registration in PROSPERO is the only place they can be accessed. This is in line with other research that has identified issues with the quality of reporting, publication and outcome reporting biases in systematic review protocols in general 3,9,11,13,19,20 . As proposed in the PRISMA-P statement paper, actions and potential benefits to encourage adherence to PRISMA-P will take a joint effort on the part of a host of stakeholders, including reviewers, registries, and journal editors 5,21 .

Conclusions
PROSPERO provides reviewers with the opportunity to be transparent in their planned methods and demonstrate efforts to reduce bias. However, where the PROSPERO record is the only available source of a priori reporting, there is a significant shortfall in the items reported, compared to those recommended in PRISMA-P. This presents peer reviewers and others wishing to assess the validity of the final review with challenges in interpretation. PROSPERO records are not peer reviewed or assessed for methodological quality, it is the responsibility of those registering their review to complete the registration form fully or provide access to a complete protocol. There are several areas requiring particular attention when completing the registration form. These include explaining the rationale for undertaking the review in the context of what is known; providing information sources beyond a list of databases to be searched; and reporting reproducible process methods for data management, study selection and risk of bias assessment. In addition, defining variables for data extraction, how specified outcomes will be measured, and the planned analyses, with criteria for undertaking a quantitative synthesis should all be included in detail.
This study only looked at whether recommended items were reported or not in PROSPERO records. Further research is needed to assess the quality of the planned methods in systematic review protocol registrations.