Developing a core outcome set for a congenital abnormalities surveillance programme in Rwanda – a Delphi consensus study

In 2015 it was reported that approximately 300,000 Background: newborns die within four weeks of birth every year, worldwide, due to congenital anomalies. This represents approximately 11% of neonatal deaths. This has led scientists, clinicians and public health authorities to establish congenital abnormality registries (CARs). There is currently no CAR in Rwanda. In establishing such a registry, it was determined that the first step was to identify the core outcome set (COS) (or minimal data-set) of variables and outcomes for the registry to ensure that the final results are meaningful and employable. This study aimed to use Delphi consensus methods to identify a methodologically robust COS for a congenital abnormalities surveillance programme in Rwanda. A three-round, modified Delphi study was undertaken between Methods: April and June 2017. Round 1 was a literature and internet search followed by an open and closed question round with experts in Rounds 2 and 3, respectively. An initial draft COS of 136 outcomes was created from a review of Results: 15 African studies and 14 international repository tools including the European Surveillance of Congenital Anomalies and the World Health Organization surveillance guidance. In total, 36 and 34 participants took part in Rounds 2 and 3, respectively. A total of 32 new outcomes were added by participants in Round 2. 103 outcomes met the pre-defined consensus criteria and made up the final COS in Round 3. This is the first core outcome set for a congenital abnormality Conclusions: surveillance programme in an African nation identified in the literature. The next stage is to field-test the surveillance programme using passive case-finding in teaching hospitals in Rwanda. 1-3 1 3


Introduction
Congenital abnormalities are defined as malformations of organs or body parts during development in utero, present at birth and are therefore of prenatal origin 1,2 . The prognosis of neonates with congenital abnormalities is often poor 3 . Annually, approximately 300,000 newborns die within four-weeks of birth, worldwide, due to congenital anomalies, representing approximately 11% of neonatal deaths 4 . The most common congenital abnormalities are congenital heart abnormalities, neural tube defects and Down syndrome 4 . Causes of congenital abnormalities are genetic, environmental or idiopathic. More than 90% of these newborns are born in low-and middle-income countries (LMICs) 5 . In resource-limited settings, despite the balance of burden of disease, there is limited epidemiological data about the rate, risk factors and types of congenital abnormalities. In Rwanda, data monitoring is already being undertaken via the Integrated Health Management Information System (HMIS) 6 . This is a nationwide data-surveillance programme, with health facilities reporting the total number of births with congenital anomalies, but no detail of individual cases. Though a positive start it doesn't capture rich enough data for meaningful objectives to be met.
When epidemiological data has been presented the outcomes described are commonly not consistent. The importance of prevalence of congenital abnormalities has lead scientists and public health authorities to establish congenital abnormality registries (CARs). These surveillance systems based on highquality epidemiological data are required to identify preventable causes and for policymakers to plan care provisions 7 . A large number of CARs have already been established, predominantly in high-income countries. The European Surveillance of Congenital Anomalies (EUROCAT) initiative is a good example of this. EUROCAT aims to carry out epidemiologic surveillance of congenital anomalies in Europe 8 . There are several objectives to EUROCAT initiative, including; i) provision of essential epidemiologic information on congenital anomalies in Europe, ii) facilitating the early warning of teratogenic exposures, iii) evaluating the effectiveness of primary prevention, to assess the impact of developments in prenatal screening, and iv) acting as an information and resource centre regarding clusters or exposures or risk factors for concern 8 . Finally the EUROCAT initiative aims to act as a catalyst for the setting up of registries throughout Europe collecting comparable, standardized data 8 . These are admirable objectives for the European continent, and resource-limited settings, with significant burdens of congenital abnormalities, should aim to establish registries and surveillance programmes with equally ambitious goals. The World Health Organization (WHO) have also given high-quality guidance on congenital abnormality surveillance programmes 2 . This guidance includes similar objectives to EUROCAT and also gives further objectives such as detecting clusters (outbreaks) of congenital anomalies 2 .
When undertaking research or creating registries, the outcome measures should be valid, reliable and feasible 9 . That is, outcomes should adequately meet the criteria of truth, be sensitive to change and be easily applied and interpreted. They should also be relevant to the setting and the stakeholders who will engage with the data. More emphasis is being placed on ensuring the high quality of outcomes measured in research and surveillance programmes. If research has not been conducted to identify the most appropriate outcomes, several problems may impair the usefulness of the research findings in informing clinical practice. For example, researchers may choose outcomes to suit their own needs, heterogeneous outcomes can impair future synthesis of research findings, and without pre-defined outcomes, it is difficult to know if publishing authors have neglected to include outcomes found in their research 10,11 . It is for these described reasons that many researchers are starting their research and/or registry development with a significant piece of research work to identify the "core outcome set" (COS) in the relevant research field, which may also be known as a "minimal data set". During this process, a great deal of research energy and time is invested into identifying the variables and outcomes to ensure that the final results of the future research and data-collection are meaningful and employable. There are several considerations regarding the choice of method to use to when developing a COS, which include factors such as the need for methodological rigor in the consensus process, ensuring a diverse range of stakeholder opinions, and finally financial constraints and carbon costs that might limit the practicality of face-to-face meetings 10,11 . The Delphi technique is a well-respected tool for establishing a COS and is a structured process utilizing a series of 'rounds' to gather information until group consensus is reached. Each Delphi round employs individuals across diverse geographical locations and diverse areas of expertise. The anonymous nature of the Delphi technique also avoids domination of the consensus process by one or a few experts 12 .
In Rwanda there is a long-term goal to develop a Rwandan surveillance program to provide monitoring and epidemiologic data that could be a first step in identifying risk factors and to improve the provision of care of the families of children with congenital abnormalities.
The WHO have given guidance that the variables to be included in a surveillance programme (registry) may vary, depending on the capacity and resources of the health-care system and surveillance programme 2 . This Delphi-study aimed to use consensus methods to identify a methodologically robust COS for a Rwandan congenital abnormalities surveillance program. This COS was intended to include risk factors, clinical features, syndromes, and outcomes.

Study design
A three-round, modified Delphi-study was undertaken between April and June 2017. Reporting of the study is in accordance with the Sinha and Williamson (COMET, Core Outcome Measures in Effectiveness Trials initiative) checklists for creating a COS using Delphi techniques 10,11 . No study protocol has previously been published for this study.

Round1
Aim. Round 1 aimed to produce a draft COS using published resources. This is a common practice in Delphi studies with the justification that the number of possible domains and outcomes to include in the COS of a Congenital Abnormality Surveillance program is substantial. Without providing an initial draft COS to participants the level of recruitment and engagement would be low 11 .

Search strategy.
A PubMed literature and Google internet search was performed to identify CARs (globally) or epidemiological studies investigating the prevalence and description of congenital abnormalities (African continent). PubMed was searched using Medical Subject Headings (MeSH) keywords and synonyms for: "newborn" AND "congenital abnormalities" AND "developing countries" (see Extended data, Additional Supporting File 1 for a full search strategy 13 ). The search was limited to human studies in the English language with no date limits imposed, the final search being undertaken on 26 th January 2017. Epidemiological studies providing a description of all congenital abnormalities of an entire population were included (See Supporting File 1). Articles looking at specific congenital abnormalities alone (e.g. cranial defects, congenital heart defect) were excluded. Google was searched using the above synonyms along with synonyms for "registries".
Obtaining outcomes. When contact details were available in the article, authors of the articles/registries were contacted by email to gain their outcome sets, questionnaires and/or datacollection tools. When contact with the author was not possible the outcome set was extracted from the materials available (i.e. the journal article or website).
Coding. Individual outcomes (e.g. gestation, cleft palate, Pierre Robin, etc.; see Additional file 2) were then coded for content and frequency. The outcome codes were categorized into domains (e.g. maternal details, clinical signs, etc.; see Table 1).

Round 1 consensus.
We aimed for a minimum of 15 outcome sets from either published papers or repositories. Consensus was predefined to include any domain or outcome found in two or more of the identified registries or journal articles. These domains and items were then added to the first draft of the COS.

Selectionofstudyparticipants(Rounds2and3)
Inclusion criteria. Participants were all medically qualified physicians and needed current or previous experience of working in a resource-limited setting, such as Rwanda. All participants needed to have experience of caring for children with congenital abnormalities. Physicians were chosen as Round 1 will have identified existing epidemiological tools, and in Rwanda physicians are likely to be the major stakeholders in collecting and utilizing data.
Sampling/enrolment. Recruitment was undertaken from the following sources: i) Rwandan paediatricians (n=53) via the Rwandan Pediatric Association (RPA) records, ii) Rwandan pediatric residents (n=49) via the University of Rwanda (UR) records, iii) International paediatricians previously working on the Human Resources for Health programme 14 (n=35) via the Ministry of Health (MoH) records, and iv) Authors of journal articles from Round 1 of the Delphi process (n=15) via the correspondence address. Invitations were sent via email with a link to the questionnaire, which is available as Extended data 13 .

Samplesize(Rounds2and3)
We aimed to gain responses from a minimum of 15-30 respondents in each round, which is considered the required number for gaining consensus in Delphi techniques 15 . The response rate was predicted to be 10-20%. Therefore, 152 participants were invited.

Procedures:Round2
Feedback. Feedback from Round 1 was given to participants in the form of the first draft COS, available as Extended data 13 .

Instructions to participants.
Each domain was presented individually, with its respective outcomes, followed by an openquestion; "On reviewing the above outcomes for the domain of <<domain title here>> are there any ADDITIONAL outcomes that you feel that SHOULD be included in the Rwandan Congenital Abnormalities Surveillance Program? (You may list these or write free text, as you wish. You do not need to write a justification of your additions)". Participants were presented with an open-ended text-box where they could "free text" any additional outcomes to include.

Coding and consensus.
Outcomes suggested by participants were coded in Microsoft Excel. Consensus was predefined as a minimum of two independent persons suggesting an outcome prior to it being included in the COS for Round 3 11 . The initial draft outcome set from Round 1 and the new outcomes meeting consensus were combined to create the second draft COS.

Procedures:Round3
Instructions to participants. In Round 3, closed questioning was employed. Each outcome of the second draft COS was presented using a 1-9 point scale, as described by Guyatt and the GRADE development group 9,16 . Outcomes were presented within their domain with the following instructions: "Where 1-3 are unimportant, 4-6 is ambivalent and 7-9 is important, how important is it that the following are included in the Rwandan Congenital Abnormalities Surveillance Program".
Feedback. Feedback was given to participants by giving the frequency the outcome was described in Round 1 (as a percentage) or if it was a new addition from Round 2.

Consensus.
Consensus for inclusion in the final COS was pre-defined as greater than 70% of participants scoring 7-9 (important) AND less than 15% of participants scoring 1-3 (not important) 11 .

Datacollection,managementandanalysis
In Round 1, outcomes and domains were coded and input into Microsoft Excel. For Rounds 2 and 3, Google Forms was used to administer the questionnaire and collect data. A web link to the Google Forms questionnaire was sent individually to each participant via email. Questionnaires 13 for Rounds 2 and 3 were kept open for two weeks. An email reminder was sent once for each round. Participants were reminded of the importance of completing both Rounds 2 and 3 of the Delphi process and asked to complete both rounds. However, because of the fully anonymous nature of responses, non-responders from Round 2 were still invited to Round 3. New participants were not invited between Rounds 2 and 3. Data analysis was undertaken in Microsoft Excel and was descriptive based on the above defined consensus.

Ethicalissues
This research proposal was approved by the Institutional Review Board (IRB) at the University of Rwanda (155/ CMHS IRB/2017). A statement of consent was included in the email communication with participants. Responding to the questionnaire was deemed as informed consent.

Results
We undertook a modified Delphi-study. A study flow diagram is available in Figure 1).

Round1search
Our PubMed literature search revealed 545 results. Titles were reviewed along with the abstract, where necessary. We identified 15 relevant journal articles giving descriptions of congenital abnormalities on the African continent and meeting the inclusion criteria (see Additional Supporting File 1). This number of articles was appropriate to deliver consensus on key outcomes for the initial draft COS. Therefore, a further journal/literature search was not performed for studies outside of the African continent. The authors of the included papers were contacted and only one of the authors replied with details of their COS. Therefore, outcomes were extracted from the journal articles.
Regarding existing congenital abnormality surveillance programmes (registries), we were unable to identify any active congenital abnormality repositories/registries from the African continent. We therefore extended our search to other continents. This search revealed 14 repositories: seven from Europe (Belarus = 1, Finland = 1, France = 2, Georgia = 1, Malta = 1, United Kingdom = 1), four from North America (USA = 3, Canada =1) and one from Israel. The included datasets also included the tools from EUROCAT 8 and the WHO 2 which are the best described guidance on surveillance programmes.

Round1:CreatingthefirstdraftCOS
All outcomes from the 15 journal articles and 14 repositories were coded in Microsoft Excel. From these 29 sources a total of 285 outcomes were identified of which 136 (48%) met our pre-defined criteria for consensus to be included in the first draft of the COS. These outcomes were categorized into 10 domains (Table 1). The consistency of outcome sets from these was low with only 8 of the 285 outcomes (3%) being present in ten or more of the 29 articles/repositories (Additional Supporting file 2). Each of the 285 outcomes was found in an average of 2.7 (9.3%) of the 29 journals/repositories (Standard deviation, SD=2.8). Each of the 136 items included in the first draft COS outcomes were found in a mean of 4.6 (15.8%) of the 29 journals/repositories (Standard deviation = 3.1). The most commonly described outcomes were "spina bifida" and "cleft palate/lip" which were both present in 16 (55%) of the articles/repositories. Each journal/repository described a mean of 27 outcomes (SD=14). Repositories had more outcomes than journal articles with averages of 30 and 24 outcomes respectively.

Round2and3participants
A total of 37 and 34 participants (Table 2) responded, giving a response rate of 24% and 22% for Rounds 2 and 3, respectively. This exceeded our expected response rate of 10% and resulted in significantly more than the 15-30 participants needed for validity of each round. In Round 2, one participant had never treated children with congenital abnormalities and didn't meet the inclusion criteria. In total, 73% and 63% of participants treated children with congenital abnormalities either frequently or very frequently. Participants were experienced with a mean of 12 years and 11 years of pediatric experience for Rounds 2 and 3, respectively. Participants included general pediatricians, neonatologists, geneticists, neurologists and pediatric residents.

Attrition
Questionnaires were fully anonymous. Participants did not know the identities of the other individuals in the group, nor did they know the specific answers that any other individual gave. However, year of birth and initials were given by subjects in order to assess attrition rate. Of the respondents from Round 2, 17 (46%) contributed to Round 3, giving an attrition rate of 20 subjects (54%).

Round2COS
Outcomes were presented within their domain on a separate page of the electronic questionnaire. After coding, a total of 219 new outcomes were suggested by participants (Table 3). In total, 62 (28%) of these outcomes were either "general suggestions" or already found in this or another domain. There were therefore 157 genuinely new outcomes. Of the 157 new outcomes, 32 (20%) of these were independently suggested by two or more participants and therefore met the pre-defined definition of consensus and   were added to the COS and carried through into the final round of the Delphi process.

Round3
The questionnaire 13 was divided into ten sections reflecting the ten domains of the second-draft COS. Outcomes were individually presented in their respective domain. There were 168 outcomes presented for scoring between 1 (non-important) and 9 (important) by participants. In total, 103 outcomes (61%) met the pre-defined consensus criteria to be included in the final COS.
The final steps were to: i) Ensure outcomes were in appropriate domains, ii) Attach International Statistical Classification of Diseases and Related Health Problems (ICD) coding and naming conventions 17 to the outcomes, iii) Place all clinical features (n=52) into anatomical systems, and iv) Alphabetise/ order outcomes for ease of use (See Supporting File 3).

Discussion
This study aimed to create a COS for use in a Rwandan Congenital abnormalities surveillance programme. Using structured consensus methods and taking into account the perspectives of experienced pediatric clinicians, a COS (minimal dataset) of 103 variables and outcomes, within ten domains, has been created.

Draft-COSdevelopedinRound1
Our PubMed and internet search identified 29 articles and repositories. There was a large number of excluded descriptive studies of specific abnormality types (e.g. neural tube defects only). The literature search has shown that research studies on congenital abnormalities have been done in Africa but wellstructured mechanisms for their surveillance are quasi-absent (i.e. journal articles are available but no repositories/registries were found); this is concerning when one considers the finding that more than 90% of newborns with congenital abnormalities are born in LMICs 4,5 . These findings support the need for upscaling of surveillance programs on the African continent where congenital abnormalities are responsible for a significant burden of disease.

Stakeholders
With a lack of structured mechanisms or tools to detect and follow-up neonates with congenital abnormalities in Rwanda, it was judged an essential first step to gain consensus from physicians who will case-find regarding which items to be included in the surveillance tool. The Delphi technique was found to be both cost-effective and practical with a methodological rigor to reach a large number of diverse experts and with an advantage of avoiding the negative effects of dominant individuals. Pediatricians are well placed to advise on the development of a surveillance programme for congenital abnormalities since they care for affected neonates and children in their daily practice. In Rounds 2 and 3, we received a higher than anticipated response rate and therefore gained more participants than expected. This was a welcome finding and therefore our results offer consensus of a larger body of experienced clinicians. The participants were also from several different settings giving a broader range of experience.
The high response rate is also a sign of how important and relevant participants found the subject and hopefully reflects the "buy-in" regarding a future surveillance program.

Variables
The consistency of outcomes described in the journals and repositories in Round 1 was low. Each of the 136 outcomes included in the first-draft COS was found, on average, in only 16% of articles/repositories. This lack of consistency supported the need for this Delphi study to develop a locally relevant COS, such as the one described here. The finding that the 14 repositories held a mean of 30 outcomes suggests that our COS may hold too many outcomes. This may be due to the pre-defined threshold for consensus in Round 3. However, it is interesting to note the high number of new outcomes suggested by participants in Round 2 which could suggest that these additional outcomes are in fact needed.

Applicationoffindings
The next step in developing the Rwandan CAR is to field test (pilot) the COS in the clinical environment. This is to ensure that completing data-collection using the COS is feasible for phyisicians within day-to-day clinical practice. The WHO guidance on surveillance programmes describes that a programme may be population based or hospital/facility based and can use active or passive case ascertainment (case-finding) 2 . We intend to use the COS developed in this study to commence passive case ascertainment in teaching hospitals, in order to establish baseline data and feasibility of such a surveillance programme.

Strengthsandweaknessesofthestudy
Strengths of the study include the fact that the Delphi technique is a commonly used consensus method with several advantages whilst minimizing some of the disadvantages associated with collective decision making, for example, domination by individual interests 9 . We felt it was a priority to include the variables from EUROCAT 8 and the WHO 2 guidance on surveillance programmes and have used Delphi methods to build on these data-sets. We used physicians from different settings and different levels of clinical experience to ensure there was no dominance of a particular domain.
Limitations include the fact that our participants were limited to clinicians who had experience of caring for children with congenital abnormalities. Patients or their families, researchers and biostatisticians were not included. It was felt that parents/ carers would not have a significant enough understanding of the risks, clinical findings and/or syndromes being assessed. In hindsight, involving collaborators from nursing and allied specialties could have been beneficial as they are often primary caregivers in this setting and therefore will be future stakeholders in future surveillance programmes. Further limitations were the attrition rate from Round 2 to Round 3. The use of a computer-administered questionnaire will have inadvertently excluded less computer literate participants. However, it has been found that participants who are willing to participate in consensus panels are generally representative of their colleagues 9 . A final limitation is the use of non-African repository data-sets in the first draft. The first draft of the COS included outcomes from settings non-similar to our intended setting.

Conclusions
This is the first COS for congenital abnormalities identified in the literature from the African continent. It has been developed for use in Rwanda but could relevant for use in the region and other resource-limited settings. However, each setting should develop their data-set based on the resources available to them and the objectives of the surveillance programme 18 . This project contains the following underlying data:

Grantinformation
The author(s) declared that no grants were involved in supporting this work.

Paula R. Williamson
Department of Biostatistics, University of Liverpool, Liverpool, UK This article presents the items comprising a minimum data set for a registry of congenital abnormalities in Rwanda, determined through a survey of health professionals. It addresses an important topic, since the set up and maintenance of a registry is no small feat, and increasing the likelihood that key, relevant items are measured is critical. I believe the authors are proposing a 'minimum data set' (MDS) rather than a core outcome set (COS). A MDS typically includes a COS plus additional variables (for example baseline, risk factors and treatment variables). Use of the term 'items' rather than 'outcomes', and 'MDS' rather than 'COS', throughout would be preferable therefore.

Background
How did the 15 African studies reviewed determine items for their registries? This information would be of interest to the reader. Is it clear that they did not use a consensus process, and if not, how did they determine what to measure?

Methods
In round one, a long list of potential items has been created. This is not usually referred to as a 'draft COS' however since it consists of a list of any outcome ever measured by one or more groups rather than being based on any consensus process.
It does not appear that an individual participant had the opportunity to review scores from other participants, to reflect on their own view, and then to rescore. The study design appears to be a series of two surveys therefore rather than a Delphi survey. This should be clarified.

Results
The authors state there was an ineligible participant but continue to include their results. This should be clarified and explained, and the tables amended as appropriate.
The addition of 32 items is high. What might this imply? How many of these additional items were in the final 103?
The number of new outcomes suggested in round 2 in the 'syndrome/diagnosis' domain is 6 in Table 1 but 8 in Table 3. Please clarify this inconsistency.  There were 8 items common to the previous registries. Were these 8 in the final set? This would be an interesting discussion point.

Discussion
When referring to the number of 'outcomes' in previous registries, is it outcomes or rather items?
The COS-STAD standards require patients (here parents) to be involved in the determination of important outcomes. Some discussion as to why parents were not involved in this project would be of interest to the reader.
MDS and COS projects often include an in-person meeting to finalise the set. Some discussion as to whether this was considered and, if so, why it was not pursued would be of interest to the reader.
Why is 'the use of non-African repository data-sets in the first draft' necessarily a limitation?

Conclusion
The authors conclude that a new set of registry items should be developed for each setting. I believe this approach could contribute to research waste. I would recommend that each group wishing to implement a standardised set of registry items should consider existing sets first, to assess their generalisability to the setting at hand, and critically appraise the methods used. If a new set is still needed for the new setting, then a consensus process should be followed.

Typos
'core outcome' not 'core-outcome' Abbreviations -COMET (Core Outcome Measures In Effectiveness Trials)

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Partly

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Core outcome set development I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com