Collecting clinical data in primary ciliary dyskinesia- challenges and opportunities

Context for questionnaire development – NIPC study

In 2011 we decided to characterize the clinical features of PCD patients in Israel. A prospective National Israeli PCD Consortium (NIPC) study was conducted between the years 2011–2013 in subjects presenting with the typical clinical phenotype of PCD in 14 pediatric pulmonology centers (for details see ref. 12). Ethical approval was obtained from Institutional Review Boards at each centre that collected patients’ data (Ziv Medical Center, Schneider Children’s Medical Center, Edith Wolfson Medical Center, Soroka Medical Center, Hadassah-Hebrew University Medical Centers, Rambam Medical Center, Western Galilee Hospital, Saint Vincent De-Paul Hospital, Assaf Harofeh Medical Center, Shaare Zedek Medical Center, Edmond & Lili Safra Children’s Hospital, Sheba Medical Center, Carmel Medical Center, Dana Children’s Hospital) as well as from the Israeli Ministry of Health. and all patients and guardians signed written informed consent. The study was registered on ClinicalTrials.gov (NCT 01070914). To complement the clinical phenotypic data and in order to verify the diagnosis, the subjects had a comprehensive study visit whereby they also underwent a series of tests including nasal NO (nNO), nasal brushing of samples for transmission electron microscopy (TEM), immunofluorescence (IF), high-speed video microscopy (HVM) and blood sampling for genetics. In this study, the presence of at least 2 abnormal results for these tests were used as criteria to define PCD³. To capture the most accurate clinical information in this study, a special questionnaire had to be developed.

Development, validation and revisions of the questionnaire

Only a few questionnaires have been previously used to specifically collect clinical data in PCD and none of them as far as we know, have been systematically and critically evaluated^5–11.

To validate the content of the NIPC questionnaire, a national expert panel was created. It consisted of three pediatric pulmonologists, one adult pulmonologist and one ENT surgeon with expertise in PCD. All members had been in practice at least 15 years after certification and have run PCD clinics for at least 10 years.

The initial draft of the NIPC questionnaire was developed by the panel members after reviewing existing questionnaires^5,6,8–11 and selecting content. Across multiple iterations the panel provided feedback regarding the content relevance and representativeness with respect to the clinical presentation of PCD. Discussions between panel members were conducted electronically and by conference calls. Based on feedback of the panel, modifications to the questionnaire were made after each iteration. The major concern of the panel was the excessive length and detail of the initial draft. It was felt that both organizational and content changes had to be addressed in order to make completion of the questionnaire feasible within a reasonable amount of time.

As the proposed study had planned to enroll subjects who were clinically suspected of having PCD, there were many items the panel found unnecessary or irrelevant. For example, history probing about meconium aspiration was considered irrelevant for the diagnosis of PCD. Likewise, items regarding disorders of family members extending to relatively remotely-related members (e.g., biological maternal grandfather) were also deemed unnecessary. While genetic predisposition is important in PCD, the panel felt that there are more direct ways to obtain such information (e.g. questions on consanguinity and pedigree creation). Another problem identified by the panel was overlapping content. Many questions were repeated in various sections while providing similar information. For example, radiological evidence for sinus abnormality appeared both in the medical history and in the test sections. Consensus on questionnaire content was reached after approximately 10 iterations. During the revision process, we tried to balance the need of adding new clinical data with reducing the burden of a long and impractical questionnaire.

The main sections of the revised NIPC questionnaire were: A. Demographic details (9 items), B. Family history (8 items), C. Past medical history (52 items) and D. Physical examination and basic tests (radiology, spirometry, sputum etc.) (38 items). Response options were in the form of: YES/NO/Do not know; commentary or multiple choice. Some items included subsidiary questions. For example, when enquiring about the presence of chronic cough, a subsidiary question followed asking when the symptom began.

Beyond the history regarding PCD diagnosis, the questionnaire included several "rule out" questions such as normal sweat tests for cystic fibrosis.

Questionnaire completion, distribution and collection

The NICP questionnaire was emailed to the local principal investigator (PI) of each center (n=14) approximately one week prior to a scheduled visit. The local PI was asked to forward the questionnaire to the other physicians in their group (e.g. a center may have three physicians where each one physician is responsible for the care of a few patients with PCD). The physicians were asked to print out the questionnaire and complete it for each of their scheduled patients within one week.

The national PI and 2 or 3 research assistants (RA) joined the local PIs during actual patient visits (study sessions) in order to assist in the ancillary diagnostic test procedures (nasal brushing, video-microscopy etc.). There were 20 study sessions in all of the participating centers spanning an almost two-year period during which some centers had more than one session. Completed questionnaires were physically collected from each center by the study PI (IA) at the end of each session. Questionnaires were scanned and reviewed for missing or unclear data by the study PI and a RA. Unanswered questions were noted and a reminder to complete the questions was e-mailed to the local PI within 2 weeks of collection.

A second attempt to complete unanswered questions was done by transferring the received questionnaire data to an Microsoft Excel 2010 table, marking all missing or unclear answers, and e-mailing the table to the responsible physician. When a third e-mail reminder was necessary, the first author also personally phoned the local PI at each center to encourage them to complete the unanswered questions.

Data analysis of completed questionnaires to evaluate feasibility and utility

Quantitative analysis. Completion rates of each item on the questionnaire, as an indicator of feasibility, were calculated as a percentage, and then averaged across items for each section of the questionnaire (Microsoft Excel 2010). Completed items were coded as 1, whereas uncompleted items were coded as either relevant (code 2) or not relevant to the particular patient (code 3, e.g. an uncompleted item about fertility in a child). If an uncompleted item included a N/A answer, it was also coded as 3. Data were analyzed for the total group, and for those eventually diagnosed with PCD.

Qualitative analysis. Qualitative evaluation of the questionnaire feasibility and utility was completed from two post–hoc perspectives. The first was based upon guidelines on best practices for questionnaire development¹³. These guidelines focus on: a) features of the questionnaire (visual presentation, language and format, mode of administration), b) characteristics of the participant completing the questionnaire (their workflow, their degree of cooperation, their relationship with the researcher), and c) interaction of the participant with the questionnaire (i.e., what must the participant do to and data sources required to complete the questionnaire?).

The second was the physicians’ post-hoc user feedback. This feedback was collected through electronic communications, oral discussions and through an electronic survey using http://www.questionpro.com/a/listSurveys.do.

The survey is included in under Data availability (Dataset 1). Participants were first presented with specific items with a low completion rate and then were asked to suggest potential reasons for this. They were also surveyed about more general topics such as relevancy and representativeness of the items for PCD, preference for electronic vs paper records, time required to complete the questionnaire and future use in their practice.

Validation analysis

Content validation was conducted as previously described during the development stage. To assess internal consistency, we first computed frequencies and descriptive statistics for all items using IBM SPSS Statistics for Windows, Version 23.0³⁸ Attention was paid to the proportion of responses coded as not completed or unknown and not relevant. Items with a large proportion of responses coded as not relevant were often subsidiary questions to a lead question. Judgments were made regarding the added value of retaining these subsidiary questions for the analysis of internal consistency, over and above the information already provided by the lead question. Items with a large proportion (i.e., greater than 0.25) of responses coded as not completed or unknown also warranted a closer look at the item with judgments made on whether it should be included in further investigations of internal consistency. Internal consistency for the two major subsections (medical history and physical examination) was then examined using Cronbach’s alpha. Missing data were handled using the default option of listwise deletion in IBM SPSS Statistics for Windows, Version 23.0³⁸.

Taken together, the results were used to inform revisions to the PCD questionnaire.

Quantitative analysis (Completion rates)

Twenty-two physicians in 14 centers completed questionnaires for 173 subjects (of which 104 were eventually confirmed as PCD). Table 1 summarizes the initial completion rates for each subsection of the PCD questionnaire by group (i.e., total, PCD, non-PCD).

Overall, the average initial response rate was 82% and increased to 88% following two reminders. Subsections A (range=95–97%) and B (range=75–80%) had high initial completion rates. Subsections C (range=52–59%) and D (range=55–59%) had lower completion rates (p<0.01). This pattern of completion rates corresponded to items that required recall of readily available information about the individual and their past/family medical history (subsections A and B), whereas subsections C and D required more specific tests information. No significant differences in response rates were observed between PCD and non-PCD groups. Figure 1 illustrates the changes in response rate (%) between initial completion and following reminders for major individual questions.

Figure 1. Changes in response rate (%) between initial completion (red columns) and following reminders (green columns) for selected questions.

All questions ending with a (?) had yes/no/unknown answers. The rest had free text answers (e.g. birth date). A, B, C and D are the questionnaire sections. Abbreviations: NRDS-Neonatal Respiratory Distress Syndrome; LFT- Lung Function Test; FVC- Forced Vital Capacity; FEV1-Forced Expiratory Volume at 1^st second; CT- Computed Tomography; CF- Cystic Fibrosis.

Among the uncompleted items, seven were identified where a response was relevant and required (i.e., item coded as 2). One item was in the family history subsection B, five items were in the medical history subsection C, and one item was in the physical examination subsection D. Two of these items, both in the medical history subsection C, had a non-response of > 50%. The first of these items (64% non-response rate) enquired about the patient’s hearing loss with sub-questions to identify its nature: conductive, neuronal, or unknown. A potential reason why this item might not have been answered is that the response options were not exhaustive. If the patient did not have hearing loss, then none of the response options would have been the appropriate answer. Therefore, leaving the item unanswered was one course of action. The second item with >50% non-response was a sub-question about lung transplant which was found to be confusing. It was only post-hoc analysis that revealed the problematic nature of these items.

Five uncompleted yet relevant items had non-response rates between 25–50%. For example, one item enquired whether the patient had family members who suffered from any of 12 various symptoms. Although the symptoms were specific (e.g., chronic rhinitis, fertility disorders), patients may not have known their family history to that degree of detail or the information may not have been entered in the chart. Other questions with low response rate asked for the results of specific tests such as the sweat chloride concentration. If this information was not readily available from the chart or from the patient, this question would not have been answered.

Evaluation based on best practice guidelines for questionnaire development

Features of the questionnaire. There were two areas which could have been improved upon. First was consideration of terminology used and its consistency in interpretation across participants. For example, questions about “hearing loss”, and its sub-categories may not have a universally accepted definition thus resulting in differing interpretations¹¹. Second was consideration of questionnaire length. At 98 questions, the NIPC questionnaire was an improvement in length when compared with previously used questionnaires increasing the chances of collecting complete data¹⁴.

Characteristics of participants. In terms of the participant group, the questionnaire content and length may have impacted completion given how busy physicians’ practices are. However, continuous email and phone support provided by the study PI to the physicians likely helped to achieve completion rates of almost 90% using personal reminders.

Interaction between the participant and questionnaire. The effort to complete the clinical questionnaire ranged from minimal (e.g., completing demographic section) to at least moderate (e.g., completing family history or physical examination). Answers for some questions required specific data for which information within the records/charts were not easily available. In these cases, the physician had to complete the questionnaire during or after the actual patient visit. Analysis of free text answers was challenging and was not contributory in most cases.

Evaluation based on feedback from end users

Seventeen out of the 22 end users (i.e., physicians) provided feedback and identified reasons for low completion rates of some items. For example, 70% identified lack of available data as a reason for a low completion rate of the question about polysplenia. Twelve percent suggested that this item was irrelevant. A low completion rate for the item enquiring about onset of chronic cough and onset of wheezing were attributed to lack of available data in 53% of responders. This was also the case with “hearing disorders” (response rate of 47%). Fifty-eight percent thought that the fertility question was irrelevant to their subjects (children). Most (63%) reported that it took them less than an hour to complete the questionnaire in one session; 31% reported they required several sessions to do it; 81% preferred the use of electronic rather than paper versions and 75% said they will use the questionnaire as a diagnostic tool in their practice.

Internal consistency

Several items had a high proportion (>50%) of responses coded as not relevant. All these items were subsidiary questions to lead questions and were carefully reviewed before internal structure analysis. For example, the lead question C5 asks whether situs inversus is present. The follow-up questions C6 to C9 had 67.1% responses coded as not relevant, likely because situs inversus was not present in their patient. The PI reviewed these items individually and determined that question C5 captures the presence of this primary feature (i.e., situs inversus) within the patient sample whereas questions C6–C9 are subsumed conceptually by C5 and could be excluded from further analysis of internal structure. For the medical history subsection, Cronbach’s alpha was 0.63 and for the physical examination/tests subsection Cronbach’s alpha was 0.72.