Critical thinking about treatment effects in Eastern Africa: development and Rasch analysis of an assessment tool

Designing the questionnaire

For this study we included both ability items and the items measuring intended behaviour and self-efficacy.

We planned on removing MCQs with sub-optimal measurement properties based on the results of this study. Therefore, we included more MCQs than we plan to use in the trial (two MCQs per Key Concept). The educational intervention we will evaluate in the randomised trials addresses nine Key Concepts (Table 1). For each of those concepts, we included three MCQs in the questionnaire, a total of 27 MCQs assessing ability. All MCQs included 3 response options.

We included three items that assess intended behaviour and four items that assess self-efficacy. The Likert scales include four response options ranging from very likely to very unlikely (intended behaviour) or very difficult to very easy (self-efficacy), and a fifth option: ‘I don’t know’.

In addition, we included demographic questions asking about gender, age, educational level, country of residence, training in research methods, and experience with participation in randomised trials. Gender, age, and country of residence were important for the psychometric analysis (testing for differential item functioning). The other background factors were used to ascertain that we were able to recruit people with a spread in ability level (ability to assess treatment claims). Level of education and familiarity with research methods have been shown to be associated with more correct answers.¹⁴

In preparation for this study, we conducted cognitive interviews and piloted the questionnaire with individuals from our potential target groups in Uganda, Kenya, and Rwanda.^11,15 The objective was to get feedback from members of our target groups in the three contexts on the acceptability and relevance of the terminology and formats used in the questionnaire. Even though the items included in the Claim Evaluation Tools item bank have previously gone through an extensive development process in Uganda, we considered it important to get feedback from people in our target groups in Rwanda and Kenya, where the items had not been tested before.

We recruited schools in May- August 2021 through the project’s teacher networks. In the interviews the students were encouraged to think aloud about how they understood the scenarios and response options, and to identify any issues they had regarding comprehension of terminology or format. The researcher noted down all identified issues. All feedback was summarised by the lead investigators and the findings was discussed in the project group including the research teams in all three contexts.

Piloting took place in a classroom setting. The purpose and instructions of the test was introduced to the students by a member of the research team in collaboration with the teacher, observations were made regarding time taken to complete the questionnaire and comprehension of the format (incorrectly filled in response options).

Findings coming out of the interviews and pilots led to only minor changes, such as changing some of the names and other terminology used in the MCQs to improve familiarity in the two new contexts. We also changed the format of the intended behaviour and self-efficacy items from a traditional Likert-scale to resemble a multiple-choice format, keeping the same response options (Figure 1).

Figure 1. Example of an intended behaviour item.

We made that change because the Likert-scale format was unfamiliar to some of the students in the three contexts, and the MCQ format was more familiar and acceptable to the students. The pilot studies also provided us with information about the time needed to complete the questionnaire (between 30 and 60 minutes) and what we could expect in terms of missing responses in the upcoming trial.

Previously, several tests have been developed from the claim evaluation tools item bank. The test developed for this study was named the Critical Thinking about Health test. A copy of the test evaluated in this study is available us extended data.³⁶

Inclusion criteria

There is no gold standard for the number of respondents needed for Rasch analysis. This is a pragmatic judgement considering the number of items evaluated and the statistical power needed to identify item bias resulting from background variables.^16–18 Rasch analysis does not require a representative sample. However, the sample should include enough people to allow for evaluating differential functioning and a spread in ability. Studies have found that a sample of 200-250 people per group is suitable for detecting differential item functioning (DIF).^19,20 We expected both item-sets to work in the same way for children and adults and to have no differential functioning by gender.¹¹ For this evaluation, we also needed a sample of people with different ability to assess treatment claims. There are few background variables that may predict ability to assess treatment claims, but higher education involving training in statistics or research methods may be a factor.¹⁴ Consequently, we estimated that recruiting approximately 500 people in each country, with an equal distribution of men and women, and lower secondary school students and adults would be adequate (Table 2). We also made sure to recruit people from higher education contexts, through the university networks in each context, as well as people in our local communities, social media, and students from schools participating in piloting of the educational intervention. We commenced data collection in July 2021 and was completed December in the same year.

Recruitment

All recruitment and data collection were done during lock down due to COVID-19, leading us to use varied strategies for recruiting our respondents.

In Uganda we recruited participants using our networks there including teachers, students, and National advisory panel networks. For students, we used three strategies, including visiting students at their homes, reaching out through the student network, and also requested teachers who were conducting online revision classes to introduce us to their students via the platforms to introduce the project and share the questionnaire link via WhatsApp or Telegram (both media apps for communication) after obtaining consent. For adults, we recruited people with higher education qualifications through university platforms i.e., the University faculty platforms, a PhD forum which has over 40 PhD fellows, students studying medicine WhatsApp groups, and a teachers’ network WhatsApp group. However, for the local communities, we visited food and clothes markets and asked them to complete the questionnaires. All data collection was done in the central region (Kampala and Wakiso) and the northern region (in Gulu district) of Uganda.

In Kenya we recruited students from three schools that participated in piloting the IHC secondary school resources. In those schools, we purposively included all the participants from one stream except those that had been selected for the pilot. Each school had about three-four classes and each class had about 40 students. For adults, we included the student’s institution of tertiary education and members of the community with low education levels (secondary and below), and those that could read and owned a Smartphone. For the students, we purposively included students from two faculties (Health and Arts and Sciences). Through the Dean of students, we invited them to a meeting where we introduced the project, outcome measure and sought their verbal consent. We then shared the link to the test and asked them to log in and participate. For community members, we used our database to recruit members that were actively involved in the institute’s previous and ongoing community-based projects in rural settings in Butere sub-County. Although we reached out to many members, only a few members responded thus we resorted to recruit more from the student’s fraternity (pursuing diploma and certificate courses). We used a similar recruitment and consenting process described for the students above.

In Rwanda, for adults, we used WhatsApp and recruited using the snowballing method through our networks, including the projects teachers’ network and students’ network in Rwanda. The teachers network included lower secondary school teachers who were from different schools, and they varied in terms of work experience, age, subject area and schools they teach from. Similarly, the students’ network included students from similar schools as members of teacher’s network. They also varied in their age, sex, and history of school performance (high or low performing students). We also used emails and reached out to adults who work or previously worked with the school of public health researchers in Rwanda. We also engaged a teacher’s network who also responded to the test. We recruited students through schools that participated in the development and pilot of the intervention in Kigali city and surrounding neighborhoods.

Data collection

Most of the data collection was done online, using a service hosted by the University of Oslo (Nettskjema). One small sample (students in Kenya) used paper questionnaires in a classroom setting and administrated as an exam as part of pilot testing of the IHC secondary school resources. The test was administrated by a teacher under the instructions of the research team. The paper questionnaires were scanned and added to the data collected online.

Ethical statement

Ethical approval was obtained from the relevant authorities in each country; Masinde Muliro University of Science and Technology, Institutional Ethics Review Committee (MMUST/IERC/75/19, License No: NACOSTI/P/21/8103) the Rwanda National Ethics Committee 916/RNEC/2019, School of Medicine Research Ethics Committee (REC REF 2020-139)/Uganda National Council of Science and Technology (HS916ES).

All participants were given written information about the purpose of the study and that participation was voluntary, and how the findings would be used to improve the validity and reliability of the Critical Thinking about Health test. Children participating through their schools were also given oral information. We obtained written consent from all adult participants, the minor’s guardians, and written assent from the minors.

Since this was a knowledge test, just as a regular school exam, this study did not collect any personal or other sensitive information that could lead to identification of the respondents. None of the members of this project group had access to information that could identify individual participants during or after data collection.

Rasch analysis

Rasch analysis is a dynamic way of developing measurement tools with construct validity.¹⁴ The approach is used to address important measurement issues required for validating an outcome measure, including internal construct validity (by testing for unidimensionality), invariance of the items (item-person-interaction), and item bias (differential item function).^21,22

We imported the data from Excel (version 2208) into RUMM2030 (https://www.rummlab.com.au/) and followed the basic steps of Rasch analysis as recommended in the literature.^21,23 R is a freely accessible software environment for statistical computing and graphics including Rasch analysis that can be used to run a similar analysis (https://www.r-project.org/). We analysed the two item-sets separately based on the assumption that these measure different underling traits. The MCQs were scored dichotomously as correct or incorrect. We applied the polytomous model to the intended behaviour and self-efficacy items.²² When entered into RUMM2030, missing data was coded as “0”.

The first step in the analysis involved exploring the class interval structure (number and size of ability groups) and the summary statistics (person-Item distribution). In Rasch analysis, the ratio between any two items should be constant across different ‘ability’ groups. The response patterns to an item-set is tested against what is expected by the model which is a probabilistic form of Guttman scaling.²¹ In other words, the easier the item is, the more likely it will be ‘passed’, and the more able the person is the more likely he or she will pass.²¹ We explored this relationship using the summary statistics function in RUMM2030.²³ In RUMM2030, the item-person interaction is presented on a logit scale, where the mean item location is ‘0’. If the instrument is a well-targeted measure (not too easy or too difficult), the mean location for individuals would be around the value of zero.²² If the person location is higher than zero, this indicates that the test is easy, if the person location is lower than zero this indicates that the test is difficult. The item and person fit residual statistics assess the degree of divergence (or residual) between the expected and observed data for each person item when summed for all items and all individuals respectively for each test set.²² In RUMM2030 this is reported as an approximate z-score, representing a standardized normal distribution.²² Ideally, item and person fit should have a mean of zero and a standard deviation of one.²²

We calculated Cronbach’s alpha to assess the reliability of both item-sets by removing missing data. A Cronbach’s alpha above 0.7 was considered acceptable.²²

The principal component analysis/t-test protocol is used to test the hypothesis of unidimensionality. This is done by identifying the two most divergent item subsets (using the residual principal component function in RUMM2030), and then calculating t-tests.²² If ≤5% of tests are significant, strict unidimensionality can be inferred.²⁴ However, the concept of ‘unidimensionality’ is not ‘definite’ but relative and should be supplemented with quantitative or qualitative interpretation of the explicit variable definition and considering the context and purpose of the measurement.^24,25

We tested for local dependency by using the residual correlations function in RUMM2030. Data from this output was copied into Excel (version 2208) and any residual correlations greater than 0.2 above the average was considered as potential problematic dependency.²²

We identified individuals and items with ‘misfit’ to the Rasch model by chi-square statistics and by exploring the fit residuals. Items with statistically significant chi-square probabilities do not fit the model at 0.01 significance level, items within a ±2.5 fit residual range are considered to be potentially problematic.²² Similarly, individuals with a fit residual of ±2.5 were considered as not fitting the model. Such extreme values can be an indication of, for example, guessing or copying, and that the item-set is not appropriate.

We examined differential item functioning (DIF) by age, gender, and country of residence. It was our objective to include only items that could be applied fairly across these demographic variables. Ideally, all items in the Claim Evaluation Tools item bank are expected to work in the same way for men and women, and across age groups. There are two types of DIF. Uniform DIF is when the difference between groups for an item is systematic - for example adults having systematically higher ability compared to lower secondary school students. This is less problematic (when it is known) than non-uniform DIF, where the difference between groups on an item is inconsistent across ability groups.²¹ For this study, we considered non-uniform DIF as unacceptable. We predicted that we would find uniform DIF by country, as we know from other studies that there are differences in ability-by-concept across countries.¹⁴ Uniform DIF by gender and age was unwanted but would be considered in relation to the other findings from the Rasch analysis. The reason for this was that the questionnaire will be used for measuring differences between an intervention and a comparison group, and systematic DIF would therefore not be a problem in our study.

In the item characteristic curve plot the expected scores and the observed scores for the class intervals of the different ability levels are displayed. We observed the item characteristic curve for each item and made note of items that showed under-discrimination, over-discrimination, or had several deviating ability groups.²² We considered items with under-discrimination and classic over-discrimination for removal. Marginal over-discrimination was not considered to be a problem for our purposes.

For the polytomous items we explored the threshold ordering (fit to the expected logical order of the response options) to check for disordered thresholds. Disordered thresholds suggest that the scoring categories are not progressing as expected, and that the item is not working properly.²²

This study follows the STROBE-reporting standards.³⁸

Summary statistics

A total of 1,671 responses were entered into the analysis distributed across 10 ability groups identified by the RUMM2030 software of which 49% were women and 40% were young people (under 18). Of these, 35% were from Kenya, 34% from Uganda, and 31% from Rwanda. Missing data was minimal only 0.004%, and thus had no impact on the analysis.

The person-item distribution shows that both item sets are well targeted (mean person location was -0.218 for the ability item set and 0.084 for the Likert item-set.

For the ability items, the person fit residual was -0.204 (SD 0.741) and thus showed satisfactory fit to the model. The items’ fit residual was 0.712 (SD 2.235) and warranted further investigation in subsequent analyses.

For the Likert items, the item fit residual was 0.543 (SD 0.938), indicating reasonable fit. However, the high standard deviation for the person fit residual (-0.546, SD 1.783) suggested some misfit to the model.

Both item-sets were found to be reliable, with a Cronbach’s alpha of 0.72 and 0.79 for the ability and Likert item-sets respectively.

Individual person and item fit

In the analysis of the ability item-set, we identified one person with a highly negative fit residual (adult, female, Rwanda) and two with highly positive fit residuals (male, young person, Rwanda and adult, female, Rwanda). Of the 27 MCQs, three items had extreme negative values, and four items had extreme positive values.

There were no items with extreme values in the Likert item-set. However, several misfitting persons were identified (296 individuals) with high negative residuals and two individuals with high positive residuals.

DIF and item characteristic curve-analyses

The majority of the ability items had a good fit to the item characteristics curve (Figure 2). Four items showed evidence of classic overdiscrimination, of which two of these also had very high negative fit residuals (Figure 3). Four items showed sign of classic underdiscrimination and were considered candidates for removal (Figure 4). Most Likert-items showed a good fit, although two items were slightly overdiscriminating, this was considered acceptable.

Figure 2. Example of ability-item with satisfactory fit.

Figure 3. Example of ability-item with classic over-discrimination.

Figure 4. Example of ability-item with classic under-discrimination.

In the DIF analysis of the 27 ability items, two items showed uniform DIF by gender (one item where males did systematically better and one where females had higher ability). Three items showed DIF by age, of which two were uniform (one item where young people performed better and one item where adults had higher ability). One item had non-uniform DIF by age. Uniform DIF by country was found for 10 items, the ranking of the three countries differing across these items.

There was no DIF by gender, age, or country in the analysis of the Likert item-set.

In the Likert item-set, two items were found to be slightly over-discriminating and were therefore considered acceptable. The remaining items showed very good fit.

When exploring the ordering of the thresholds, we found that the three Likert items evaluating intended behaviour were disorganized. A reanalysis of these suggest that these could be improved by dichotomising the response options. The four items evaluating self-efficacy showed a good fit.

Test of unidimensionality

In the analysis of the ability item-set, 8% of the T-tests were significant.

The magnitude of multidimensionality in Likert-items were found satisfactory at 5% and considered to be unidimensional.

Local dependency

There were no item-pairs correlations above 0.2 of the average value in any of the item-sets, suggesting no important redundancy.

Revision of the questionnaire

The outcome measure to be used in the final trial was reduced to include only two MCQs for each Key Concept to be assessed. We removed the ability-items with suboptimal fit. Since the Likert-items were all found to have good fit, these remained unchanged.

The revised outcome measure has been published as extended data.³⁷

Strengths and limitations

One limitation of this study is that the adult population included more people with higher education than the general population in each of these three settings. Thus, the test might be more difficult for people with less education. However, although participants with higher education are somewhat more likely to answering the ability questions correctly, there does not seem to be a strong association.^14,33 Another limitation is that the findings of this study are exclusive to the three Eastern African countries, and the validity and reliability in other contexts are uncertain. The item-sets validated in this study should therefore undergo further psychometric testing if used elsewhere.

The strategy of using pilot testing and a Rasch analysis have been found to be a robust method for developing measurement tools in several contexts.^10–13 An important strength of this study is that we used explicit and transparent methods, following the principal steps recommended for Rasch analysis.^21–23 Another strength is that we were able to recruit enough people despite the fact all three countries were burdened by the pandemic during the data collection. The results of this study and subsequent design of the questionnaire based on these results ensures that both the ability and Likert item-sets are a valid and reliable outcome measure for the randomised trials of the IHC lower secondary school intervention in all three countries.