The Norwegian public’s ability to assess treatment claims: results of a cross-sectional study of critical health literacy

The Claim Evaluation Tools item bank

The Claim Evaluation Tools item bank was originally developed in English for use in the two randomised trials described above^23,24. Since the item bank was first created items have been translated and validated in several settings and languages, including Uganda (in Luganda and English), Mexico, and China^26–30.

Our starting point for developing the educational interventions and the Claim Evaluation Tools item bank was a list of Key Concepts that people need to understand and apply to assess claims about treatment effects and make informed health choices (hereafter referred to as Key Concepts)³¹ (Table 1). We define “treatment” broadly to include any intervention (action) intended to improve health, including preventive, therapeutic and rehabilitative interventions, and public health or health system interventions. The list of concepts provides a framework for researchers, teachers and others to develop interventions and measure people’s ability to assess treatment claims and make informed choices. We review and amend the list yearly⁹.

The item bank is revised according to the Key Concepts list yearly. When we conducted this study, it included four multiple-choice questions (MCQs) for 32 of the Key Concepts shown in Table 1. The item bank also includes questions that assess intended behaviours and self-efficacy. The item bank is an open-access resource. Teachers and researchers can select items for questionnaires tailored to their specific needs.

To our knowledge only one survey has attempted to measure the ability to understand and apply any of the IHC Key Concepts in a representative sample of Norwegian adults³². That study only addressed four of the Key Concepts. The purpose of this study was to map the ability of Norwegian adults to assess treatment claims and make informed health choices, using MCQs from the Claim Evaluation Tools item bank. The findings can be used to inform the development of learning resources and communication of information to patients and the public, and for international comparisons. We also wanted to take the opportunity to compare the findings of this study with the results of our previous Ugandan study. Despite any differences that may exist between these two contexts, it is our experience from previous projects that the challenges associated with assessing claims in everyday life in Uganda and Norway are very similar. However, knowledge of specific Key Concepts and overall understanding may differ. This study will consequently provide us with comparative information of understanding of the concepts in Uganda and Norway, but also explore any differences in understanding for specific Key Concepts. Furthermore, the study in Uganda included children and thus offered us the opportunity to compare our results from the adult population in Norway with those of children in Uganda.

Ethical statement

This study was considered for ethical approval by the Norwegian Institute of Public Health. This project was considered to not to require a full ethical review (reference 18/11854), because no sensitive data was included. All information was collected through Nettskjema (a web-based survey system), ascertaining a high level of data security and safety.

The population was drawn from the National registry who provided us with a CD-ROM with respondents’ addresses. This CD was destroyed once data collection was terminated.

All respondents were given information about the purpose of the study and how the results would be managed and presented. To gain access to the questionnaire, each participant had to provide written consent electronically. The questionnaire was anonymous and once submitted, the information could not be traced back to the respondent.

Development of the questionnaires

Recruitment of participants and administration of the questionnaires

Based on evidence from two systematic reviews that evaluated strategies to improve participation in research^35,36, we developed attractive postcards personally addressed to each potential participant inviting people to take part in the study. The postcard is shown in Figure 1. The postcard included a short description of the purpose of the study as well as a URL (“link”) to one of the four online questionnaires. Participants received full information about the study when they accessed the website and were asked to provide informed consent. Evidence also suggests greater participation rates can be expected if people are informed how the research may benefit them. This was stated on the postcard and the website provided information about how the study results can be accessed.

Figure 1. The postcard used for recruitment.

In January and February 2019, we mailed postcards to a representative sample of 4500 adults (≥18 years) living in Norway. The sample was provided to us by the Norwegian National Registry, considering level of education, sex and geographical spread³⁷. The questionnaires were administrated electronically using Nettskjema, a service provided by University of Oslo³⁸. In addition to the first postcard, one reminder postcard was sent out to each person.

Rasch analysis

The questionnaires were evaluated for their psychometric properties as part of this study. Although some MCQs were tested in Norway as part of a previous pilot, this is the first full-scale psychometric evaluation of the complete available battery of the Claim Evaluation Tools Item bank^26,27.

Rasch analysis is a dynamic and practical approach to address important measurement issues required for validating an outcome^39–41. In this study we followed the fundamental steps of Rasch analysis including testing for internal construct validity (multidimensionality), invariance of the items (Item-Person Interaction), and item bias (differential item functioning), as well as testing for reliability^40,41. Rasch analysis can be used for both dichotomous and polytomous data^40,42,43. Raw data were exported from the electronic data collection service (Nettskjema) as Excel-files and entered into RUMM2030 for Rasch analysis. Results can be replicated using the open-access software WINSTEPS⁴⁴. Each questionnaire set was evaluated separately. Questionnaires 3 included also polytomous questions assessing intended behaviour and self-efficacy questions. Therefore, for questionnaire 3 specifically, additional analysis was done for each item block separately (MCQ- item block and intended behaviour and self-efficacy questions-block respectively).

Survey analysis

Each MCQ was scored as correct or incorrect. The Key Concepts for which there were two MCQs were scored as “understood” if a participant answered both MCQs correctly. For questions about intended behaviours and self-efficacy, we dichotomised the responses in the analysis, for example as likely (very likely or likely) vs not likely (very unlikely, unlikely, or don’t know). As previously mentioned, the Key Concepts are reviewed and revised annually. Over the duration of this study, two Key Concepts were revised into the new Key Concept 1.1a. Considering that the MCQs were no longer appropriately covering this concept, these MCQS were therefore taken out of the analysis.

Based on a previous survey we conducted³², we anticipated that women and respondents with higher education levels would be more likely to respond. To address such non-random non-response, we used iterative post-stratification to match marginal distributions of sex and educational attainment level of the sample to the Norwegian population⁴⁵. We also adjusted for region of residence. Due to an error, we did not collect data on participants’ age and could not use it for post-stratification, as planned.

We obtained Eurostat data on the marginal distributions of sex and region of residence for all Norwegians, and educational attainment level for Norwegians aged 15 to 64 years^46,47. Participants reported their county of residence. We mapped counties to the corresponding Nomenclature of Territorial Units for Statistics (NUTS 2) regions of Norway used by Eurostat⁴⁸. Participants reported the highest level of education they attained. We mapped these to the corresponding International Standard Classification of Education (ISCED) 2011 categories used by Eurostat (levels 0–2, 3–4, and 5–8). We used multiple imputation with chained equations to account for the uncertainty introduced by missing values of the post-stratification variables sex, region of residence, and educational level⁴⁹, and iteratively post-stratified each imputed data set. Based on these adjustments, we estimated the percentage of the Norwegian population that understands each key concept, and that responded positively for each question about intended behaviours and self-efficacy. We used the R packages tidyverse, mice, mitools, and survey.

We present summaries of the results for participants and the post-stratified population estimates for understanding of the 30 Key Concepts, and the intended behaviours and self-efficacy questions. We quantified the uncertainty of our estimates using 95% confidence intervals and protected the family-wise coverage probability of the confidence intervals for the four Key Concepts included in all questionnaires via Bonferroni-correction (i.e., we report a 98.75% CI for each of those concepts). For each Key Concept, we calculated the likelihood of answering both questions (or the one question for two Key Concepts) correctly if participants randomly guessed the answer. These probabilities vary between six and 25%, depending on the number of MCQs and the number of response options (between two and four) for each MCQ.

We compared Norwegian and Ugandan adults’ understanding of the Key Concepts to that of Ugandan children in the intervention arm of our randomized trial. We compared mean test scores, and probabilities of achieving passing and mastery scores, using estimates from one year after the IHC primary school intervention for the Ugandan children and their teachers^50,51 and estimates from the control group for parents in the IHC podcast trial, one year after parents in the intervention group listened to the podcast^51,52. We used predetermined thresholds of at least 13 of 24 questions answered correctly for a passing score and at least 20 of 24 for mastery.

We estimated mean scores and odds ratios using generalized linear mixed-effects models (GLMMs; normal errors and identity link for mean scores, binomial errors and logit link for passing and mastery) using the lme4 R package. In the trials, children and teachers were randomized in clusters (schools), while parents were individually randomized. We modelled this clustering structure as a random intercept for each randomized unit. We did not adjust for covariates such as those used for stratified sampling in the Ugandan trials because those variables are not defined for all samples (e.g., school ownership was used in the trial on Ugandan children, but there is no analogous concept for Norwegian adults). No data were missing. It is not possible to use the lme4 package to apply post-stratification weights for the Norwegians.

Exploratory analyses

We conducted exploratory analyses to investigate associations between understanding each Key Concept and the demographic covariates sex, research training, research participation, education level, and health professional background. Based on findings from our previous survey³², we hypothesised that better understanding of the Key Concepts would be associated with having a research background or a higher education level; that there would be little difference between health professionals and others; and that there would be little difference between women and men. We used generalized linear models (GLMs; quasibinomial errors and logit link) as before, modelled the covariates as categorical variables, and used multiple imputation and post-stratification as in the main analysis.

We used data from the first questionnaire to perform exploratory analyses to investigate how Norwegians’ mean scores and achievement of passing and mastery scores are associated with the demographic covariates sex, research training, research participation, education, and health professional background. We used GLMs (normal errors and identity link for mean score; quasibinomial errors and logit link for passing and mastery) to model the outcomes in terms of the covariates, which were modelled as before. Multiple imputation and post-stratification were used as in the main analysis. The variables included in imputation were the post-stratification variables (sex, region of residence, and educational attainment); demographic variables that coded for whether participants had research training or a health professional background, and whether they had been a research participant; and mean score. We did not include passing and mastery in imputation, as they can be calculated from the mean score.

Sample size calculation

We performed three power analyses. The first two analysed power to estimate differences, of at least 5% from random guessing, in the proportion of the Norwegian population that understands the concepts. The first analysis focused on the four Key Concepts that would be included on all four questionnaires while protecting the 95% coverage of the four confidence intervals as a family, and the second analysis focused on a single prototypical Key Concept that would be included on only one questionnaire. We assumed that each concept would be probed by two MCQs, each having four options, such that we would expect that 6.25% of Norwegians would appear to “understand” a concept if all participants guessed at random. We further calculated confidence interval widths, aiming to estimate proportions with precision no worse than ±0.05 (i.e., ±5%) for those four Key Concepts and no worse than ±0.08 (i.e., ±8%) for the other Key Concepts. The third analysis estimated power to detect a difference of at least 10% in the mean score (proportion of correct answers) between Norwegians and Ugandan children, parents, and teachers. Finally, drawing on experience with our previous survey, we explored a range of response rate scenarios.

These analyses suggested that sending a total of 4500 postcards would be sufficient to achieve a margin of error no greater than ±5% and ±8% for the first two analyses, and approximately 97% power to detect differences greater than 10% in the mean score compared to results from the Ugandan trials.

Participant characteristics

Table 2 shows participant characteristics across the four questionnaires⁵³. Of the 771 respondents, 21 (2.7%) did not provide data on at least one of the covariates sex, research training, research participation, education, health professional background, and county of residence.

The validity and reliability of the questionnaires

The log describing the Rasch analysis is described in more detail in Additional file 2 (Extended data)⁵⁴. Overall, the four questionnaires showed acceptable fit to the Rasch model. Three out of four questionnaires were found to be unidimensional. The questionnaires including intended behaviour and self-efficacy items suggested that this sub-scale may include more than one dimension. Separate analyses of the two item-sets in Questionnaire 3 (Key Concept MCQs and intended behaviour and self-efficacy items) suggest that the MCQ sub-test works very well, with no apparent validity issues. Overall, the analysis of the sub-test consisting of items evaluating intended behaviour and self-efficacy items also shows satisfactory fit. However, two of the intended behaviour items show disordered thresholds. We did not observe any important local dependency in Questionnaire 2, 3 and 4. However, in Questionnaire 1, twelve correlations was observed. Two questionnaires were underpowered for this analysis (Questionnaire 1 and 4). However, based on the available data, most MCQs in all questionnaires showed acceptable fit to the ICC-curve, and no important differential item functioning by gender was identified in any of the questionnaires. Five MCQs showed evidence of under discriminating with one ability class deviating: two MCQs in Questionnaire 1 (q11 and q19.1), two MCQs in Questionnaire 2 (q3 and q18.3), and one MCQ in Questionnaire 4. In the two questionnaires that were adequately powered, Cronbach’s alpha was 0.6 and 0.7, respectively.

Norwegians’ understanding of the Key Concepts

The mean width of the confidence intervals for the four concepts common to all questionnaires was ±4.6% (range ±3.6% to ±5.9%). One concept (An outcome may be associated with a treatment but not caused by it) was not estimated to the desired precision. The mean width of the other confidence intervals was ±7.92% (range ±3.37% to ±11.9%; 15 of the 30 concepts had confidence intervals wider than ±8%).

Figure 2 shows estimates of the percentage of Norwegian adults who understand each Key Concept. The adjusted proportion of Norwegian adults who answered correctly five of the Key Concepts was > 80% and more than 50% understand 17 concepts. On the other hand, less than half answered correctly for 13 concepts, and for seven of those concepts, the proportions of correct answers were no better than if people had randomly guessed.

Figure 2. Estimates of the percentage of Norwegian adults who understand each Key Concept.

Intended behaviours and self-efficacy

Intended behaviours and self-efficacy are presented in Figure 3. The results suggest that most Norwegian adults are likely to question the basis of treatment claims (75%; 95% CI 66–85%), find out if treatment claims are based on research evidence (70%; 95% CI 59–81%), and say “yes” if asked to participate in a study comparing treatments (67%; 95% CI 63–72%). However, only about one out of five Norwegian adults find it easy to assess the relevance of studies that compare treatments (21%; 95% CI 9–33%), assess whether a treatment claim is based on a study comparing treatments (18%; 95% CI 12–25%), find studies that compare treatments (18%; 95% CI 9–28%), or assess the trustworthiness of the results of studies that compare treatments (16%; 95% CI 6–26%).

Figure 3. The Norwegian population’s intended behaviours and self-efficacy.

Comparison of Norwegian and Ugandan adults to Ugandan children

The results for Norwegian and Ugandan adults are compared to results for Ugandan children in Table 3 to Table 5. The mean score for Norwegian adults who participated in the survey (Questionnaire 1) was 17% higher (95% CI 14–20%) than the mean score for Ugandan children in the intervention arm of the randomised trial of the IHC primary school resources one year after the intervention and similar to the mean score for Ugandan teachers in the intervention arm of the trial (Table 3)²³. The scores for Ugandan teachers in the control group were similar to those of the children in the intervention group, and the scores of Ugandan parents in the control group of the IHC podcast trial²⁴ were 16% lower (95% CI -19 to -13%).

Nearly all the Norwegian adults (209 out of 210) who participated in the survey and Ugandan teachers in the intervention group (77 out of 78) had a passing score (Table 4). This was 15% more than the Ugandan children in the intervention group. However, because only one Norwegian and one Ugandan teacher in the intervention group did not have a passing score, the estimated differences may be unreliable for those comparisons. The proportion of Ugandan teachers in the control group with a passing score (88%) was similar to that of the children (84%), and 47% fewer (95% CI -56% to -37%).

Three quarters (66%; 95% CI 56–74%) of the Norwegian adults who participated in the survey and 70% (95% CI 59–80%) of the Ugandan teachers in the intervention group had a mastery score (Table 5). Compared to Ugandan children in the intervention group, the Odds for achieving a mastery score by the Norwegian participants was 5.3 (95% CI 3.5–7.9%), whereas the Odds for achieving a mastery score was 0.24 (95% CI 0.15–0.4) for the Ugandan parents compared to the children.

Associations between participant characteristics and their responses

We did not find strong associations between gender, health professional background, or having participated in research and how well participants did (Table 6). Having 1–2 years of tertiary school education (ISCED levels 5–8) was associated with a mean score that was 11% higher (95% CI 4.0–19%), and a higher likelihood of having a mastery score (OR 3.5; 95% CI 1 to 12). Norwegian men were more likely than women to state that they are likely to find out the basis of treatment claims (OR 3.0; 95% CI 1.1 to 8.2), and more likely to find it easy to assess the trustworthiness of the results of studies that compare treatments (OR 3.8; 95% CI 1 to 14). People who have participated in research may be more likely to find it easy to find research based on studies that compare treatments (OR 5.7; 95% CI 1.9 to 17). People with secondary school education (ISCED levels 3–4) may be less likely than people with no more than primary school education (ISCED levels 0-2) to find it easy to assess the trustworthiness of results of studies that compare treatments (OR 0.057; 95% CI 0.0049 to 0.68). Besides these associations, we did not find evidence suggesting that gender, education level, health professional background, or prior participation in research were associated with intended behaviours or self-efficacy. Associations between participant characteristics and their understanding of specific Key Concepts, intended behaviour and self-efficacy are reported in Additional file 3 (Extended data)⁵⁵.

Comparison with other studies

In the previous Norwegian study, four Key Concepts were evaluated³². In both studies the majority responded correctly to the question “Weighing the benefits and harms”. However, in the present study, the respondents were less likely to respond correctly to “Relative effects of treatments can be misleading” (29% vs 64%) and “The use of p-values may be misleading” (19% vs 51.5%). In contrast, more responded correctly to “An outcome may be associate with a treatment but not caused by it” (64% vs 30%). It is difficult to interpret these differences, other than that these evaluations were done using different question sets. Consequently, one explanation may be difference in the questions’ difficulty level. The gender distribution in the studies were identical, however a higher percentage in the present study had at least one year of education beyond secondary school (68% vs 52%). We did not find gender or having a health professional background to be a good predictor of participants’ understanding of the Key Concepts in any of studies. This is consistent with other studies and findings that both health professionals and patients feel challenged finding, appraising, and applying relevant evidence for use in health decisions^58–63.

There are few other studies to which we can compare these results⁶. The European health literacy survey included all domains of health literacy, responses were given as self-assessments, and results are reported as overall scores²⁵. Consequently, comparison with our study is difficult. They concluded that a little more than one tenth (12%) of those surveyed had insufficient health literacy and almost one half (47%) had limited (insufficient or problematic) health literacy. Both our survey and the European health literacy survey examined associations with education and, unsurprisingly, found that people with higher education have higher health literacy skills. However, the European health literacy survey reported that this result varies by country.

Strengths and limitations

The strength of this study is that we provide new evidence on people’s ability to assess treatment claims using multiple-choice questions. To our knowledge, few such objective surveys have been conducted in Europe. Our work complements studies in which participants assessed their own abilities.

Many conceptualisations of critical thinking exist. Not all are based on explicit criteria, and unlike the Key Concepts used in this survey, few if any are subject to annual revision⁶⁴. The questionnaires we administrated were based on a framework that has been developed from methodological literature and input from multiple disciplines and people with methodological expertise⁶⁵.

We validated the questionnaires using robust methods. Although the results of the Rasch analysis are promising (Additional file 2, Extended data)⁵⁴, it suggests the potential for improvements. Across all four questionnaires, we found that only five MCQs warranted improvement. However, Rasch analysis of two of the questionnaires was underpowered, so the validity and reliability of these should be assessed again in future studies.

Our analysis of the questionnaire including intended behaviours and self-efficacy suggests that intended behaviour and self-efficacy items measure two different dimensions. This might be because intended behaviour is more complex and dependent on self-efficacy, values, knowledge, and other factors. Except for Questionnaire 1, we did not identify any important dependencies between items.

We did not collect information on age for this study, and thus the association between age and ability to assess treatment claims should be explored in future studies. In our previous study conducted in Norway, younger respondents had a higher proportion of correct responses and higher total scores. This may suggest that there may have been improvements over time in the ability of both health professionals and patients to assess treatment claims³².

Another possible limitation of our study is the low response rate, which we anticipated. Our sample was similar to the general population in terms of the percentage of health professionals, but people with a higher education were over-represented among respondents. We addressed this issue using prespecified post-stratification in the survey analyses but did not address this in the comparisons to Ugandans. The results of those analyses therefore cannot be assumed to apply to the general population of Norwegian adults.

Implications and future research

The results of this study can inform the development and evaluation of educational interventions that address Key Concepts that Norwegians appear to poorly understand. Up to now, few such interventions have been evaluated. There is a need to evaluate interventions for health professionals as well as for the general public to help ensure that they can think critically about treatment claims and choices. The results also can inform the development and evaluation or strategies for improving communication of information about the effects of treatments by researchers, health professionals, and others.

Studies like this one in other countries would help to map similarities and differences in people’s abilities across different countries and settings. Such information could help to determine the extent to which interventions should be tailored to address different Key Concepts for different populations.

Health professionals and others who communicate health information should be aware that patients may not be able to think critically about treatment claims and may therefore struggle to process information necessary to making informed decisions.

Our Rasch analysis suggests that the MCQs can be used in Norway. Most MCQs performed well, and this evaluation is the first step in developing a calibrated item bank that can be used for Computer Assisted Testing. The results of our Rasch analysis can be used to improve the multiple-choice questions that did not perform well.

Underlying data

Harvard Dataverse: Replication Data for: Raw data Norwegian Claim Study_2019. https://doi.org/10.7910/DVN/R3DHA5⁵³

This project contains the following underlying data:

With respect to the potential ability to triangulate identities, access to data is restricted. Ethical approval was granted with an informed consent including a statement where people were guaranteed to be anonymous. Prospective data users can contact the authors to access the data, which can be rejected or approved by the data owner based on certain conditions (must be for legitimate research purposes and requestors are required to sign a data use agreement).

Zenodo: multinormal/fhi.informed-health-choices-norway.2019: Version 1.0. https://doi.org/10.5281/zenodo.3669964⁵¹

This project contains the following underlying data:

Extended data

Figshare: Additional file 1. Claim Evaluation Tools_tests administrated in Norway_2019. https://doi.org/10.6084/m9.figshare.11439651.v1³³

Figshare: Additional file 2 Log Rasch analysis Norwegian Claim Study_2019.docx. https://doi.org/10.6084/m9.figshare.10109840.v1⁵⁴

Figshare: Additional file 3. Supplementary tables and figures_Norwegian Claim Study_2019. https://doi.org/10.6084/m9.figshare.11439915.v1⁵⁵

Figshare: Additional file 5. English versions of Claim Evaluation Tools_tests administrated in Norway_2019. https://doi.org/10.6084/m9.figshare.11652342.v1³⁴

Reporting guidelines

Figshare: Additional file 4. STROBE_checklist_cross-sectional_Norwegian Claim Study_2019. https://doi.org/10.6084/m9.figshare.11439753.v2⁶⁶

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).