The Claim Evaluation Tools item bank was originally developed in English for use in the two randomised trials described above ^{23, 24} . Since the item bank was first created items have been translated and validated in several settings and languages, including Uganda (in Luganda and English), Mexico, and China ^{26– 30} .

Our starting point for developing the educational interventions and the Claim Evaluation Tools item bank was a list of Key Concepts that people need to understand and apply to assess claims about treatment effects and make informed health choices (hereafter referred to as Key Concepts) ³¹ ( Table 1). We define “treatment” broadly to include any intervention (action) intended to improve health, including preventive, therapeutic and rehabilitative interventions, and public health or health system interventions. The list of concepts provides a framework for researchers, teachers and others to develop interventions and measure people’s ability to assess treatment claims and make informed choices. We review and amend the list yearly ⁹ .

The item bank is an open-access resource. When we conducted this study, it included at least four multiple-choice questions (MCQs) for 32 of the Key Concepts shown in Table 1. The item bank also includes questions that assess intended behaviours and self-efficacy. Teachers and researchers can select items for tests tailored to their specific needs.

To our knowledge only one survey has attempted to measure the ability to understand and apply any of the IHC Key Concepts in a representative sample of Norwegian adults ³² . That study only addressed four of the Key Concepts. The purpose of this study was to map the ability of Norwegian adults to assess treatment claims and make informed health choices, using MCQs from the Claim Evaluation Tools item bank. The findings can be used to inform the development of learning resources and communication of information to patients and the public, and for international comparisons.

This study was considered for ethical approval by the Norwegian Institute of Public Health. This project was considered to not to require a full ethical review (reference 18/11854), because no sensitive data was included. All information was collected through Nettskjema (a web-based survey system), ascertaining a high level of data security and safety.

The population was drawn from the National registry who provided us with a CD-ROM with respondents’ addresses. This CD was destroyed once data collection was terminated.

All respondents were given information about the purpose of the study and how the results would be managed and presented. To gain access to the questionnaire, each participant had to provide written consent electronically. The questionnaire was anonymous and once submitted, the information could not be traced back to the respondent.

The Claim Evaluation Tools item bank was developed in English. For this study, a language specialist (KFO) translated all items to Norwegian. Three researchers trained in evidence-based medicine and epidemiology reviewed this translation (AD, ADO, and Atle Fretheim). The questions were not modified other than changing some of the names of people in the scenarios to more familiar names in Norwegian. For example, “Dr. Acheng” was changed to “Dr. Anker” (a more familiar name in Norwegian). MCQs for 32 Key Concepts were administrated as part of this study. However, as previously mentioned, the Key Concepts are reviewed and revised annually. Over the duration of this study, two Key Concepts were revised into the new Key Concept 1.1a. Considering that the MCQs were no longer appropriately covering this concept, these MCQS were therefore taken out of the analysis.

All MCQs in the item bank include a scenario with a treatment claim, and response options, with one answer being the “best” (correct) and the remaining options considered “worse” (incorrect). An example of a multiple-choice question is shown in Box 1. We planned on including questions about intended behaviours and self-efficacy, in addition to two MCQs for each Key Concept. Anticipating that few respondents would be prepared to answer this number of MCQs, we split the questions across four tests (see Additional file 1 and Additional file 5, Extended data) ^{33, 34} .

We included a common set of eight MCQs in all four tests, two for each of four Key Concepts:

An outcome may be associated with a treatment but not caused by it.

We considered these four Key Concepts to be amongst the most important Key Concepts that can be understood and used by children, as well as adults. These MCQs were also included in the test used in the randomised trials in Uganda ³⁵ .

The first test was identical to the test used for the randomised trials in Uganda ^{23, 24} , facilitating comparison between Norwegian adults and Ugandan children, parents, and teachers. The second test included the remaining items that address concepts in the first (Claims) category as well as items from the second (Comparisons) category ( Table 1). The third test included items from the third (Choices) category. This test also included three items evaluating intended behaviours regarding assessing treatment claims and agreeing to participate in a study evaluating treatments, and four items regarding self-efficacy. The fourth test included MCQs addressing Key Concepts that we judged to be more difficult, mostly belonging to the Comparisons category.

All four tests included the same set of questions regarding the participant’s sex, level of education, health professional background, training in research methods, and prior participation in research.

Based on evidence from two systematic reviews that evaluated strategies to improve participation in research ^{36, 37} , we developed attractive postcards personally addressed to each potential participant inviting people to take part in the study. The postcard is shown in Figure 1. The postcard included a short description of the purpose of the study as well as a URL (“link”) to one of the four online tests. Participants received full information about the study when they accessed the website and were asked to provide informed consent. Evidence also suggests greater participation rates can be expected if people are informed how the research may benefit them. This was stated on the postcard and the website provided information about how the study results can be accessed.

In January and February 2019, we mailed postcards to a representative sample of 4500 adults (≥18 years) living in Norway. The sample was provided to us by the Norwegian National Registry, considering level of education, sex and geographical spread ³⁸ . The questionnaires were administrated electronically using Nettskjema, a service provided by University of Oslo ³⁹ . In addition to the first postcard, one reminder postcard was sent out to each person.

The tests were evaluated for their psychometric properties as part of this study. Although some MCQs were tested in Norway as part of a previous pilot, this is the first full-scale psychometric evaluation of the complete available battery of the Claim Evaluation Tools Item bank ^{26, 27} .

Rasch analysis is a dynamic and practical approach to address important measurement issues required for validating an outcome measure and can be used for both dichotomous and polytomous data ^{40– 42} . In this study we followed the fundamental steps of Rasch analysis including testing for internal construct validity (multidimensionality), invariance of the items (Item-Person Interaction), and item bias (differential item functioning), as well as testing for reliability ^{41, 42} . Rasch analysis can be used for both dichotomous and polytomous data ^{41, 43, 44} . Raw data were exported from the electronic data collection service (Nettskjema) as Excel-files and entered into RUMM2030 for Rasch analysis. Results can be replicated using the open-access software WINSTEPS ⁴⁵ . Each test set was evaluated separately. Test 3 included also polytomous questions assessing intended behaviour and self-efficacy questions. Therefore, for test 3 specifically, additional analysis was done for each item block separately (MCQ- item block and intended behaviour and self-efficacy questions-block respectively).

Each MCQ was scored as correct or incorrect. The Key Concepts for which there were two MCQs were scored as “understood” if a participant answered both MCQs correctly. For questions about intended behaviours and self-efficacy, we dichotomised the responses in the analysis, for example as likely (very likely or likely) vs not likely (very unlikely, unlikely, or don’t know).

Based on a previous survey we conducted ³² , we anticipated that women and respondents with higher education levels would be more likely to respond. To address such non-random non-response, we used iterative post-stratification to match marginal distributions of sex and educational attainment level of the sample to the Norwegian population ⁴⁶ . We also adjusted for region of residence. Due to an error, we did not collect data on participants’ age and could not use it for post-stratification, as planned.

We obtained Eurostat data on the marginal distributions of sex and region of residence for all Norwegians, and educational attainment level for Norwegians aged 15 to 64 years ^{47, 48} . Participants reported their county of residence. We mapped counties to the corresponding Nomenclature of Territorial Units for Statistics (NUTS 2) regions of Norway used by Eurostat ⁴⁹ . Participants reported the highest level of education they attained. We mapped these to the corresponding International Standard Classification of Education (ISCED) 2011 categories used by Eurostat (levels 0–2, 3–4, and 5–8). We used multiple imputation with chained equations to account for the uncertainty introduced by missing values of the post-stratification variables sex, region of residence, and educational level ⁵⁰ , and iteratively post-stratified each imputed data set. Based on these adjustments, we estimated the percentage of the Norwegian population that understands each key concept, and that responded positively for each question about intended behaviours and self-efficacy. We used the R packages tidyverse, mice, mitools, and survey.

We present summaries of the results for participants and the post-stratified population estimates for understanding of the 30 Key Concepts, and the intended behaviours and self-efficacy questions. We quantified the uncertainty of our estimates using 95% confidence intervals and protected the family-wise coverage probability of the confidence intervals for the four Key Concepts included in all questionnaires via Bonferroni-correction (i.e., we report a 98.75% CI for each of those concepts). For each Key Concept, we calculated the likelihood of answering both questions (or the one question for two Key Concepts) correctly if participants randomly guessed the answer. These probabilities vary between six and 25%, depending on the number of MCQs and the number of response options (between two and four) for each MCQ.

We compared Norwegian and Ugandan adults’ understanding of the Key Concepts to that of Ugandan children in the intervention arm of our randomized trial. We compared mean test scores, and probabilities of achieving passing and mastery scores, using estimates from one year after the IHC primary school intervention for the Ugandan children and their teachers ^{51, 52} and estimates from the control group for parents in the IHC podcast trial, one year after parents in the intervention group listened to the podcast ^{52, 53} . We used predetermined thresholds of at least 13 of 24 questions answered correctly for a passing score and at least 20 of 24 for mastery.

We estimated mean scores and odds ratios using generalized linear mixed-effects models (GLMMs; normal errors and identity link for mean scores, binomial errors and logit link for passing and mastery) using the lme4 R package. In the trials, children and teachers were randomized in clusters (schools), while parents were individually randomized. We modelled this clustering structure as a random intercept for each randomized unit. We did not adjust for covariates such as those used for stratified sampling in the Ugandan trials because those variables are not defined for all samples (e.g., school ownership was used in the trial on Ugandan children, but there is no analogous concept for Norwegian adults). No data were missing. It is not possible to use the lme4 package to apply post-stratification weights for the Norwegians.

We conducted exploratory analyses to investigate associations between understanding each Key Concept and the demographic covariates sex, research training, research participation, education level, and health professional background. Based on findings from our previous survey ³² , we hypothesised that better understanding of the Key Concepts would be associated with having a research background or a higher education level; that there would be little difference between health professionals and others; and that there would be little difference between women and men. We used generalized linear models (GLMs; quasibinomial errors and logit link) as before, modelled the covariates as categorical variables, and used multiple imputation and post-stratification as in the main analysis.

We used data from the first questionnaire to perform exploratory analyses to investigate how Norwegians’ mean scores and achievement of passing and mastery scores are associated with the demographic covariates sex, research training, research participation, education, and health professional background. We used GLMs (normal errors and identity link for mean score; quasibinomial errors and logit link for passing and mastery) to model the outcomes in terms of the covariates, which were modelled as before. Multiple imputation and post-stratification were used as in the main analysis. The variables included in imputation were the post-stratification variables (sex, region of residence, and educational attainment); demographic variables that coded for whether participants had research training or a health professional background, and whether they had been a research participant; and mean score. We did not include passing and mastery in imputation, as they can be calculated from the mean score.

We performed three power analyses. The first two analysed power to estimate differences, of at least 5% from random guessing, in the proportion of the Norwegian population that understands the concepts. The first analysis focused on the four Key Concepts that would be included on all four tests while protecting the 95% coverage of the four confidence intervals as a family, and the second analysis focused on a single prototypical Key Concept that would be included on only one test. We assumed that each concept would be probed by two MCQs, each having four options, such that we would expect that 6.25% of Norwegians would appear to “understand” a concept if all participants guessed at random. We further calculated confidence interval widths, aiming to estimate proportions with precision no worse than ±0.05 (i.e., ±5%) for those four Key Concepts and no worse than ±0.08 (i.e., ±8%) for the other Key Concepts. The third analysis estimated power to detect a difference of at least 10% in the mean score (proportion of correct answers) between Norwegians and Ugandan children, parents, and teachers. Finally, drawing on experience with our previous survey, we explored a range of response rate scenarios.

These analyses suggested that sending a total of 4500 postcards would be sufficient to achieve a margin of error no greater than ±5% and ±8% for the first two analyses, and approximately 97% power to detect differences greater than 10% in the mean score compared to results from the Ugandan trials.

Table 2 shows participant characteristics across the four questionnaires ⁵⁴ . Of the 771 respondents, 21 (2.7%) did not provide data on at least one of the covariates sex, research training, research participation, education, health professional background, and county of residence.

The log describing the Rasch analysis is described in more detail in Additional file 2 ( Extended data) ⁵⁵ . Overall, the four tests showed acceptable fit to the Rasch model. Three out of four tests were found to be unidimensional. The test including intended behaviour and self-efficacy items suggested that this sub-scale may include more than one dimension. Separate analyses of the two item-sets in Test 3 (Key Concept MCQs and intended behaviour and self-efficacy items) suggest that the MCQ sub-test works very well, with no apparent validity issues. Overall, the analysis of the sub-test consisting of items evaluating intended behaviour and self-efficacy items also shows satisfactory fit. However, two of the intended behaviour items show disordered thresholds. We did not observe any important local dependency in Test 2, 3 and 4. However, in Test 1, twelve correlations was observed. Two tests were underpowered for this analysis (Test 1 and 4). However, based on the available data, most MCQs in all tests showed acceptable fit to the ICC-curve, and no important differential item functioning by gender was identified in any of the tests. Five MCQs showed evidence of under discriminating with one ability class deviating: two MCQs in Test 1 (q11 and q19.1), two MCQs in Test 2 (q3 and q18.3), and one MCQ in Test 4. In the two tests that were adequately powered, Cronbach’s alpha was 0.6 and 0.7, respectively.

The mean width of the confidence intervals for the four concepts common to all questionnaires was ±4.6% (range ±3.6% to ±5.9%). One concept (An outcome may be associated with a treatment but not caused by it) was not estimated to the desired precision. The mean width of the other confidence intervals was ±7.92% (range ±3.37% to ±11.9%; 15 of the 30 concepts had confidence intervals wider than ±8%).

Figure 2 shows estimates of the percentage of Norwegian adults who understand each Key Concept. We estimate that more than 80% of Norwegian adults understand five of the Key Concepts and more than 50% understand 18 concepts. On the other hand, less than half of Norwegian adults understand 13 of the Key Concepts, and for seven of those concepts, the proportions of correct answers were no better than if people had randomly guessed.

Intended behaviours and self-efficacy are presented in Figure 3. The results suggest that most Norwegian adults are likely to question the basis of treatment claims (75%; 95% CI 66–85%), find out if treatment claims are based on research evidence (70%; 95% CI 59–81%), and say “yes” if asked to participate in a study comparing treatments (67%; 95% CI 63–72%). However, only about one out of five Norwegian adults find it easy to assess the relevance of studies that compare treatments (21%; 95% CI 9–33%), assess whether a treatment claim is based on a study comparing treatments (18%; 95% CI 12–25%), find studies that compare treatments (18%; 95% CI 9–28%), or assess the trustworthiness of the results of studies that compare treatments (16%; 95% CI 6–26%).

The results for Norwegian and Ugandan adults are compared to results for Ugandan children in Table 3 to Table 5. The mean score for Norwegian adults who participated in the survey (Test 1) was 17% higher (95% CI 14–20%) than the mean score for Ugandan children in the intervention arm of the randomised trial of the IHC primary school resources one year after the intervention and similar to the mean score for Ugandan teachers in the intervention arm of the trial ( Table 3) ²³ . The scores for Ugandan teachers in the control group were similar to those of the children in the intervention group, and the scores of Ugandan parents in the control group of the IHC podcast trial ²⁴ were 16% lower (95% CI -19 to -13%).

Nearly all the Norwegian adults (209 out of 210) who participated in the survey and Ugandan teachers in the intervention group (77 out of 78) had a passing score ( Table 4). This was 15% more than the Ugandan children in the intervention group. However, because only one Norwegian and one Ugandan teacher in the intervention group did not have a passing score, the estimated differences may be unreliable for those comparisons. The proportion of Ugandan teachers in the control group with a passing score (88%) was similar to that of the children (84%), and 47% fewer (95% CI -56% to -37%).

Three quarters (66%; 95% CI 56–74%) of the Norwegian adults who participated in the survey and 70% (95% CI 59–80%) of the Ugandan teachers in the intervention group had a mastery score ( Table 5). Compared to Ugandan children in the intervention group, the Odds for achieving a mastery score by the Norwegian participants was 5.3 (95% CI 3.5–7.9%), whereas the Odds for achieving a mastery score was 0.24 (95% CI 0.15–0.4) for the Ugandan parents compared to the children.

We did not find strong associations between gender, health professional background, or having participated in research and how well participants did ( Table 6). Having 1–2 years of tertiary school education (ISCED levels 5–8) was associated with a mean score that was 11% higher (95% CI 4.0–19%), and a higher likelihood of having a mastery score (OR 3.5; 95% CI 1 to 12). Norwegian men were more likely than women to state that they are likely to find out the basis of treatment claims (OR 3.0; 95% CI 1.1 to 8.2), and more likely to find it easy to assess the trustworthiness of the results of studies that compare treatments (OR 3.8; 95% CI 1 to 14). People who have participated in research may be more likely to find it easy to find research based on studies that compare treatments (OR 5.7; 95% CI 1.9 to 17). People with secondary school education (ISCED levels 3–4) may be less likely than people with no more than primary school education (ISCED levels 0-2) to find it easy to assess the trustworthiness of results of studies that compare treatments (OR 0.057; 95% CI 0.0049 to 0.68). Besides these associations, we did not find evidence suggesting that gender, education level, health professional background, or prior participation in research were associated with intended behaviours or self-efficacy. Associations between participant characteristics and their understanding of specific Key Concepts, intended behaviour and self-efficacy are reported in Additional file 3 ( Extended data) ⁵⁶ .

In the previous Norwegian study, four Key Concepts were evaluated ³² . In both studies the majority responded correctly to the question “Weighing the benefits and harms”. However, in the present study, the respondents were less likely to respond correctly to “Relative effects of treatments can be misleading” (29% vs 64%) and “The use of p-values may be misleading” (19% vs 51.5%). In contrast, more responded correctly to “An outcome may be associate with a treatment but not caused by it” (64% vs 30%). It is difficult to interpret these differences, other than that these evaluations were done using different question sets. Consequently, one explanation may be difference in the questions’ difficulty level. The gender distribution in the studies were identical, however a higher percentage in the present study had at least one year of education beyond secondary school (68% vs 52%). We did not find gender or having a health professional background to be a good predictor of participants’ understanding of the Key Concepts in any of studies. This is consistent with other studies and findings that both health professionals and patients feel challenged finding, appraising, and applying relevant evidence for use in health decisions ^{59– 64} .

There are few other studies to which we can compare these results ⁶ . The European health literacy survey included all domains of health literacy, responses were given as self-assessments, and results are reported as overall scores ²⁵ . Consequently, comparison with our study is difficult. They concluded that a little more than one tenth (12%) of those surveyed had insufficient health literacy and almost one half (47%) had limited (insufficient or problematic) health literacy. Both our survey and the European health literacy survey examined associations with education and, unsurprisingly, found that people with higher education have higher health literacy skills. However, the European health literacy survey reported that this result varies by country.

The strength of this study is that we provide new evidence on people’s ability to assess treatment claims using multiple-choice questions. To our knowledge, few such objective surveys have been conducted in Europe. Our work complements studies in which participants assessed their own abilities.

Many conceptualisations of critical thinking exist. Not all are based on explicit criteria, and unlike the Key Concepts used in this survey, few if any are subject to annual revision ⁶⁵ . The questionnaires we administrated were based on a framework that has been developed from methodological literature and input from multiple disciplines and people with methodological expertise ⁶⁶ .

We validated the questionnaires using robust methods. Although the results of the Rasch analysis are promising (Additional file 2, Extended data) ⁵⁵ , it suggests the potential for improvements. Across all four tests, we found that only five MCQs warranted improvement. However, Rasch analysis of two of the tests was underpowered, so the validity and reliability of these should be assessed again in future studies.

Our analysis of the test including intended behaviours and self-efficacy suggests that intended behaviour and self-efficacy items measure two different dimensions. This might be because intended behaviour is more complex and dependent on self-efficacy, values, knowledge, and other factors. Except for Test 1, we did not identify any important dependencies between items.

We did not collect information on age for this study, and thus the association between age and ability to assess treatment claims should be explored in future studies. In our previous study conducted in Norway, younger respondents had a higher proportion of correct responses and higher total scores. This may suggest that there may have been improvements over time in the ability of both health professionals and patients to assess treatment claims ³² .

Another possible limitation of our study is the low response rate, which we anticipated. Our sample was similar to the general population in terms of the percentage of health professionals, but people with a higher education were over-represented among respondents. We addressed this issue using prespecified post-stratification in the survey analyses but did not address this in the comparisons to Ugandans. The results of those analyses therefore cannot be assumed to apply to the general population of Norwegian adults.

The results of this study can inform the development and evaluation of educational interventions that address Key Concepts that Norwegians appear to poorly understand. Up to now, few such interventions have been evaluated. There is a need to evaluate interventions for health professionals as well as for the general public to help ensure that they can think critically about treatment claims and choices. The results also can inform the development and evaluation or strategies for improving communication of information about the effects of treatments by researchers, health professionals, and others.

Studies like this one in other countries would help to map similarities and differences in people’s abilities across different countries and settings. Such information could help to determine the extent to which interventions should be tailored to address different Key Concepts for different populations.

Health professionals and others who communicate health information should be aware that patients may not be able to think critically about treatment claims and may therefore struggle to process information necessary to making informed decisions.

Our Rasch analysis suggests that the MCQs can be used in Norway. Most MCQs performed well, and this evaluation is the first step in developing a calibrated item bank that can be used for Computer Assisted Testing. The results of our Rasch analysis can be used to improve the multiple-choice questions that did not perform well.

Harvard Dataverse: Replication Data for: Raw data Norwegian Claim Study_2019. https://doi.org/10.7910/DVN/R3DHA5 ⁵⁴

This project contains the following underlying data: -

Quiz-1-data.xlsx

With respect to the potential ability to triangulate identities, access to data is restricted. Ethical approval was granted with an informed consent including a statement were people were guaranteed to be anonymous. Prospective data users can contact the authors to access the data, which can be rejected or approved by the data owner based on certain conditions (must be for legitimate research purposes and requestors are required to sign a data use agreement).

Zenodo: multinormal/fhi.informed-health-choices-norway.2019: Version 1.0. https://doi.org/10.5281/zenodo.3669964 ⁵²

This project contains the following underlying data:

Uganda-children-one-year-follow-up.csv

Figshare: Additional file 1. Claim Evaluation Tools_tests administrated in Norway_2019. https://doi.org/10.6084/m9.figshare.11439651.v1 ³³

Figshare: Additional file 2 Log Rasch analysis Norwegian Claim Study_2019.docx. https://doi.org/10.6084/m9.figshare.10109840.v1 ⁵⁵

Figshare: Additional file 3. Supplementary tables and figures_Norwegian Claim Study_2019. https://doi.org/10.6084/m9.figshare.11439915.v1 ⁵⁶

Figshare: Additional file 5. English versions of Claim Evaluation Tools_tests administrated in Norway_2019. https://doi.org/10.6084/m9.figshare.11652342.v1 ³⁴

Figshare: Additional file 4. STROBE_checklist_cross-sectional_Norwegian Claim Study_2019. https://doi.org/10.6084/m9.figshare.11439753.v2 ⁶⁷

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).