Key concepts for informed health choices: Where’s the evidence?

Background The Informed Health Choices (IHC) Key Concepts is a framework that provides a basis for developing educational resources and evaluating people’s ability to think critically about health actions. We developed the original Key Concepts framework by reviewing texts and checklists for the public, journalists, and health professionals and collecting structured feedback from an international advisory group. We revised the original 2015 framework yearly from 2016 to 2018 based on feedback and experience using the framework. The objectives of this paper are to describe the development of the framework since 2018 and summarise their basis. Methods For the 2019 version, we responded to feedback on the 2018 version. For the current 2022 version, in addition to responding to feedback on the 2019 version, we reviewed the evidence base for each of the concepts. Whenever possible, we referenced systematic reviews that provide a basis for a concept. We screened all Cochrane methodology reviews and searched Epistemonikos, PubMed, and Google Scholar for methodology reviews and meta-epidemiological studies. Results The original framework included 32 concepts in six groups. The 2019 version and the current 2022 version include 49 concepts in the same three main groups that we have used since 2016. There are now 10 subgroups or higher-level concepts. For each concept, there is an explanation including one or more examples, the basis for the concept, and implications. Over 600 references are cited that support the concepts, and over half of the references are systematic reviews. Conclusions There is a large body of evidence that supports the IHC key concepts and we have received few suggestions for changes since 2019.


Introduction
At the end of the last century, at least 28 frameworks were published describing the need for competencies and curriculum changes for primary and secondary schools in the 21 st century. 1 Since then, many countries have moved or are moving from knowledge-based to competence-or skill-based primary and secondary school curricula. 2[3] Critical thinking is not a new idea.It dates at least to Socrates over 2,400 years ago, and the need to teach critical thinking in school has been argued for over 100 years. 4,5There are many different definitions of critical thinking, 5 and debate about how it should be taught. 6However, generally, the focus is on dispositions and competences that help people to decide what to believe or do.
A major new challenge to deciding what to believe or do is increased access to information online and in social media, and the need to evaluate that information, much of which is misinformation. 4A huge amount of health information can be found online, in addition to information that is disseminated through other channels of communication.This problem has been exacerbated by the COVID-19 pandemic, which has been accompanied by an 'infodemic'an overload of information including false or misleading information. 7 the context of health, the skills needed to decide what to believe or do are sometimes referred to as critical health literacy. 8,9Although both critical thinking and health are widely included in primary and secondary school curricula, critical thinking about health or critical health literacy may not be, [10][11][12][13] and many people find it difficult to make decisions about what to believe or do regarding 'treatments' or 'health actions' (things that they can do to care for their health or the health of others). 14ing able to understand and apply some basic principles or concepts is essential for using reliable information appropriately and avoiding being misled by unreliable information.This includes concepts about claims, comparisons (research evidence from treatment comparisons), and choices (Table 1).As noted by Dewey, 15 "it would be impossible to overestimate the educational importance of arriving at conceptions: that is, of meanings that are general because applicable in a great variety of different instances in spite of their difference; that are constant, uniform, or self-identical in Table 1.Three main groups of concepts and 10 high-level concepts in the 2022 version of the Informed Health Choices (IHC) Key Concepts. 18Claims 2. Comparisons

Choices
Claims about effects that are not supported by evidence from fair comparisons are not necessarily wrong, but there is an insufficient basis for believing them.
Studies should make fair comparisons, designed to minimize the risk of systematic errors (biases) and random errors (the play of chance).
What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms, and costs.
1 REVISED Amendments from Version 1 In response to the Reviewer Reports, we made the following changes.We added information to the Introduction and the Methods, including references.We removed some redundant text, and we made some minor edits.Detailed responses to each of the reviewers' comments, including the changes that we made, can be found in our responses to the Reviewer Reports.A minor spelling correction was made in Tables 1 and 4, and a footnote was added to Table 2.A duplicate file was removed from the Underlying data.
Any further responses from the reviewers can be found at the end of the article what they refer to, and that are standardized, known points of reference by which to get our bearings when we are plunged into the strange and unknown." 16e Informed Health Choices (IHC) Network has developed a framework that includes 49 key concepts as a starting point for deciding what to teach. 17,18The framework provides a basis for developing learning and teaching resources and evaluating learners' ability to think critically about health actions.Most of the concepts in the framework are relevant to other types of actions or interventions, including agricultural, educational, and environmental interventions. 19e concepts serve as the basis for developing learning resources to help people understand and apply the concepts when claims about the effects of health actions are made, and when they make health choices. 20They are also the basis for an item bank of multiple-choice questions (the Claim Evaluation Tools item bank) that can be used for assessing people's ability to apply the IHC Key Concepts. 2e concepts are principles for recognising when a claim about the effects of health actions has an untrustworthy basis, recognising when evidence from comparisons of health actions is trustworthy and when it is not, and making wellinformed health choices.They can help anyone, not just researchers, to think critically about whether to believe a claim and what to do.The concepts are intended for people using research, not for doing research.
The concepts and the basis for the concepts tend to focus on ways in which claims and comparisons (evidence) can be misleading, and choices can be misinformed.This is not because we underestimate the tremendous gains that have been made in health care based on appropriate use of research.Our aim is to promote healthy scepticism, not excessive scepticism, and to help people decide what to believe and what to do.How sure we can be about the effects of health actions varies, as does how sure we can be about the balance between the advantages and disadvantages of health actions.When teaching, learning, or using these concepts, it is important to bear this in mind.
Although most people prefer to make medical decisions together with a health professional, some people prefer to delegate decisions to a health professional. 21Unfortunately, health professionals' perceptions of their patients' desire to be involved in decisions are often inaccurate. 22They may be more likely to underestimate the extent to which patients prefer to be involved in decisions.Regardless of who decides, decisions should be consistent with a patient's values.Decision aids can help patients to clarify their values and may help them to make choices that are more consistent with their values compared to choices made without decision aids. 23There is some evidence that patients choose more conservative approaches when they become better informed. 246][27][28][29][30] Although much of this information can simply be ignored, to avoid being misled by claims with an untrustworthy basis, people need to be able to recognise claims about the effects of health actions, question the basis for those claims, and recognise when a claim is relevant and important, and warrants reflection.
People have the right to participate collectively as well as individually in the planning and implementation of their health care. 31In addition to being a democratic right, public participation in deliberative and decision-making processes has the potential to improve the quality of the judgements and decisions that are made, build trust, improve adherence, and help to ensure transparency and accountability. 32However, the extent to which potential benefits of public participation are achieved (and potential harms are avoided) depends, among other things, on the ability of citizens to think critically about collective decisions as well as personal choices.
People often disagree about the effects of health actions, including experts.Who makes a claim, how likable they are, or how much experience and expertise they have do not provide a reliable basis for assessing how reliable their claim is.This does not mean that conflicting opinions should be given equal weightor that the existence of conflicting opinions means that no conclusion can be reached.It also does not mean that all sources of information are equally reliable.How much weight to give an opinion should be based on the strength of the evidence supporting it.How much trust to place in a source of information about health actions should be based on whether it provides information about the effects of health actions based on systematic reviews. 25e objective of this paper is to describe the development of the IHC Key Concepts from 2019 to 2022 and to summarise the development of the framework over the past decade, its basis, and how it has been used.

Development of the Informed Health Choices Key Concepts 2013-2018
We have previously summarized development of the IHC Key Concepts up to the 2018 version. 17Development of the IHC Key Concepts started in 2013 as the first step in a five-year research project funded by the Research Council of Norway (GLOBVAC project 220603/H10). 33The objectives of this project were to develop and evaluate resources for primary schools and mass media to improve people's ability to assess claims about the effects of treatments.We began by extracting all the concepts addressed in Testing Treatments, 34 a book that was written to promote more critical public assessment of claims about the effects of treatments.We then searched the literature for other relevant material, including books and checklists for the public, journalists, and health professionals. 35Our aim has been to include all concepts that are important for people to consider, while minimising redundancy.
Initially, we collected structured feedback from an international advisory group using four questions: 35 1. Are concepts included that should not be?
2. Are there important concepts that are missing?
3. Are the concepts organised in a logical way?

Do you have any other comments regarding the concepts?
We published the first version of the list in 2015. 35That list included 32 concepts in six groups.Subsequently, we collected feedback at a series of workshops in 2017 and 2018, 17 and from colleagues working in other fields besides health in 2018. 19,36We have used four criteria in deciding on changes to the list of concepts. 17New key concepts must: • be within the scope of the IHC Key Conceptsstandards for judgment, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (research) used to support claims, and to inform treatment choices, • address ways in which treatment claims and comparisons are frequently misleading or ways in which poorly informed decisions are taken, • be useful for people without a research background to use research, not just for researchers or for doing research, and • overlap as little as possible with other key concepts.
In addition, revisions have been informed by a review of related frameworks in 2018, 37 and using the concepts to: • systematically review the effects of educational interventions to improve people's understanding of the key concepts, 38,39 • create a database of educational interventions to improve people's understanding of the key concepts, 40 1][42][43][44][45][46][47][48][49][50][51][52][53] • develop an item bank and outcome measures with multiple-choice questions that assess an individual's understanding of and ability to apply the key concepts, [54][55][56][57][58][59] • develop a plain language glossary of evaluation terms, 60 • ensure coverage of an international core curriculum for teaching evidence-based health care to health professional students, 20,61 • survey the public's ability to assess treatment claims, 14 and • systematically review the quality of information in news reports about the effects of health interventions. 62e published revised versions yearly from 2016 to 2018.The 2016 version included 34 concepts in three groups, and the 2017 version included 36 concepts.The 2018 version included 44 concepts reorganised within each of the three main groups, and we added three subgroups (higher-level concepts) to each of the three main groups.This helped to clarify the logic behind how the concepts were organised and helped to make the long lists of concepts less overwhelming.

Methods
The following invitation was included in the 2018 and 2019 versions: Please send any comments or suggestions to: contact@informedhealthchoices.org.In addition, we asked people to provide feedback at workshops where we presented the Key Concepts.We selected examples that clearly illustrate the concept and that can easily be understood with little explanation by a wide audience without a medical background.
Whenever possible, we have referenced systematic reviews that provide a basis for a concept.We started with reviews with which we were familiar, including some that we had co-authored.Additional systematic reviews were identified by searching and screening the following sources: • All Cochrane methodology reviews 64 (n = 36, on 29 June 2021) • Epistemonikos 65 using the terms "methodology review" OR "meta-epidemiological" in the title or abstract (n = 161, on 11 October 2021) • PubMed using the terms "methodology review" OR "meta-epidemiological" (n = 193, on 11 October 2021) • Google Scholar using the terms "methodology review" OR "meta-epidemiological" in the title, restricted to "Review articles" (n = 370, on 11 October 2021) We did not restrict searches or exclude reviews based on language or the date of publication.
In addition, we searched Epistemonikos, PubMed, and Google Scholar for systematic reviews that support the explanation for each concept using key terms or phrases from the explanation or related terms.These searches were conducted and screened by one of the authors (ADO).The searches varied for each concept.For example, when our initial searches did not find a recent systematic review, we used citation searches in Google Scholar to search for a recent review or more recent research.We did not record the searches that were conducted for each concept.The purpose of these searches was to summarise the evidence supporting each concept, in addition to the logical explanations.
We included systematic reviews of methodological studies and overviews of reviews, such as Cochrane methodology reviews, methodological studies based on a systematic review of research ("meta-epidemiological" studies), and systematic reviews of treatment effects.We categorised as systematic reviews any review of research that included a methods section that described the search strategy for finding studies and selection criteria.

Ethical considerations
This research was undertaken as part of two larger projects funded in part by the Research Council of Norway (Project numbers 220603/H10 and 284683).Because the projects will not generate new knowledge about health and disease, approval by the Regional Committee for Medical Research Ethics in Norway was not required, as confirmed by the committee (reference number 30713).
The only people who participated directly in this research were people who provided feedback on earlier versions of the IHC Key Concepts framework.That feedback was given voluntarily with the understanding that it would be used to improve the framework.

Results
The 2019 version of the framework included 49 concepts. 63We added five new concepts in response to feedback: • Assumptions that fair comparisons are not relevant can be misleading.
• Your own prior beliefs may be wrong.
• Consider the baseline risk or the severity of the symptoms when estimating the size of expected effects.
• Consider how important each advantage and disadvantage is when weighing the pros and cons.
• Important uncertainties about the effects of treatments should be reduced by further fair comparisons.
We organised the concepts in the same way as in the 2018 version of the framework, under three higher-level concepts in each of the three main groups (Table 2).
In response to feedback, 69 we also edited the list of concepts in the 2019 version of the framework to make their descriptions more consistent, and we edited some of the explanations.We also clarified our goal (Table 3), increased the list of competences needed to achieve that goal from 10 to 20, and increased the list of dispositions from 10 to 15.
We received little feedback on the 2019 version (10 suggestions) and decided that revisions were not needed in 2020 and 2021. 69The 2022 version 18 includes the same concepts as the 2019 version.We now provide examples in the explanation for each concept and the basis for each concept, as well as the implications.The 2022 version includes over 600 references, over half of which are to systematic reviews.
We incorporated eight suggestions in the explanations for concepts in this version.For example, a suggestion to include the concept that harms can be irreversible was incorporated in the explanation for the concept "do not assume that treatments are safe" by adding the following to the explanation for that concept: "The harm that is caused may be minor, but treatments also sometimes cause serious, irreversible harms, including death." 69We also reorganised the concepts into four subgroups (high-level concepts) within each of the first two main groups (claims and comparisons) and into two subgroups within the third main group (choices) (Table 1).We did this to make the organisation of the concepts more  *The competences and dispositions were reformulated and reorganised, as well as expanded.
Table 3. Goal of the Informed Health Choices (IHC) Key Concepts.
Our goal is to enable people to make good decisions* about which claims to believe about the effects of things they can do for their health, the health of others or for other reasons, and about what to do to achieve their goals.
*A good decision is one that makes effective use of the information available to the decision maker at the time the decision is made.A good outcome is one that the decision maker likes.The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking (for example genetic and environmental factors). 70 logical and the long list of concepts in some of the subgroups less overwhelming.The 10 high-level concepts also make it easier to get the gist of the concepts and makes the list for some of the subgroups less overwhelming and easier to remember.Table 4 is an overview of the 49 concepts in the three main groups and 10 subgroups.

Discussion
We made many changes to the IHC Key Concepts after they were first published in 2015. 35The original version included 32 concepts in six groups.There are now 49 concepts in three main groups and 10 subgroups.In addition, there are 20 competences and 15 dispositions.There have been few suggestions for changes to the 2019 version and we have made only minor changes to the explanations for some of the concepts.We therefore have decided that the 2022 version will be the last revision made by us. 18This does not mean that this list of concepts cannot be further improved, but we will leave any further development of the IHC Key Concepts to others.
Although we have attempted to use plain language in describing the key concepts and their basis, the list of key concepts is not intended to be a learning resource for people with no relevant research background.It is a framework, or starting point, for identifying and developing learning resources and evaluation tools.][73] We have organised the concepts logically.Although it may sometimes make sense to organise learning resources using the same logic, the logic that is used does not reflect the difficulty of the concepts or the order in which the concepts should be learned.Moreover, the full list of concepts can be overwhelming, and it is likely to be necessary to prioritise which concepts to include in learning resources and evaluation tools. 51For example, some concepts are too difficult for primary school children to understand and use, and it may not be possible to incorporate all the concepts in secondary and primary school curricula. 41,515][76] However, there are many other demands on what to include in primary and secondary school (and other) curricula. 77This is reflected in the findings of a process evaluation of the IHC primary school intervention in Uganda. 44The intervention was shown to have a large effect on primary school children's ability to think critically about health claims, 43 which was sustained after one year. 45eachers who used the primary school intervention in the trial thought the IHC Key Concepts were important.They were motivated to teach the concepts, and the children were enthusiastic about the lessons.Nonetheless, use of the resources has not been scaled up.A key barrier to scaling up use of the intervention was the need to incorporate the lessons in the national curriculum.The IHC lessons were viewed as an addition to what was already a packed primary school curriculum.

Conclusions
The IHC Key Concepts framework is central to critical thinking and evidence-based practice, both of which have broader scopes than this framework. 37An important weakness of frameworks with a broader focus is that they usually do not provide an adequate basis (i.e., necessary concepts) for thinking critically about claims about effects and decisions about what to do.The IHC Key Concepts are applicable to a great variety of claims about the effects of interventions, not just health interventions, 19 and they are essential points of reference for deciding which claims to believe and what to do.
There is a substantial body of evidence supporting the 49 concepts, as well as logic.Although it is possible to further improve this framework, we will leave any further development of the IHC Key Concepts to others.More importantly, there is a need to incorporate the key concepts into school curricula and to develop, evaluate, and scale up the use of effective educational interventions.

Response:
We have edited the sentence to read: A major new challenge to deciding what to believe or do is increased access to information online and in social media, and the need to evaluate that information, much of which is misinformation.

3.
p4: If all of p4 refers to prior revisions, the suggestion is to include the time frame in the heading, e. Response: There was, as is typical, a delay between when the work was done and when it was published.We have edited the first sentence to read: Subsequently, we collected feedback at a series of workshops in 2017 and 2018, 17 and from colleagues working in other fields besides health in 2018. 19,23 have edited the second sentence to read: In addition, revisions have been informed by a review of related frameworks conducted in 2018, 24 5.
p5, 1 st paragraph: refer to the supplemental material for underlying data within the text

Response:
We don't understand this comment.There is not relevant underlying data 6.
for the first paragraph on page 5.The underlying data for this paper is cited in the first paragraph of the Methods section (the second paragraph on page 5).
p6, table 2: include a description of the added competences and disposition in the table (like was done for the added concepts); or provide an explanation why they are not listed; or add reference to where they may be found

Response:
We have added the following footnote to the table : *The competences and dispositions were reformulated and reorganised, as well as expanded. 7.
p7, table 3: "[…] many other factors affect outcomes […]" -this wording sparks curiosity about those other factors and weakens the importance of critical thinking

Response:
We have edited the sentence to read: The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking (for example genetic and environmental factors). 58 p8, 2 nd paragraph: provide an example (within the text) of a suggestion that was incorporated in the explanations for concepts

Response:
We added the following sentence to this paragraph: For example, a suggestion to include the concept that harms can be irreversible was incorporated in the explanation for the concept "do not assume that treatments are safe" by adding the following to the explanation for that concept: "The harm that is caused may be minor, treatments also sometimes cause serious, irreversible harms, including death."p9, data availability: The suggestion is to add the following to the 2 nd sentence: The project contains the following underlying data "in two different formats (pdf, docx)".
As the files have different names, the addition makes it clearer that they contain the same information.

Response:
We deleted the reference to the Word document. 12.
Competing Interests: No competing interests were disclosed.
Reviewer Report 31 October 2023 https://doi.org/10.5256/f1000research.135115.r175378 In the introduction, the authors presented a justification for the IHC initiative, stating that "many people find it difficult to make decisions about what to believe or do regarding treatments or health actions."Typically, individuals rely on their physicians to make these judgments on their behalf.People often feel that it is not their responsibility to make decisions about treatments or health actions.It may be beneficial for the authors to address this point in the introduction to alleviate any concerns readers may have.
In Table 1, the concept presented in item 1.4 states, "Trust based on the source of a claim alone can be misleading."However, this concept poses some challenges.Considering the vast array of expertise in various fields, it is unrealistic for anyone to be able to critically evaluate the validity of every significant claim.For example, personally, I find it difficult to assess claims related to climate change, relying instead on trust in certain sources.In light of this, perhaps the IHC initiative could offer guidance to help individuals discern which sources of information are more trustworthy, rather than emphasizing that trust should not be based solely on the source of a claim.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound?Yes

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?Yes

Are the conclusions drawn adequately supported by the results?
the extent to which potential benefits of public participation are achieved (and potential harms are avoided) depends, among other things, on the ability of citizens to think critically about collective decisions as well as personal choice.In Table 1, the concept presented in item 1.4 states, "Trust based on the source of a claim alone can be misleading."However, this concept poses some challenges.
Considering the vast array of expertise in various fields, it is unrealistic for anyone to be able to critically evaluate the validity of every significant claim.For example, personally, I find it difficult to assess claims related to climate change, relying instead on trust in certain sources.In light of this, perhaps the IHC initiative could offer guidance to help individuals discern which sources of information are more trustworthy, rather than emphasizing that trust should not be based solely on the source of a claim.

Response:
We have added this paragraph to the Introduction: People often disagree about the effects of health actions, including experts.Who makes a claim, how likable they are, or how much experience and expertise they have do not provide a reliable basis for assessing how reliable their claim is.This does not mean that conflicting opinions should be given equal weight -or that the existence of conflicting opinions means that no conclusion can be reached.It also does not mean that all sources of information are equally reliable.How much weight to give an opinion should be based on the strength of the evidence supporting it.How much trust to place in a source of information about health actions should be based on whether it provides information about the effects of health actions that is based on systematic reviews. 76he benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

3. 1
Evidence should be relevant.a) Be clear about what the problem or goal is and what the options are.Consider the relevance of b) the outcomes measured in the research, c) fair comparisons in laboratories, animals, or highly selected people, d) the treatments that were compared, and e) the circumstances in which the treatments were compared.3.2 Expected advantages should outweigh expected disadvantages.a) Weigh the benefits and savings against the harms and costs of acting or not.Consider b) the baseline risk or severity of the symptoms when estimating the size of expected effects, c) how important each advantage and disadvantage is when weighing the pros and cons, d) how certain you can be about each advantage and disadvantage, and e) the need for further fair comparisons.

4 .
g.: "Development of the Informed Health Choices Key Concepts from 2013-2018" Response: We have added 2013-2018 to the heading.These two points created confusion: -p4, 2 nd paragraph: the cited works from colleagues were published after 2018 (Aronson et al., 2019 and Stewart et al., 2022) -p4, 3 rd paragraph: the given reference was published in 2020

74
World Health Organization.Declaration of Alma-Ata.Geneva: World Health Organization; 1978.https://apps.who.int/iris/handle/10665/34787975 Norheim OF, Abi-Rached JM, Bright LK, Baerøe K, Ferraz OLM, Gloppen S, et al.Difficult trade-offs in response to COVID-19: the case for open and inclusive decision making.Nat.Med.2021;27(1):10-3.10.1038/s41591-020-01204-6 Seven individuals provided feedback on the 2018 version and four provided feedback on the 2019 version.The suggestions came from members of the IHC Network and other researchers.For the 2019 version, we reviewed and responded to feedback on the 2018 version.For the current, 2022 version, in addition to reviewing and responding to feedback on the 2019 version, 17,63 we reviewed the evidence base for each of the concepts.For each concept, we have provided one or more examples to illustrate the explanation, and the basis for the concept.The examples, for the most part, were taken from relevant methodological research.Due to the nature of the research, most of the examples are medical.

Table 2 .
Overview of revisions to the Informed Health Choices (IHC) Key Concepts.

Table 4 .
Overview of the Informed Health Choices (IHC) Key Concepts.18

•
Suggestions for changes to the IHC Key Concepts 2018 -2022.pdf.(Suggestions for improvements to the 2018 and 2019 versions of the Informed Health Choices Key Concepts and responses).Informed Health Choices Key Concepts 2022.pdf(Key Concepts for assessing claims about treatment effects and making well-informed treatment choices [Version 2022] -the 2022 version of the Informed Health Choices (IHC) Key Concepts).Nsangi A, Oxman M, et al.: Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach.Pilot Feasibility Stud.2019; 5: 155.PubMed Abstract|Publisher Full Text 63.Oxman AD, Chalmers I, Dahlgren A, et al.: Key Concepts for assessing claims about treatment effects and making well-informed treatment choices.Version 2019.IHC Working Paper.2019.Publisher Full Text 64.Clarke M, Oxman AD, Paulsen E, et al.: Guide to the contents of a Cochrane methodology protocol and review.Cochrane Handbook for Systematic Reviews of Interventions version 5.0.2008.Thinking and Deciding.4th ed.Cambridge, UK: Cambridge University Press; 2008.71.Austvoll-Dahlgren A, Semakula D, Nsangi A, et al.: Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.BMJ Open.2017; 7(5): e013184.Research Center for Inclusive Education (RCIE), Institute of Education Research and Teacher Education, University of Graz, Graz, Austria In general, the paper is easy to follow and well-structured.The objectives are clear, and methods and results are reported in a precise and sound manner.The high degree of transparency is especially notable.Here are a few minor suggestions for clarification: a more substantive description (overview) of the key concepts would be helpful If all of p4 refers to prior revisions, the suggestion is to include the time frame in the heading, e.g.: "Development of the Informed Health Choices Key Concepts from 2013-2018" Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).Zenodo: Key Concepts for assessing claims about treatment effects and making well-informed treatment choices (Version 2022).http://doi.org/10.5281/zenodo.6611932.18Theprojectcontainsthe following underlying data:• 12. Ssenyonga R,Sewankambo NK, Mugagga SK, et al.: Learning to think critically about health using digital technology in Ugandan lower secondary schools: a contextual analysis.PLoS One.2022; 17(2): e0260367.PubMed Abstract|Publisher Full Text 13.Lund HM, Mathisen PE, Rekkavik ME, et al.: Teaching critical thinking about health claims: market analysis for Norwegian primary and lower secondary school.Zenodo.2018.Publisher Full Text 14. Dahlgren A, Furuseth-Olsen K, Rose CJ, et al.: The Norwegian public's ability to assess treatment claims: results of a crosssectional study of critical health literacy.F1000Res.2021; 9(179).Publisher Full Text 15.Dewey J: How We Think.Boston: DC Heath and Company; 1910.Reference Source 16.Dewey J: The educational significance of concepts.How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process.Boston: DC Heath and Company;2nd ed.1933; p. 153-4.17.Oxman AD, Chalmers I, Austvoll-Dahlgren A: Informed Health Choices Group.Key Concepts for assessing claims about treatment effects and making well-informed treatment choices.F1000Res.2019; 7: 1784.26.Glenton C, Paulsen EJ, Oxman AD: Portals to Wonderland: health portals lead to confusing information about the effects of health care.BMC Med Inform Decis Mak.2005; 5: 7. Publisher Full Text 27.Eysenbach G, Powell J, Kuss O, et al.: Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review.JAMA.2002; 287(20): 39.Cusack L, Del Mar CB, Chalmers I, et al.: Educational interventions to improve people's understanding of key concepts in assessing the effects of health interventions: a systematic review.Syst.Rev. 2018; 7(1): 68.PubMed Abstract|Publisher Full Text 40.Castle JC, Chalmers I, Atkinson P, et al.: Establishing a library of resources to help people understand key concepts in assessing treatment claims-The "Critical thinking and Appraisal Resource Library" (CARL).PLoS One.2017; 12(7): e0178666.44.Nsangi A, Semakula D, Glenton C, et al.: Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation.BMJ Open.2019; 9(9): e030787.PubMed Abstract|Publisher Full Text 45.Nsangi A, Semakula D, Oxman AD, et al.: Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects, 1-year follow-up: a cluster-randomised trial.51.Agaba JJ, Chesire F, Mugisha M, et al.: Prioritisation of Informed Health Choices (IHC) Key Concepts to be included in lowersecondary school resources: a consensus study.medRxiv.2022.Publisher Full Text 52.Semakula D, Nsangi A, Oxman AD, et al.: Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about treatment effects: one-year follow up of a randomised trial.Trials.2020; 21(1): 187.PubMed Abstract|Publisher Full Text 53.Ringle VAM: Developing and testing the effects of an educational podcast to improve critical appraisal of healthcare claims.Doctoral dissertation.Miami: University of Miami.2020.54.Austvoll-Dahlgren A, Semakula D, Nsangi A, et al.: Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.BMJ Open.2017; 7(5): e013184.PubMed Abstract|Publisher Full Text 55.Austvoll-Dahlgren A, Guttersrud O, Nsangi A, et al.: Measuring ability to assess claims about treatment effects: a latent trait analysis of items from the 'Claim Evaluation Tools' database using Rasch modelling.BMJ Open.2017; 7(5): e013185.PubMed Abstract|Publisher Full Text 56.Semakula D, Nsangi A, Oxman AD, et al.: Measuring ability to assess claims about treatment effects in English and Luganda: evaluation of multiple-choice questions from the "Claim Evaluation Tools" database using Rasch modelling.2017.61.Albarqouni L, Hoffmann T, Straus S, et al.: Core competencies in evidence-based practice for health professionals: consensus statement based on a systematic review and Delphi survey.JAMA Netw.Open.2018; 1(2): e180281.Publisher Full Text 62. Oxman M, Larun L, Gaxiola GP, et al.: Quality of information in news media reports about the effects of health interventions: systematic review and meta-analyses.F1000Res.2022; 10: 433.○ p4: ○ Is

the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? what
to do.How sure we can be about the effects of health actions varies, as does how sure we can be about the balance between the advantages and disadvantages of health actions.When teaching, learning, or using these concepts, it is important to bear this in mind.
Nsangi A, Oxman M, Rosenbaum SE, Oxman AD, Austvoll-Dahlgren A, et al.Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach.Pilot Feasibility Stud.