Comparison of the Informed Health Choices Key Concepts Framework to other frameworks relevant to teaching and learning how to think critically about health claims and choices: a systematic review

Background: The Informed Health Choices (IHC) Key Concepts are principles for evaluating the trustworthiness of claims about treatment effects. The Key Concepts provide a framework for developing learning-resources to help people use the concepts when treatment claims are made, and when they make health choices. Objective: To compare the framework provided by the IHC Key Concepts to other frameworks intended to promote critical thinking about treatment (intervention) claims and choices. Methods: We identified relevant frameworks from reviews of frameworks, searching Google Scholar, citation searches, and contact with key informants. We included frameworks intended to provide a structure for teaching or learning to think critically about the basis for claims, evidence used to support claims, or informed choices. For a framework to be included, there had to be a description of its purpose; a list of concepts, competences, or dispositions; and definitions of key terms. We made independent assessments of framework eligibility and extracted data for each included framework using standardised forms. Results: Twenty-two frameworks met our inclusion criteria. The purpose of the IHC Framework is similar to that of two frameworks for critical thinking and somewhat similar to that of a framework for evidence-based practice. Those frameworks have broader scopes than the IHC Framework. An important limitation of broad frameworks is that they do not provide an adequate basis (concepts) for deciding which claims to believe and what to do. There was at most some overlap between the concepts, competences, and dispositions in each of the 22 included frameworks and those in the IHC Framework. Conclusions: The IHC Key Concepts Framework appears to be unique. Our review has shown how it and other frameworks can be improved by taking account of the ways in which other related frameworks have been developed, evaluated, and made useful.


Introduction The Informed Health Choices (IHC) Key Concepts Framework
Claims about what people can do to improve or protect their health (treatments) are ubiquitous. They are found in the mass media, advertisements, and everyday personal communication. Some are based on trustworthy evidence. Many are not, and many people have difficulties determining which claims to believe and act on. Acting on untrustworthy claims and not acting on ones that are trustworthy can result in unnecessary suffering and waste.
In response to these challenges, we developed the Informed Health Choices (IHC) Key Concepts as the first step in the IHC project [1][2][3][4] . The aim of the IHC project is to help people, particularly primary and secondary school students, learn to assess treatment claims and make informed health choices 5 .
We use 'treatment' to refer to any intervention or action intended to protect or improve health 6 . People in other fields have found the IHC Key Concepts relevant for assessing claims about the effects of other types of interventions 7 . This includes agricultural, educational, environmental, management, social welfare, economic, international development, nutrition, policing, and veterinary interventions.
The IHC Key Concepts provide a framework for designing curricula, learning resources, and evaluation tools 5,8 . We first published the framework in 2015 1 and have continued to update it yearly. The current (2019) framework includes 49 concepts in three groups (Table 1), 20 competences in four groups (Table 2), and 16 dispositions in four groups (Table 3) 4 . The concepts are principles for evaluating the trustworthiness of treatment claims and the evidence used to support these, and for making informed choices. The methods used to develop the framework are described elsewhere 1,3 . The framework is a starting point to help teachers, journalists, researchers and other intermediaries to identify and develop resources to help people learn to assess treatment claims and make informed choices.

Other frameworks relevant to the IHC Key Concepts Framework
There are many other frameworks that include concepts, competences, or dispositions that are relevant to thinking critically about treatment claims, comparisons, and choices. These include critical thinking frameworks, logical fallacies and argumentation frameworks, cognitive frameworks, frameworks for scientific thinking, and frameworks related to evidence-based health care. For each category of frameworks there are disagreements about definitions and what is included. For example, learning to think critically is widely held as an aim of education 9 , but there is not agreement on the definition of "critical thinking" and there are several different frameworks (conceptual structures intended to serve as a support or guide) for critical thinking 10-14 . Similarly, there are different definitions and frameworks for scientific thinking (reasoning and literacy) [15][16][17][18] , epistemic cognition and meta-cognition 19,20 , health literacy [21][22][23] , and various aspects of evidence-based health care [24][25][26] . There is also overlap across these different framework categories, some of which have been grouped together as frameworks for "productive thinking" 12 .

Terminology
Definitions of terms that we use in this paper are shown in Table 4.

Objective
The objective of our review was to systematically compare the IHC Key Concepts Framework to other frameworks that are relevant to teaching and learning how to think critically about treatment claims, evidence, and choices. We examined similarities and differences between the IHC Key Concepts Framework and other frameworks -particularly in the context of primary and secondary school education -including: • The purposes and definitions of key terms • The elements included and domains in which they are grouped • How the frameworks have been developed and evaluated • How the frameworks have been used to develop curricula, teaching and learning resources, and assessment tools

Methods
We conducted a systematic review of frameworks relevant to teaching and learning to think critically about treatment claims, evidence used to support those claims, and choices. The protocol for the review is published on our website 27 .

Criteria for considering frameworks for inclusion
We included frameworks that are intended to provide a structure for teaching or learning to think critically about at least one of the following: • The basis (justification) for claims or arguments about the effects of interventions and the reliability of those justifications • The extent to which evidence used to support claims about the effects of interventions (comparisons) is fair and reliable • Choices about what to do in order to achieve a goal To be included, the sources for each framework had to include: • a description of the purpose of the framework; • a list of the framework's elements; and • definitions of the key terms used to describe the purpose of the framework, its elements and domains (in which elements are grouped, if there are any).
Frameworks that are modifications of another framework were considered together with the framework that had been modified.

Search methods for identification of frameworks
We began by considering 41 frameworks reviewed in Frameworks for Thinking: A Handbook for Teaching and Learning 12 and frameworks with which we were already familiar [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35] . We searched for other relevant frameworks using Google Scholar between October 2018 and June 2019 using the search strategies found in Extended data File 1. We supplemented these searches by conducting citation searches and contacting key informants for each category of the frameworks.

What to do depends on judgements about a problem, the relevance of the evidence available, and the balance of expected benefits, harms, and costs.
1.1 It should not be assumed that treatments are safe or effective -or that they are not.
a) Treatments can cause harms as well as benefits.
b) Large, dramatic effects are rare.
c) It is rarely possible to be certain about the effects of treatments.

Seemingly logical assumptions are not a sufficient basis for claims.
a) Treatment may not be needed.
b) Beliefs alone about how treatments work are not reliable predictors of the presence or size of effects.
c) Assumptions that fair comparisons of treatments in research are not applicable in practice can be misleading.
d) An outcome may be associated with a treatment but not caused by it.
e) More data is not necessarily better data.
f) Identifying effects of treatments depends on making comparisons.
g) The results of one study considered in isolation can be misleading.
h) Widely used treatments or those that have been used for decades are not necessarily beneficial or safe.
i) Treatments that are new or technologically impressive may not be better than available alternatives.
j) Increasing the amount of a treatment does not necessarily increase its benefits and may cause harm.
k) Earlier detection of 'disease' is not necessarily better.
l) It is rarely possible to know in advance who will benefit, who will not, and who will be harmed by using a treatment.

Trust in a source alone is not a sufficient basis for believing a claim.
a) Your existing beliefs may be wrong.
b) Competing interests may result in misleading claims. c) Personal experiences or anecdotes alone are an unreliable basis for most claims.
d) Opinions alone are not a reliable basis for claims. e) Peer review and publication by a journal do not guarantee that comparisons have been fair.

Comparisons of treatments should be fair.
a) Comparison groups should be as similar as possible.
b) Indirect comparisons of treatments across different studies can be misleading.
c) The people being compared should be cared for similarly apart from the treatments being studied. f) Outcomes should be assessed using methods that have been shown to be reliable.
g) It is important to assess outcomes in all (or nearly all) the people in a study.
h) People's outcomes should be counted in the group to which they were allocated.

Syntheses of studies need to be reliable.
a) Reviews of studies comparing treatments should use systematic methods.
b) Failure to consider unpublished results of fair comparisons may result in estimates of effects that are misleading.
c) Treatment claims based on models may be sensitive to underlying assumptions.

Descriptions should clearly reflect the size of effects and the risk of being misled by the play of chance.
a) Verbal descriptions of the size of effects alone can be misleading.
b) Relative effects of treatments alone can be misleading.
c) Average differences between treatments can be misleading. d) Small studies may be misleading. e) Results for a selected group of people within a study can be misleading.
f) The use of p-values may be misleading; confidence intervals are more informative.
g) Deeming results to be "statistically significant" or "nonsignificant" can be misleading.
h) Lack of evidence of a difference is not the same as evidence of "no difference".

Evidence should be relevant.
a) Attention should focus on all important effects of treatments, and not surrogate outcomes.
b) Fair comparisons of treatments in animals or highly selected groups of people may not be relevant.
c) The treatments compared should be similar to those of interest.
d) There should not be important differences between the circumstances in which the treatments were compared and those of interest.

Expected advantages should outweigh expected disadvantages.
a) Weigh the benefits and savings against the harms and costs of acting or not.
b) Consider the baseline risk or the severity of the symptoms when estimating the size of expected effects.
c) Consider how important each advantage and disadvantage is when weighing the pros and cons. d) Consider how certain you can be about each advantage and disadvantage.
e) Important uncertainties about the effects of treatments should be addressed in further fair comparisons.

Selection of frameworks
One review author (ADO) initially screened frameworks for possible inclusion. Both review authors then independently assessed full-text articles for each potentially relevant framework using an eligibility form (Extended data File 2). We discussed disagreements and reached a consensus. Frameworks that were assessed for inclusion by both authors and then excluded are listed with the reasons for exclusion in Table 5. c) being aware of when people are making treatment claims themselves *A good decision is one that makes effective use of the information available to the decision maker at the time the decision is made. A good outcome is one that the decision maker likes. The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking 36 .

Data collection and assessment of included frameworks
For each included framework, we compiled a list of publications that describe the framework, its development and evaluation, and its use as the basis for curricula, learning resources, and assessment tools.
We recorded independently the following information for each framework, using a data collection form (Extended data File 3): • Its purpose We compared the data that each of us had collected, discussed disagreements, and reached a consensus.
Based on this information, we assessed independently: • strengths and weaknesses of how each framework had been developed and evaluated • strengths and weaknesses of how each framework has been or could be used • any other strengths or weaknesses  Does not provide a framework for thinking critically about claims, comparisons or choices "TRIZ is a systematic, creativity and innovation process devised as an aid to practical problem-solving, especially in engineering." (12, p. 122). De Bono's lateral and parallel thinking tools 12 Does not provide a framework for thinking critically about claims, comparisons or choices The emphasis of this framework is on problem-solving techniques which promote generative, or productive thinking (12, p. 133).
Jewell's reasoning taxonomy for gifted children 12 Does not provide a framework for thinking critically about claims, comparisons or choices This taxonomy is presented, largely from a philosophical perspective, in response to a perceived need to understand how gifted students think and reason. (12, p. 170). Petty's six-phase model of the creative process 12 Does not provide a framework for thinking critically about claims, comparisons or choices Consists of six phases: "inspiration; clarification; evaluation; distillation; incubation; and perspiration" (12, p. 175).
Bailin's intellectual resources for critical thinking 12,37 Does not provide a framework for thinking critically about claims, comparisons or choices Aims at establishing clarity regarding the concept of critical thinking and suggests proposals for an appropriate pedagogy. (12, p. 178). Focus is on "intellectual resources" for critical thinking, which includes "knowledge of key critical concepts", but these are not specified 37  Does not provide a framework for thinking critically about claims, comparisons or choices A description of learner behaviour in complex domains. The main ingredients of this model are an elaboration of the "hypothesis space" and "experiment space", and a representation of learners' knowledge states during discovery. Styles of reasoning framework 15 Does not provide a framework for thinking critically about claims, comparisons or choices This is a broad framework that only indirectly addresses judgments about claims and comparisons.
Scaffolding framework for evidence-based arguments 40,41 Does not provide a framework for thinking critically about claims, comparisons or choices Provides the basis for a website that supports formulating claims and evidence to support claims but does not provide a framework with support for making judgements about the extent to which evidence used to support claims about the effects of interventions is trustworthy. Kuhn's developmental model of critical thinking [42][43][44][45][46][47] This framework is considered together with related epistemological models Focuses on how individuals respond to every day, ill-structured problems that lack definitive solutions.
King and Kitchener's reflective judgment model 42,48 This framework is considered together with related epistemological models Focuses on the epistemic assumptions that underlie reasoning.

Problem solving 49
This framework is considered together with Baron's model of the good thinker 36 Conceptual model of the well-structured problem-solving process.
We compared our assessments, discussed disagreements, and reached a consensus.
Analysis of the data 1. We summarised key characteristics of the included frameworks in tables.
2. Using Venn diagrams, we mapped the extent to which the purposes of the different frameworks overlap with those of the IHC Key Concepts Framework.
3. We compared the concepts, competences and dispositions in each framework with those in the IHC Key Concepts Framework. We considered separately any elements that could not be categorised as concepts, competences or dispositions.
4. We reflected on our assessments of the frameworks and identified implications for how we might improve the IHC Key Concepts Framework, and its usefulness.
We conducted these analyses independently and then compared our analyses, discussed disagreements, and reached consensus.

Results
We screened over 1600 references retrieved using Google Scholar (search strategy: Extended data File 1). In addition, we screened the reference lists in the articles that we retrieved. We identified over 80 frameworks and assessed 35 of these for eligibility based on one or more full-text articles ( Figure 1). We excluded 13 of these (Table 5), so ended up including 22 frameworks (Table 6).
We included four frameworks on critical thinking, three on logic and argumentation, four on cognition, four on scientific thinking, and seven on evidence-based healthcare. We grouped several frameworks together for five types of frameworks -logical fallacies, cognitive biases, epistemological models, systems thinking, and health literacy. We also considered related frameworks together with the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework and the Cochrane Risk of Bias Tool. The purpose and background of each of the included frameworks are shown in Table 6, and definitions of the key term for each framework are shown in Table 7.

Comparison of the included frameworks to the IHC Key Concepts Framework
We summarise our comparison of the included frameworks to the IHC Key Concepts Framework in Table 8. Two frameworks had a similar purpose: Ennis' taxonomy of critical thinking dispositions and abilities 12,50-59 and Baron's model of  Critical thinking skills are those strategies for finding ways to reach a goal.
The list is based on a book published in 1984. The original taxonomy was intended to provide a basis for the national assessment of critical thinking skills in adults in the US. Halpern subsequently revised her taxonomy and presented it, not as a taxonomy, but as a list. Using a normative theory of the nature of good thinking and of how we tend to think poorly to evaluate our actual thinking, and to know how it must be improved. In this way, we can learn to think more rationally, that is, in a way that helps us achieve our goals.

Model of the good thinker
To arrive at a prescriptive model, we ought to find out where people depart from the normative model. Then we can give practical advice to correct these departures. A logical fallacy is a flaw in reasoning. If you are aware of these, you will be better prepared to recognize and defend against them.

Logic and argumentation
There are many lists and different ways of classifying logical fallacies, dating back to Aristotle. To identify concepts and clusters of concepts which adequately define what knowledge a student must possess if he is to critically evaluate everyday discourse.

Taxonomy of concepts and critical abilities related to the evaluation of verbal arguments
The authors took a "view of argument" derived from Toulmin's presentation of inference as a rule-constituted activity and from the nature of the field of ordinary discourse. It is an analysis of concepts related to the evaluation of ordinary argument, relevant to educators concerned with the development of critical thinking skills. To describe changes in assumptions about sources and certainty of knowledge (the development of epistemic assumptions) and how decisions are justified in light of those assumptions (how epistemic assumptions affect the way individuals understand and solve problems).

Evidence based reasoning framework
Epistemology is an area of philosophy concerned with the nature and justification of human knowledge. A growing area of interest for psychologists and educators is that of personal epistemological development and epistemological beliefs: how individuals come to know, the theories and beliefs they hold about knowing, and the manner in which such epistemological premises are a part of and an influence on the cognitive processes of thinking and reasoning. To help account for how people evaluate information, including inaccurate information and the role that cognitions play in people's evaluation of inaccurate (as well as accurate) information.

AIR model of epistemic cognition
Educational and developmental psychologists have investigated human cognitions about epistemic matters. These are cognitions about a network of interrelated topics including knowledge, its sources and justification, belief, evidence, truth, understanding, explanation, and many others. Different researchers have used different terms for these cognitions, including personal epistemology, epistemological beliefs, epistemic beliefs, epistemic positions, epistemic cognition, epistemological reflection, and reflective judgment.

PISA framework for scientific literacy
Organisation for Economic Co-operation and Development (OECD). The Programme for International Student Assessment (PISA) is a collaborative effort among the OECD member governments to provide a new kind of assessment of student achievement on a recurring basis.
1997 [118][119][120][121][122][123][124] The main benefit of constructing and validating the framework is improved measurement. Other potential benefits include: a common language, an analysis of the kinds of knowledge and skills associated with successful performance, and identifying and understanding particular variables that underlie successful performance.
PISA is designed to collect information through three-yearly cycles and presents data on the reading, mathematical and scientific literacy of 15-year-old students, schools and countries. It provides insights into the factors that influence the development of skills and attitudes at home and at school, and examines how these factors interact and what the implications are for policy development.

Framework, who developed it, and when
Purpose Background

Framework for K-12 science education
National Research Council (NRC) Committee on a Conceptual Framework for New K-12 Science Education Standards, USA. The committee included professionals in the natural sciences, mathematics, engineering, cognitive and developmental psychology, the learning sciences, education policy and implementation, research on learning science in the classroom, and the practice of teaching science.
2010 [125][126][127][128][129][130][131][132] To articulate a broad set of expectations for students in science. The overarching goal is to ensure that by the end of 12th grade, all students have some appreciation of the beauty and wonder of science; possess sufficient knowledge of science and engineering to engage in public discussions on related issues; are careful consumers of scientific and technological information related to their everyday lives; are able to continue to learn about science outside school; and have the skills to enter careers of their choice, including (but not limited to) careers in science, engineering, and technology.
The framework was the first part of a two-stage process to produce a next generation set of science standards for voluntary adoption by states in the USA.

Systems thinking
Ideas about holistic thinking and change processes can be traced back to the ancient Greeks. The start of modern systems thinking is attributed the articulation of systems ideas by Ludwig von Bertalanffy, an Austrian biologist who started lecturing and writing in the 1930's on what he called "general system theory"; and to Aleksandr Bogdanov, a Russian revolutionary, philosopher and scientist.
Systems theory is the transdisciplinary study of the abstract organisation of phenomena, independent of their substance, type, or spatial and temporal scale. Systems can be used to represent the complex organisation of virtually any collection of real-world entities into an ordered form that we can better understand. There are several conceptualizations of systems thinking in education. To investigate the relationship of the constructs scientific thinking, selfregulation in research, and creativity in a measurement model.

Model for scientific thinking
Feist investigated whether personality traits consistently distinguish artists from nonartists and scientists from non-scientists. Magno 144 , building on Feist's work 143 , investigated the relationship between scientific thinking, self-regulation, and creativity.

Health literacy frameworks
The term 'health literacy' was first coined in 1974 by Scott Simonds, Professor of Health Education, University of Michigan, School of Public Health, USA. Several frameworks have been developed since then.
Simonds wrote in 1974 that: "Minimum standards for 'health literacy' should be established for all grade levels K through 12. Those school districts that fall below standard should be provided with federal aid to develop programs with teachers qualified to teach health education" 150,151 . Since then, it has been estimated that approximately 80 million Americans have limited health literacy, and multiple studies have found that low health literacy is associated with poorer health outcomes and poorer use of health care services 152 .

Evidence-based practice (EBP) core competencies
International EBP leaders led by team at Bond University, Australia To develop a consensus-based set of core EBP competencies that EBP teaching and learning programs should cover The term evidence-based medicine was first developed in the field of medicine in the early 1990s, but as its use expanded to include other health disciplines, it became known as EBP. EBP provides a framework for the integration of research evidence and patients' values and preferences into the delivery of health care. Although many teaching strategies have been used and evaluated, a lack of EBP knowledge and skills is still one of the most commonly reported barriers to practicing EBP. One of the potential explanations is the inconsistency in the quality and content of the EBP teaching programs.

Framework, who developed it, and when
Purpose Background

GRADE (and related frameworks)
The GRADE Working Group, which includes methodologists, health researchers, systematic review authors, guideline developers 2000 25,30, [154][155][156][157][158][159][160] Grading of Recommendations Assessment, Development, and Evaluation (GRADE) offers a transparent and structured process for developing and presenting summaries of evidence, including its quality, for systematic reviews and recommendations in health care. The purpose of Evidence to Decision (EtD) frameworks is to help people use evidence in a structured and transparent way to inform decisions in the context of clinical recommendations, coverage decisions, and health system or public health recommendations and decisions.
Since the 1970s a growing number of organisations have employed various systems to grade the quality (level) of evidence and the strength of recommendations. Different organisations have used different systems, resulting in confusion and impeding effective communication. The GRADE Working Group began as an informal collaboration of people with an interest in tackling the shortcomings of prior grading systems.

Bradford-Hill criteria
Austin Bradford Hill, Professor Emeritus of Medical Statistics, University of London, UK 1965 28, [161][162][163][164][165] To address: "What aspects of an association between two variables should we especially consider before deciding that the most likely interpretation of it is causation?" This framework was developed to identify the causes of diseases and particularly to determine the role of smoking in lung cancer, but its use has been extended to public health decision making, a domain where questions about causal effects relate to the consequences of interventions that have often been motivated by the identification of causal factors. It has proven useful and has driven decision making in public health for decades.

Critical appraisal
International teachers of evidence-based health care and research methodologists 1981 29, [166][167][168][169][170][171][172][173][174][175][176][177][178][179] To teach critical appraisal of health research. However, some critical appraisal tools are intended primarily for critically appraising research in the context of systematic reviews and some are intended primarily for reporting standards. There is an overlap among these tools and clear distinctions are sometimes not made among tools with different purposes.
"The strategies we shall suggest assume that clinical readers are already behind in their reading and that they will never have more time to read than they do now. For this reason, and because the guides that follow call for closer attention to "Materials and methods" and other matters that often appear in small type, many of the guides recommend tossing an article aside as not worth reading, usually on the basis of quite preliminary evidence. It is only through the early rejection of most articles that busy clinicians can focus on the few that are both valid and applicable in their own practices." 170

Cochrane risk of bias tool (and related frameworks)
International health research methodologists 1980's 29,31,33,180-184 To assess the risk of bias in randomised and non-randomised studies (sometimes referred to as quality or internal validity). Assessments of risk of bias are intended to help interpret findings and explain heterogeneity in systematic reviews; in addition, reviews use risk-of-bias assessments of individual studies in grading the certainty of the evidence. Reviews may exclude studies assessed as high risk of bias.
"The concern about study quality first arose in the early 1980s with the publication of a landmark paper by Tom Chalmers and colleagues and another extensive work by Hemminki, who evaluated the quality of trials done in 1965 through 1975 that were used to support the licensing of drugs in Finland and Sweden 185 .  Critical thinking is the intellectually disciplined process of actively and skilfully conceptualizing, applying, analysing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.

Catalogue of biases
http://www.criticalthinking.org/pages/defining-critical-thinking/766 List of critical thinking skills Critical thinking is the use of those cognitive skills or strategies that increase the probability of a desirable outcome. It is purposeful, reasonable, and goal directed. Also known as directed thinking 68 .

Model of the good thinker
The definition of rationality as "the kind of thinking that helps us achieve our goals. A good decision is one that makes effective use of the information available to the decision maker at the time the decision is made. A good outcome is one that the decision maker likes. The whole point of good thinking is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from good thinking. Good decision making involves sufficient search for possibilities, evidence, and goals, and fairness in the search for evidence and in inference 36 .

Logic and argumentation Logical fallacies
Fallacy is the use of invalid or otherwise faulty reasoning in the construction of an argument.
https://en.wikipedia.org/wiki/Fallacy Taxonomy of concepts and critical abilities related to the evaluation of verbal arguments The evaluation of verbal arguments is the process of applying higher-order concepts (i.e., rules or principles concerning the nature, structure, and tests of argument) to arguments occurring in ordinary verbal discourse in order to assess their acceptability. Such an evaluation requires that one understand numerous concepts and employ diverse critical abilities 85 . Evidence based reasoning framework To participate in arguments about scientific ideas, students must learn how to evaluate and use evidence. That is, apart from what they may already know about the substance of an assertion, students who are scientifically literate should be able to make judgments based on the evidence supporting or refuting that assertion 87 .

Cognition Cognitive biases
Cognitive biases are systematic patterns of deviation from norm or rationality in judgment.
https://en.wikipedia.org/wiki/List_of_cognitive_biases Framework for understanding people's theories about their own cognition Metacognitive theories are theories that integrate one's knowledge about cognition and regulation of cognition. By "theory" we mean a relatively systematic structure of knowledge that can be used to explain and predict a broad range of empirical phenomena. By a "metacognitive theory" we mean a relatively systematic structure of knowledge that can be used to explain and predict a broad range of cognitive and metacognitive phenomena 103 . Epistemological models Definitions of critical thinking are numerous and wide-ranging. However, one non-controversial claim we can make about critical thinking is that it entails awareness of one's own thinking and reflection on the thinking of self and others as an object of cognition. Metacognition, a construct that is assuming an increasingly central place in cognitive development research, is defined in similar terms as awareness and management of one's own thought, or "thinking about thinking." Metacognition originates early in life, when children first become aware of their own and others' minds. But like many other intellectual skills, metacognitive skills typically do not develop to the level we would like 47 . AIR model of epistemic cognition Epistemic cognition refers to the complex of cognitions that are related to the achievement of epistemic ends; notable epistemic ends include knowledge, understanding, useful models, explanations, and the like 116 . Scientific thinking PISA framework for scientific literacy Scientific literacy is an individual's scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, and to draw evidence-based conclusions about science-related issues, understanding of the characteristic features of science as a form of human knowledge and enquiry, awareness of how science and technology shape our material, intellectual, and cultural environments, and willingness to engage in science-related issues, and with the ideas of science, as a reflective citizen 122 . Framework for K-12 science education Science, engineering, and the technologies they influence permeate every aspect of modern life. Indeed, some knowledge of science and engineering is required to engage with the major public policy issues of today as well as to make informed everyday decisions, such as selecting among alternative medical treatments or determining how to invest public funds for water supply options. In addition, understanding science and the extraordinary insights it has produced can be meaningful and relevant on a personal level, opening new worlds to explore and offering lifelong opportunities for enriching people's lives. In these contexts, learning science is important for everyone, even those who eventually choose careers in fields other than science or engineering. By framework we mean a broad description of the content and sequence of learning expected of all students by the completion of high school-but not at the level of detail of grade-by-grade standards or, at the high school level, course descriptions and standards. Instead, as this document lays out, the framework is intended as a guide to standards developers as well as for curriculum designers, assessment developers, state and district science administrators, professionals responsible for science teacher education, and science educators working in informal settings 32 .

Systems thinking
System thinking is the ability to understand and interpret complex systems. Our conceptualisation of systems thinking is based on Riess and Mischo's definition: "as the ability to recognise, describe, model (e.g. to structure, to organise) and to explain complex aspects of reality as systems". According to this definition, Riess and Mischo stressed essential aspects of systems thinking, which include the ability to identify important elements of systems and the varied interdependency between these elements, the ability to recognise dimensions of time dynamics, the ability to construct an internal model of reality and the ability to give explanations, to make prognoses and to develop means and strategies of action based on that model 141 . Model for scientific thinking Scientific thinking is composed of a set of characteristics that includes practical inclination, analytical interest, intellectual independence, and assertiveness 144 . Broadly defined, scientific thinking includes the skills involved in inquiry, experimentation, evidence evaluation, and inference that are done in the service of conceptual change or scientific understanding. Scientific thinking is defined as the application of the methods or principles of scientific inquiry to reasoning or problem-solving situations, and involves the skills implicated in generating, testing and revising theories, and in the case of fully developed skills, to reflect on the process of knowledge acquisition and change. Participants engage in some or all the components of scientific inquiry, such as designing experiments, evaluating evidence and making inferences 16 .

Evidence-based health care Health literacy frameworks
There are various definitions of health literacy. A "new 'all inclusive' comprehensive definition capturing the essence of the 17 definitions identified in the literature" is: Health literacy is linked to literacy and entails people's knowledge, motivation and competences to access, understand, appraise, and apply health information to make judgments and take decisions in everyday life concerning healthcare, disease prevention and health promotion to maintain or improve quality of life during the life course 22 . EBP core competencies Evidence-Based Practice (EBP) is the integration of the best research evidence with clinical expertise and patient's unique values and circumstances. Core competencies are defined as the essential minimal set of a combination of attributes, such as applied knowledge, skills, and attitudes, that enable an individual to perform a set of tasks to an appropriate standard efficiently and effectively 26 . GRADE and related frameworks Quality of evidence (also referred to as certainty of the evidence or certainty of the anticipated effect) is the extent to which one can be confident that an estimate of effect is correct. Strength of the recommendation is the degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects.
https://gdt.gradepro.org/app/handbook/handbook.html#h.svwngs6pm0f2 Bradford Hill criteria An association (or correlation) in statistics is a relationship between two variables in a study, e.g. between having received a particular treatment and having experienced a particular outcome. Causation (a causal association) is an association between two variables where a change in one makes a change in the other one happen.
http://getitglossary.org/ Critical appraisal "Critical appraisal is the systematic evaluation of clinical research papers in order to establish: Bias is the result of "flaws in design, conduct, analyses, and reporting, leading to underestimation or overestimation of the true intervention effect". "It is usually impossible to know the extent to which biases have affected the results of a particular trial" 31 . Catalogue of biases Biases (systematic errors) distort effect estimates away from actual effects. Biases are caused by inadequacies in the design, conduct, analysis, reporting, or interpretation of treatment comparisons. Because it is generally not possible to know the degree to which an effect estimate is biased, judgements must be made about the risk of bias using criteria that assess factors that are known, or thought to be associated with bias, such as unconcealed allocation of participants to treatments. In everyday language, bias has other meanings, for example 'prejudice'.    The goal of the IHC Key Concepts Framework is "To enable people to make good decisions about which claims to believe about the effects of things they can do for their health, the health of others or for other reasons, and about what to do to achieve their goals" 4 . Our formulation of that goal was influenced by how Ennis and Baron formulated their goals. We have adapted Baron's definition of a "good decision" 36 to explain what this means: a good decision is one that makes effective use of the information available to the decision maker at the time the decision is made. A good outcome is one that the decision maker likes. The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking.
The purpose of one of the logic and argumentation frameworks that we included had a somewhat similar purpose to that of the IHC Key Concepts Framework. The evidence-based reasoning framework 87 was developed as an analytic tool intended as a foundation for assessing students' ability to reason from evidence in writing and classroom discussions. The relationship between argumentation -critical evaluation of argumentsand the IHC Key Concepts Framework is illustrated in Figure 3. The purposes of four of the evidence-based health care frameworks were also somewhat similar to the purpose of the IHC Key Concepts Framework: health literacy 21-23,145-149 , the Evidence-based practice (EBP) core competencies 26 , GRADE 25,30,154-160 , and critical appraisal tools 29,166-179 . Figure 4 illustrates the relationship between the evidence-based practice framework and the IHC Key Concepts Framework. Evidence-based practice is a framework for health professionals, whereas the IHC Key Concepts Framework is for young people, patients and the public, and policymakers, as well as health professionals. Evidence-based practice is a broader framework, which includes critical appraisal of other types of evidence besides evidence of effects. It also includes formulating clinical questions, acquiring evidence, and evaluatingn performance,  which are largely outside of the scope of the IHC Key Concepts Framework. The aim of evidence-based practice is to improve health outcomes, and that depends on what health professionals, patients and the public do. Thus, the IHC Key Concepts Framework -critical thinking about effects and choices -is at the centre of evidence-based practice, in much the same way as it is at the centre of critical thinking.
Health literacy also has a broader focus than the IHC Key Concepts Framework. This is most clearly illustrated by Nutbeam's framework 145,148 , which divides health literacy into functional, interactive, and critical health literacy. The IHC Key Concepts Framework is most closely related to critical health literacy, as illustrated in Figure 5.
The GRADE framework overlaps substantially with the IHC Framework with respect to critical thinking about evidence of intervention effects and decisions about what to do, as illustrated in Figure 6. However, the GRADE framework is designed primarily for judgements by authors of systematic reviews, guideline developers, and policymakers. There was at most some overlap between the concepts, competences, and dispositions in the included frameworks and those in the IHC Key Concepts Framework (Table 8). In seven of the 16 frameworks that included concepts, there was some overlap with the IHC Key Concepts Framework. Of the 13 frameworks that included competences, there was some overlap with the IHC Key Concepts Framework in five. There was very little overlap with the dispositions included in eight frameworks.

Development of the frameworks
The methods used to develop the frameworks were clearly described for only 10 of the 22 included frameworks, and the basis was clear for only six (Table 9). In total, 11 of the 22 were based in part on another framework, three on a model or theory, four on a systematic review, nine on an unsystematic review, three on a formal consensus process, and seven on an informal consensus process. The evidence-based practice core competences and Cochrane Risk of Bias Tool were the most systematically developed frameworks. Both were based in part on systematic and unsystematic reviews. The evidence-based practice core competences used a formal consensus process, whereas the Cochrane Risk of Bias Tool used an informal process.
Evaluations of the frameworks Key findings of formal and informal evaluations of the included frameworks are summarised in Table 10. We found formal evaluations of seven of the 22 included frameworks. Methods used to formally evaluate the frameworks included    Our assessment of the elements (concepts, competences or dispositions) in the 22 frameworks is summarised in Table 11. Only one framework, the framework for K-12 science education, had clear inclusion criteria for one of three dimensions ("core ideas"). We judged the elements to be coherent in five frameworks, distinct in nine, and organised logically in eight. There were no inappropriate elements in seven frameworks and no missing elements in two. Overall, the evidence-based reasoning framework 86 was the only framework that we assessed positively for all five criteria (coherent elements, distinct elements, no inappropriate elements, no missing elements, and logical grouping of the elements). That framework is a relatively simple analytic model of arguments about scientific ideas.

Use of the frameworks
Information about how the 22 frameworks have been used is summarised in Extended data File 4. We found evidence that most of the frameworks were being used. For four (the taxonomy of concepts and critical abilities related to the evaluation of verbal arguments, the evidence-based reasoning framework, the AIR model of epistemic cognition, and the model for scientific thinking) we found little evidence of use. Two had only been available for one or two years (the evidence-based practice core competences and the Catalogue of Biases), and we were uncertain about their use. Twelve of the frameworks appeared to be intended primarily for teachers and students, and we found learning resources based on 14 of the frameworks.
Nine of the frameworks appeared to be intended primarily for researchers. One (the evidence-based practice core competences) appeared to be intended primarily for curriculum developers 26 . We found at least some evidence that six other frameworks were used for curriculum development, including three of the critical thinking frameworks. We found evidence that 12 of the frameworks were used as the basis for one or more assessment tools. Other ways in which the frameworks have been used or have been proposed for use  14% 32% * More than one framework was considered. † Although more than one framework was considered, the assessment applies to this specific framework ‡ Yes or yes for some for "clear methods"; yes for other bases : Subject matter specialists in speech developed a taxonomy of concepts and abilities related to verbal argument as used in ordinary discourse. It was the purpose of this study to use data collected to assess these hypothesized abilities to determine, using factor analytic procedures, the construct validity of the taxonomy. Both derived orthogonal and derived oblique factor solutions were obtained for each of three initial factor methods. The major conclusion was that the tests based upon the taxonomy have construct validity at a level of specificity.

Yes
This is a distinctive model in that brings together in an economical form a set of concepts and abilities which can be used in many content areas.
The scope of the model is rather narrow, covering only a subset of the 15 critical thinking abilities identified by Ennis 12 .

Evidence based reasoning framework
No Yes It may be useful as a framework for assessment -evaluation of the quality of arguments 87 .

Cognition and epistemology
Cognitive biases There were numerous limitations to Perry's original study. The scheme's lower positions are more explicitly epistemological than the upper positions, which shift "away from spatial-cognitive restructuring to emotional and aesthetic assessments". Thus, while the epistemological movement from dualism to relativism is clearly noted, how knowledge is construed beyond these positions is less well defined. Perry's work came under attack in the late 1970s for the limitations of generalizing from an elite male sample to the general population of college students. One of the persistent difficulties faced by those who wished to utilize the scheme as more than a theoretical lens has been the difficulty in operationalizing the scheme and in measuring change. Perry did not conduct further research to explore linkages between his conception of epistemological development and student learning, but he did speculate in later work on possible connections among cognitive styles, learning strategies, and development. "When students radically revise their notions of knowledge, would they not be likely to change their ways of going about getting it?" Baxter Magolda attempted to explore gender-related patterns of epistemological development by studying both men and women conducted a longitudinal study of college students at one institution. Epistemology, as it appears to have been defined in this study, largely consisted of student perceptions of learning experiences.
It may be problematic that actual reflective judgment, noted in Stages 6 and 7 of King and Kitchener's framework appears to have been attained by only a minute fraction of those interviewed and has appeared consistently only among advanced graduate students. Responses to the hypothetical problems posed in the interviews may tell us little about how student beliefs are aroused in actual experiences. We know little about how reflective judgment develops in context and just how education makes a difference Kuhn appears to use a simplified three-stage representation of Perry's scheme and offers little information as to the empirical validation of this scheme, but in the connection of epìstemological theories to reasoning. Kuhn shape to meet such a goal." "In this paper it is suggested that retaining the use of scientific literacy is still appropriate, but it is necessary to relate scientific literacy to an appreciation of the nature of science, personal learning attributes including attitudes and also to the development of social values. For this, relevance of the learning plays a role and teaching materials, striving toward student enhancement of scientific literacy, need to consider a societal frame, introduction of conceptual science on a need to know basis, and to embrace the socioscientific situation that provides the relevance for responsible citizenship." "The trend in defining scientific literacy is suggested as away from the short term product approach, in which the facts and skills are paramount, towards the inclusion of issuebased teaching, the need to go beyond scientific problem solving to encompass socioscientific decision making, and the recognition that scientific literacy relates to enabling citizens to effectively participate in the real world. The trend indicates a movement that gives less attention to scientific literacy being viewed as the possession of conceptual understanding of pure science abstract ideas and emphasises more the ability to make decisions related to the technological applications of scientific ideas or socioscientific issues facing society, these being recognized as crucial learning components." 121 There is a widespread critique of many aspects of PISA in academic articles, and from many different disciplines 192 .

Informally evaluated Findings
Framework for K-12 science education Yes There was extensive feedback on a draft 32 . In general, the feedback about the draft framework indicated support for the overall approach. In the online surveys, many individuals commented that they were impressed with the document and thought it provided a good next step toward refining standards for K-12 science education. At the same time, there were many critiques and suggestions for how to improve it. In looking across all the modes of gathering feedback, some key overarching issues emerged: • concerns about the purpose, audience, and voice; • suggestions of additional fields or topics to include; • how best to incorporate and describe ideas in engineering and technology; • concerns that there was too much material; • lack of guidance or examples about how to convey the integration of crosscutting concepts, core ideas, and practices; • insufficient indication of connections to other topics or issues, such as mathematics and literacy; • need for a stronger statement about science for all and insufficient attention to diversity and equity; • lack of "standards" for curriculum, programs, assessment, and professional development similar to those that were included in the National Science Education Standards; and • lack of attention to the challenges inherent in implementing the framework. The framework does not reflect the current, more clearly articulated view of causal processes. Additionally, the guidelines used to evaluate evidence have not changed for decades, even as the causal questions have become more complex, beyond the original intent of this framework. One important limitation of the classic view of disease causation arising from the Hill criteria has been the lack of a formal basis for evaluating causal hypotheses. Only in the past several decades have investigators explored more formally the foundational mathematical and conceptual issues required for rigorous estimation of causal effects, particularly in circumstances where randomization of treatment assignment that ensures exchangeable comparison groups is unfeasible.
The inference about cause became the rationale for intervention, but the causal conclusions were not couched in the consequences of specific actions to reduce or eliminate cigarette smoking. And later, public health action was aimed at the individual smoker, rather than at the upstream system of cigarette manufacture, advertising, and distribution. This limited focus is a key characteristic of the traditional approach; causal determinations were made by epidemiologists and others in public health about various risk factors without considering the effect of a specific way of changing them.
The utility of long-used, familiar approaches for statistical analysis and causal inference to interpret the broad sweep of evidence on the causal determinants of human health is diminishing. Public health practitioners and researchers must understand the limitations of those 162 .
They have to some extent withstood the test of time, in that they are still widely recognised and taught, but they are increasingly being replaced by GRADE (and other frameworks) and they are, in some ways, not consistent with GRADE 165 .

Informally evaluated Findings
Critical appraisal Yes Katrak 29 reported a systematic review of critical appraisal tools.
Many published critical appraisal tools are available to critically appraise research reports. Many of the tools were reported to be modifications of other published tools or reflected specialty concerns in specific clinical or research areas, without attempts to justify inclusion criteria. Few of the generic critical appraisal tools could be usefully applied to any health research. Forty-two different items were extracted from the six critical appraisal tools that could be used to evaluate randomised and nonrandomised studies. The majority of the critical appraisal tools were developed for a specific research design (87%), with most designed for use on randomised studies (38%). There is also a considerable number of critical appraisal tools for systematic reviews (N = 26).
There is a lack of information on tool development processes in most cases. Only 14 out of 121 instruments (12%) were reported as having been constructed using a specified empirical approach. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Face validity was established in nine critical appraisal tools, seven of which were developed for use on experimental studies and two for systematic reviews. Intra-rater reliability was established for only one critical appraisal tool as part of its empirical development process, whereas inter-rater reliability was reported for two systematic review tools (for one of these as part of the developmental process) and seven experimental critical appraisal tools (for two of these as part of the developmental process). Hyde 175 reported a systematic review of evaluations of the effects of critical appraisal workshops, most of which use checklists. Sixteen studies met the inclusion criteria. One study was an RCT, 8 were non-randomised between group studies, and 7 were before-and-after studies. The impact of critical appraisal teaching on clinicians' behaviour (principally reading behaviour) was mixed. Of the eight comparisons for this outcome six had major threats to validity. Most, but not all, of the comparisons showed benefit of critical appraisal teaching, two acting in the opposite direction. Critical appraisal teaching was seen to consistently increase skills: fourteen of the sixteen comparisons for this outcome showed a positive effect. The strength of the effect remained when self-assessed comparisons were removed. Five comparisons were thought not to be subject to major flaws: four of these indicated a benefit of critical appraisal teaching. The strongest and most consistent impact of critical appraisal teaching was seen on knowledge outcomes: 7 of the 12 studies showed a statistically significant positive effect. However, consideration of the size of the benefit revealed heterogeneity. There were four comparisons of the impact on attitudes -all were positive, but it was not possible to separate out real effects from a tendency for participants to respond in a "desired" manner. There were inadequate data to assess whether there was variation in outcome according to the mode of delivery of the educational intervention.

Yes
There is variation among available critical appraisal tools.

Informally evaluated Findings
Risk of bias  * More than one framework was considered. † Although more than one framework was considered, the assessment applies to this specific framework ‡ Yes or yes for some for "clear methods"; yes for "coherence", "distinct", and "logical grouping"; no for "inappropriate elements" and "missing elements" § Does not mix type(s) and specificity of concepts, competencies, or dispositions ** Included concepts, competencies, or dispositions are clearly different from each other † † Concepts, competencies, or dispositions included in the framework that should not have been ‡ ‡ Concepts, competencies, or dispositions not included in the framework that should have been § § Concepts, competencies, or dispositions organised in a way that makes sense include: self-teaching; by parents, institutions, and government; by employers developing training programs; professional development; establishing norms or standards; developing ways of protecting against cognitive biases; theory development; intervention design; policy advice; and reporting standards.
Strengths and weaknesses of the frameworks Strengths and weaknesses of each framework and ideas for further development of the IHC Key Concepts Framework are summarised in Table 12. Strengths of the frameworks related to their development include international collaboration, support from international or national organisations, continued development over a long period of time, well described and systematic development, research evidence to support all of the concepts, elicitation of extensive feedback, and formal comparisons to similar frameworks. Strengths related to their usability include simplicity, a user-friendly structure for describing each concept, and wide use.
Weaknesses of the frameworks include unclear development methods, lack of formal evaluation, multiple frameworks with the same focus and no apparent agreement or effort to reach a consensus on an optimal framework, and complexity or many included concepts or competences.

Ideas for further development of the IHC Key Concepts Framework
We identified several ways in which the IHC Key Concepts Framework might potentially be improved ( Although the IHC framework drew on evidence-based health care frameworks, there was at most some similarity with the purposes of those frameworks and the purpose of the IHC Key Concepts Framework. There was some overlap in terms of concepts with GRADE, critical appraisal tools, and the Catalogue of Bias. There was overlap in terms of competences with health literacy, the evidence-based practice core competences, and critical appraisal tools.  12 conducted a comprehensive review of frameworks for thinking, which overlaps with and informed our review. However, we are unaware of other reviews with the same scope as this review, whether in terms of the included frameworks or the data that were collected for each included framework.
We used explicit inclusion criteria for frameworks and two review authors independently collected data from included frameworks using a data collection form. Both the eligibility assessments and the data collection required judgement.
Although we frequently disagreed, most of our disagreements were minor and all our disagreements were easily resolved. We did not conduct an exhaustive search for relevant frameworks. There may be other frameworks that meet our inclusion criteria. It is possible that other frameworks could add to our findings, but unlikely that they would otherwise substantially change the findings of this review.

Conclusions
As defined by Moseley and colleagues: "Framework is a general term for a structure that provides support" 12  Thanks to the authors for this important and interesting paper.
The Informed Health Choices (IHC) project aims to support people to evaluate the trustworthiness of health claims and to make informed choices. The IHC Key Concepts provide a framework for designing curricula, learning resources and evaluation tools. The framework has been updated and extended regularly. It comprises several concepts, competences and dispositions in different groups.
Objective of this systematic review was to compare the IHC Key Concepts Framework to other frameworks that are relevant to teaching and learning how to think critically about treatment claims, evidence and choices.
The frameworks were mainly identified from reviews of frameworks and by searching Google Scholar. Twenty-two frameworks were included. Two authors independently extracted information on purposes, definitions of key terms, included elements, methods of development and evaluation, and the way the frameworks were used to develop curricula, learning resources and assessment tools. Strengths and weaknesses of each framework were assessed. The authors described the frameworks in detail and displayed differences and similarities in comparison to the IHC Key Concepts Framework. They concluded that the IHC Key Concepts Framework is unique and that it can be improved by taking account of the ways in which other related frameworks have been developed, evaluated, and made useful. The findings can also be used to improve other frameworks.
A possible limitation of the manuscript is the exclusively search in Google Scholar. Underlying algorithms are unknown and searches cannot be replicated reliably. The authors themselves stated that they may have missed frameworks. We agree with them that it is unlikely that additional frameworks would substantially change the findings of the review.
theoretical underpinnings of the IHC Key Concepts Framework in existing frameworks within the fields of health literacy and critical thinking.
Systematic review methods are detailed and appropriate as are process for decision making. The authors did not conduct an exhaustive search. It is possible therefore that additional frameworks may have been excluded. the authors acknowledge this limitation and I agree with them in that it is unlikely that additional frameworks would substantially change the findings of their review.
Comparisons between the IHC framework and the included frameworks in the review are tabulated clearly and the accompanying figures demonstrating relationships between, for example, critical thinking and the key concepts framework are welcomed. It is refreshing to see that the findings of the review led the authors to revise their framework by adding concepts, competencies and dispositions.
Although this paper is focused largely on the context of primary and secondary school education, it has learning for use of the IHC framework outside of these contexts and also for the development and evaluation of other frameworks. The paper also offers a useful structure for cross comparisons of frameworks in any setting or context.

Is the statistical analysis and its interpretation appropriate? Yes
Are the conclusions drawn adequately supported by the results presented in the review? Yes