Systematic assessment of benefits and risks: study protocol for a multi-criteria decision analysis using the Analytic Hierarchy Process for comparative effectiveness research

Study population

We will invite a small group of at least eight diabetes experts from clinical (primary care, endocrinology, and pharmacy), research (epidemiology and clinical trials), operations (pharmacy and therapeutics), and public health disciplines (department of public health) related to diabetes treatment to participate. Recruitment methods will include invitations extended by email with attachment of an approved document (Appendix A). This project received ethical approval from the Institutional Review Board of Johns Hopkins University (Approval Number: NA_00048562).

Study intervention

The study intervention will be a four part structured interview consisting of a) overview of current medications and options for type 2 diabetes; b) a MCDA using the AHP; c) a cognitive interview; d) evaluation of the AHP-based priority assessment procedure.

Overview of current medications and options for type 2 diabetes

We will use the most updated version of the regulatory label for the treatment options for type 2 diabetes. When data on outcomes is not available, we will use data from our Agency for Healthcare Research and Quality (AHRQ) comparative effectiveness review¹⁵.

The AHP

The first step in AHP analysis involves defining the goal of the decision, the alternatives being considered, and the criteria to determine how well the alternatives can be expected to meet the goal^5–8. These are organized into a hierarchical decision model with the goal at the top, the alternatives at the bottom, and the criteria in between (Figure 1). In the second step, information about how well the alternatives can be expected to fulfill the decision criteria is collected. The third step consists of two parts: a) comparing the ability of the alternatives to fulfill the prespecified criteria, and b) judging the importance of the criteria relative to the decision goal. Pairs of alternative options are sequentially compared regarding their ability to fulfill the criteria using pairwise comparisons⁷.

Figure 1. Final Analytical Hierarchy Process Model.

The first step in Analytic Hierarchy Process analysis consists of defining the goal of the decision, the alternatives being considered, and the criteria to determine how well the alternatives can be expected to meet the goal. As seen above, these are organized into a hierarchical decision model with the goal at the top, alternatives at the bottom, and criteria in between.

A combined normalized ratio scale summarizes the results of the direct and indirect comparisons. Priorities for alternatives are compared using ratios; relative differences of 1.1 are considered significant according to standard AHP criteria¹⁰. A ratio, or relative difference, of 1.1 between two alternatives implies a 10% multiplicative difference with respect to how the alternatives meet a given objective at the next level above in the hierarchy. To measure the quality of AHP, we will measure the internal consistency of the judgments within a set of pairwise comparisons using a measure called the consistency ratio. A consistency ratio of 0 indicates perfect consistency. By convention, consistency ratios < 0.15 were considered acceptable⁷.

Separate judgements are made for various decision perspectives. After the pairwise comparisons of alternatives, the pairwise comparison methods are used to determine the priorities of the criteria relative to the decision goal⁷.

In the fourth step, the scales created in step three are combined to create a summary score indicating how well the alternatives can be expected to meet the decision goal⁷.

This is similar to estimating a weighted average by multiplying the scores indicating how well the alternative options meet the decision goal by the priorities assigned to the criteria and adding the results⁷. The priority of a given alternative with respect to meeting an objective at the next level up in the hierarchy is obtained by summing the products of the alternative weight and each objective weight at the level below in the hierarchy. The pair wise ratings will be transformed into relative weights by calculation of the right principal eigenvector of the relevant matrix (e.g., matrix of the pairwise comparisons between objectives at one level of the hierarchy). Expert Choice, uses the matrix multiplication method, considered to be accurate¹⁶, for this calculation. The resulting scores are commonly expressed as percentages and indicate the alternative medications' relative abilities to fulfill the decision goal.

The fifth step consists of sensitivity analyses to explore the effects of changing the estimates or judgements used in the original analysis.

All AHP analyses will be conducted using Expert Choice 11.5 standard program. We will use the ideal synthesis mode with which rank is preserved in the case of addition or removal of an 'irrelevant' alternative¹⁷. In the ideal mode, the priorities of alternatives or options at a given level of the hierarchy are divided by the priority of the highest-scoring alternative or objective (the “ideal”), and the results are weighted¹⁰.

In our example, this means that the two identical highest-priority alternatives will both receive the same weight, and it will not make any difference to the weights of the other alternatives if both or only one of the irrelevant alternatives are included. In the distributive mode, priorities are divided by the sum of priorities to give a normalized weight and this allows for rank reversal¹⁰. We will examine our results for robustness using the distributive mode.

We will also conduct a cognitive interview and process evaluation to evaluate the user perspectives on the process. The cognitive interview will consist of an investigator accompanying the participants as they complete the AHP task and asking a range of probing questions. Questioning will be guided by a checklist to ensure that all important aspects of the hierarchy, the choice tasks, and the instrument itself were functioning as intended. Qualitative feedback will be entered into NVivo (http://www.qsrinternational.com/products_nvivo.aspx), coded and analyzed to identify recurring themes. Once all participants have completed the AHP, they will be invited back for a group session. At this session group results will be presented and discussed and the participants will be asked to complete an evaluation form (Appendix B) regarding their experience with, and confidence in the AHP method. The operationalization of the AHP for our study is described below.

Step one: Defining the decision context and creating an AHP decision model

The decision goal will be placed at the top of the hierarchy. The level of the hierarchy below the decision goal will include the general objectives. More specific objectives will be placed below the general objectives with each lower level consisting of more specific objectives. General objectives and more specific objectives at a given level of the hierarchy will be comparable. We will aim for seven or fewer objectives on any given level of the hierarchy, and objectives will be expressed positively. The pharmacologic alternatives will be placed at the lowest level of the hierarchy. The decision context, model content, and structure of the hierarchy will be validated for content and refined through in-person group sessions with experts. The panel of experts will consist of clinical experts (e.g., internists, endocrinologists, and pharmacists) and research experts (e.g., epidemiologists and clinical trialists). The decision context and hierarchical model will be presented to a panel of experts on at least two occasions. Experts will be invited to these validation sessions informally in-person or by e-mail. Sessions will be 60 to 90 minutes in duration. After each session, the expert feedback obtained will be synthesized and incorporated into a revised version of the model and decision context. During each expert session, a member of the study team will present an overview of the use of the AHP in decision-making. A list of three to five open-ended questions will be distributed in hard copy to the group for feedback on the model (Supplementary Table 1).

The decision context will be ranking medication for type 2 diabetes. The clinical scenario presented will be an adult patient with moderately uncontrolled type 2 diabetes (glycated hemoglobin-7–9 g/dl). The stated decision goal will be to rank the options for type 2 diabetes treatment. Two criteria will be defined as determining the best treatment of type 2 diabetes: 1) to maximize benefits via glucose reduction; 2) to minimize medication adverse effects. The criteria for maximizing benefits will be focused on reducing glycated hemoglobin; the criteria for minimizing medication adverse events will be sub-divided into two sub criteria: minimizing non-serious harm and minimizing serious harm. The serious harms include severe hypoglycemia, congestive heart failure, acute pancreatitis, and bladder cancer. The non-serious harms include fractures, weight gain and gastrointestinal symptoms. The five medication options that will be considered are metformin, sulfonylureas, exenatide, sitagliptin and pioglitazone.

Step two: assembling and organizing outcome information and presentation of the evidence matrix

We will use several sources of data for the decision-analysis. The treatment-specific probabilities or mean differences for objectives at the lowest level of the hierarchy will be identified from the FDA label or medical review documents available for each drug as available on the FDA website. In the absence of quantitative data on the specific objectives, we will substitute data from a Comparative Effectiveness Review of diabetes medications developed under contract from AHRQ¹⁸.

These treatment-specific quantitative data (“evidence matrix”) will be presented relative to either the comparator identified in the decision context, or to placebo/usual care (see section on, “Presentation of objectives”).

The data from the evidence matrix will be formatted to create a visual representation which facilitates interpretation. Formats for presentation will be considered based on Dolan et al.¹⁹ and will include display of risks on a plot with 2 axes; bar charts displaying risks; flow charts displaying risks; and pictograms that depict probabilities by displaying a box that contains 100 items, some of which are filled. These formats will be compared and discussed among the study team, and the final presentation format selected by the study team will be used to display evidence on objectives during AHP sessions. See Supplementary Figure 1 for the different visual representations to be considered.

Step three: making comparisons among the alternatives

Comparisons among alternative drugs relative to criteria: We will compare the alternative drugs’ ability to achieve the decision goal (i.e. best treatment for type 2 diabetes) by making comparisons among the alternative drugs with regard to fulfilling each criterion. This will be conducted using standard AHP pairwise comparisons among the alternatives for each of the benefit and risk criteria defined in the previous step using the 9-point ratio scale. This “bottom-up approach” will be used to account for the potential consideration when one key assumption for AHP – that the higher level of elements in the hierarchy are independent of lower level elements – may not hold.

Comparisons among the criteria: The same pairwise comparison method will be used to determine the priorities of each of the criteria relative to the decision goal (i.e. safe and effective medication for type 2 diabetes).

Step four: combining judgments’ to see how well alternatives can meet the goal

We will use the standard AHP weighting to combine the results of the judgments made in step three to determine the relative abilities of the medications to meet our stated goals for the decision context. Relative differences > 1.1 will be considered significant.

Step five: sensitivity analysis

We will explore the impact of different judgments’ on the relative importance of the criteria varying their priorities from 0 (no importance) to 1 (most important) and recalculating their alternative scores. We will invite input on the relative importance of the criteria from various stakeholders and conduct additional sensitivity analysis to determine their impact on the decision goals.

Delivery of the AHP web based instrument. The AHP instrument will be delivered as a series of questions in an online version of Expert Choice. The first screen, or “welcome screen”, will present a comprehensive description of the decision context, instructions as to how pair-wise comparisons should be judged using a ratio scale, and instructions for navigating through the experiment. Each subsequent screen will contain a set of pair-wise questions designed to elicit each respondent’s opinion on the relative weight of each objective or alternative in terms of meeting the overall goal or relevant objective.

Alternatives will be presented first, then specific objectives, moving from the bottom of the hierarchy up to the decision goal. A bottom-up order of presentation was considered more appropriate than a top-down order because expert respondents will have underlying knowledge about details that could influence decisions at higher levels in the hierarchy in an uncontrolled manner. For pair-wise comparisons of objectives and sub-objectives, participants will see the text: “Which of the two objectives below is more important?” For pair-wise comparisons of alternatives, participants will see the text: “Which of the two alternatives below is more preferable?” Rating of alternatives is necessary in order to transfer the evidence into subjective comparisons of importance on the ratio scale. The Expert Choice software translates all judgments to a ratio scale, regardless of whether they are entered using a verbal, graphical, or numerical scale.

The maximum number of pair-wise comparisons will be presented in order to give the best possible accuracy. Participants will be able to click on information icons to display evidence-based information (described above in “Presentation of objectives”) on how each alternative meets the relevant objective. The instructions on the welcome screen and questions will be drafted according to survey methodology best practice and will be tested and redrafted as necessary by the study team. Participants will be able to see their individual intermediate results throughout the process and their overall results when the process is completed. The option for participants to see their inconsistency ratios will be turned off because presentation of the inconsistency ratio may encourage participants to aim for consistency over accuracy. At the end of the process, participants will see a “thank you” screen which will provide a brief overview of the analysis process and group session as well as contact details for the principal investigator and ethics committee. Analysis will be conducted using the ideal mode, and various sensitivity analyses will be conducted for the group session.

Conduct of the AHP sessions

Participants will be invited to participate in one individual session and one group session. Trained interviewers who are co-investigators on the project will schedule one-on-one appointments with respondents in which they will complete the instrument and probe questions. Interviews will take approximately 90 minutes. Once all pretests are completed, individual and group results will be analyzed and presented to respondents in a group debrief session lasting approximately 60 minutes. This will provide respondents with further opportunity to comment on the face validity of the approach and to raise any additional concerns about the instrument.

The individual interviews will involve an experienced investigator working one-on-one with each respondent to work through the AHP task and conduct the cognitive debriefing concurrently. Interviews will be scheduled at quiet locations and times convenient to the respondent and will take approximately 90 minutes to complete. Respondents will not be remunerated for their participation. Respondents will not be asked to provide any personal health information, which helps minimize the risk of breaches of confidentiality.

At the end, participants will be invited back for a group session. The objectives of the group session are: 1) to present the group results from the AHP and any significant findings from the cognitive debriefing; 2) to assess the extent to which respondents agree with the experiment’s findings; and 3) to assess whether they find the method sufficiently trustworthy to consider using it in other decision making contexts.

The group session will be held in a secure location. It will take approximately two hours, and respondents will not receive remuneration. The agenda for the group session will include presentation of overall results followed by results at each level of the hierarchy. For each trade-off in the hierarchy, presentation of results will be accompanied by discussion of any outlying results, with respondents encouraged to discuss reasons for differing views. Where there is significant heterogeneity in responses, as measured by the standard deviation of the priority weights, or where respondents do not agree with results, investigators will show the impact on results of alterations to responses and/or the evidence matrix. The session will conclude with a short poll on the extent to which respondents agree with the findings and trust the method.

Cognitive interview and evaluation

Cognitive interviewing will be conducted alongside the individual interviews. The cognitive debriefing protocol will be developed based on standard methodology building on methods used for cognitive debriefing of conjoint analysis instruments. The protocol will involve prospective and retrospective probing, as well as assessing information volunteered by participants to assess the AHP instrument according to the following checklist shown in Table 1. Interviews will be recorded and transcribed, if agreed to by respondents, to facilitate analysis of the cognitive interviewing information. Analysis of the cognitive interview will involve assessing the extent to which the scale performed adequately on each of the checklist items. Major themes or concerns about the instrument will be elicited from the transcripts and presented through representative quotes.

Study investigators

The study team will comprise of co-investigators with methodologic expertise in preference assessment methods, decision analysis, evaluation of pharmacotherapeutic published and unpublished literature, epidemiology, and clinical care of type 2 diabetes. The study team will meet for at least 45 minutes each week to discuss and refine the decision context and model development. One member of the team will take primary responsibility for development of the decision context and model based on study team discussions and feedback during validation sessions.

Protection of human subjects

The protocol was approved by The Johns Hopkins Medicine Institutional Review Board. Risks to human subjects are minimal because the tasks involve opinion research, with no medical interventions or collection of personal health information. Participants will be provided with information about the study at the beginning of the individual sessions and asked for their consent to participate. They will be informed that they do not have to answer all questions and that they can withdraw from the study at any time without penalty.