A discrete choice experiment to elicit preferences for a chronic disease screening programme in Queensland, Australia: designing the choice sets for the final survey

Background

According to the Australian Institute of Health and Welfare, chronic diseases such as heart disease, diabetes, cancer, and respiratory disease are the leading cause of illness, disability, and death in Australia, and their burden is projected to rise due to factors such as population aging, changing lifestyles, and environmental factors.¹ Community screening programs for chronic diseases need to be implemented to address this challenge. However, to ensure the success and sustainability of such programs, eliciting population preferences for the types of health screening services offered is crucial. Understanding the community’s preferences is critical in designing community screening programs that are both effective and well-received by the target population.² By doing so, these programs can be tailored to the specific needs and preferences of the community, increasing their acceptability, and improving their effectiveness in identifying and managing chronic diseases.

Discrete choice experiments (DCEs) are gaining popularity in finding otherwise unavailable answers for problems in healthcare. The insights provided by DCEs into patients’ and healthcare workers’ preferences can help make decisions in health services and allocate scarce resources. DCEs being a stated preference method, facilitate decisions holding multiple trade-offs simultaneously and allow consideration of the relative importance of various attributes (and their levels) when making that decision.³ This method allows insights into subtleties of decision-making within stakeholder groups, making it possible to explore the trade-offs between alternatives that sometimes are hypothetical.⁴ For example, a DCE could simultaneously examine provision of care in hospital or a community setting, care provided by a nurse or specialist, acceptable waiting times, and acceptable out-of-pocket costs.⁴

The theoretical basis of the DCEs originates from the random utility theory (RUT), which describes choices made by individuals using discrete sets of alternatives.³ A utility function can be used to describe the preference for an alternative by an individual. It is assumed that the alternative with the highest utility (most preferred) will be chosen. This utility depends on the attributes of the alternative (e.g., wait time, travel distance) and the individual making the choice (e.g., age, sex) and the unobserved attributes the data collector may not be aware of (e.g., co-morbidities of the individual making a choice). The observed (collect data on) and unobserved attributes are represented in the utility function by explanatory and random variables, respectively.³

The attributes and their levels are used for the experimental design, which develops hypothetical choice sets to be compared by the respondents.⁵ The respondents attach a utility to alternatives based on their preference for attribute-level combinations. The choices made by the respondents are considered mutually exclusive (can choose only one option) and collectively exhaustive (all the options are available for decision-making). As such, all the information must be provided within the attributes and levels for valid choices.⁵ The presentation of all relevant information within a choice set reduces the random effects of the models in question.⁶ To reduce the randomness of the developed model, it is of upmost importance to use attributes of the greatest relevance to those completing the DCE.

Some DCEs used in healthcare policy settings have failed to report rigorous methods employed to develop appropriate attributes for the alternatives.⁷ In health policy settings, the attributes are the characteristics of the intervention (e.g., community screening programme for chronic diseases), and each attribute (e.g., place of screening, waiting time) is designated with levels (e.g. hospital, community clinic, one week, one month). Attribute selection to represent the characteristics of the intervention need to follow a rigorous methodology to ensure all vital information is presented for the choice experiment.⁸ Attribute selection ideally uses mixed methods, including qualitative methods such as focus groups, interviews, quantitative prioritisation, and final determination using expert panels. The participants in this data collection should include relevant stakeholders, including patients, care providers and decision-makers to ensure the representation of all views.^8,9

An important part of the design process is where hypothetical alternatives are generated and combined to develop choice sets. By manipulating the design, the investigator can reduce the response burden, increase statistical efficiency, and construct a simpler DCE. A full factorial design containing all possible combinations of attributes may not be feasible due to too many resulting alternatives and choice sets. Therefore, a fractional factorial is often used.³ This design should be orthogonal and balanced. In orthogonal designs, attributes are statistically independent of each other so that the participants’ preferences can be estimated for each attribute. Balanced designs have each attribute occurring equally within choice sets. Choice designs that are both orthogonal and balanced are called orthogonal arrays and are not universally available for all combinations as they are not feasible for alternatives with five or more attributes with two or more levels. When orthogonal arrays are not feasible, designs need to find efficiency by trading off orthogonality and balance.³ Compared to orthogonal designs, efficient designs increase the precision of parameter estimates by reducing coefficient standard errors and allowing some limited correlation between attributes. Most designs use D-efficient designs (D-error (inverse)/D-efficiency/D-optimal) to achieve this efficiency and are recommended to maximise statistical efficiency and minimise the variability of parameter estimates.^3,5 D-efficiency ranges from 0% to 100% where 100% denotes the greatest statistical efficiency. Prior information about the parameters in the model is required for efficient designs.¹⁰ Since these priors may not always be available for the design algorithm, investigators should look for subsequent D-efficient designs after pilot surveys. The efficient design can use fixed point estimates (prior design) or probability distribution (Bayesian).¹⁰ It is preferable to pilot at least with very small priors with hypothetical directions for some parameters and estimate the models to learn the direction and magnitude of priors for the next D-efficient design.

Sample size calculations for DCE studies are evolving. However, it is an important part of the research as a appropriate sample size assures sufficient statistical power to detect a difference in preferences. Investigators tend to maximise the sample size to avoid underpowering DCEs.¹¹ However, given limited budgets, online data collections and the notorious non-responsiveness of patients and clinicians within healthcare DCEs, it is not practical to overpower the sample in most cases. There are heuristic and parametric approaches to estimating sample sizes in DCEs. The disadvantages of these methods have been discussed elsewhere, and methods to estimate sample size have been proposed based on significance level, statistical power, model, priors and design.¹¹

This paper presents methods for determining preferences for community screening programmes for chronic disease in Australia. We previously presented the attribute development for the choice sets. Here, we present the designing, pre-tests, pilots, and use of priors to organise the final design and the data collection for the DCE.

Methods

This project was designed to elicit community preferences for health screening services for individuals with chronic diseases such as diabetes, cardiovascular and liver disease, using a discrete choice experiment. This paper describes the designing of the D-efficient design for the pilot survey (Step 2), the results of the pre-test (Step 3) and the pilot survey (Step 4) and designing of the final D-efficient design for the main DCE survey (Step 5) ( Figure 1). The final DCE design was developed after a pre-test and a pilot survey, and the methods and results are presented in later sections. The pilot survey (Step 4) was used to estimate the priors to develop the D-efficient design for the main DCE survey (Step 5). In developing this study and presenting the findings, we adhered to A Reporting Checklist for Discrete Choice Experiments in Health: The DIRECT Checklist, ensuring transparency and methodological rigour throughout the study process.¹²

Figure 1. Steps for designing the final DCE choice tasks.

Ethics approval for this study was granted by the Queensland University of Technology Human Research Ethics Committee (QUT HREC), with reference number HREC/QUT/4282, on 02/08/2021. The study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki, ensuring the protection of participants’ rights, safety, and well-being throughout the research process.

Step 2: Designing DCE choice tasks – pilot survey

The final attribute list had five attributes with three levels each, and that would result in 59,049 (3¹⁰) possible choice tasks. An example choice task is given in Figure 2. Since it is not feasible to present all possible combinations (n=59,049) to all the respondents, 30 choice tasks were selected using Ngene version 1.3.0 software, in a fractional factorial design. The 30 choice tasks make up the choice set. The main aim of using a fractional factorial design was to have a manageable number of choice tasks while maximising the design’s statistical efficiency.¹³ Therefore, a multinomial logit model based on D-efficient fractional factorial design criteria (using the D-error value) was used to develop 30 pair-wise choice tasks using the design software Ngene. Evidence indicates that respondents can efficiently handle ten choice sets at a time.^8,14 Therefore, the fractional factorial design was divided into three blocks so that a respondent would only answer ten from the 30 choice tasks in the fractional factorial design. Blocking is an accepted statistical technique in a DCE design that ensures an equal number of respondents per block.⁵ We used the modified Federov algorithm to develop the D-efficient design, which is known to develop designs with attribute level balance and no dominant choice tasks.¹⁵

Figure 2. Example of a choice set seen by respondents.

In the absence of prior information on the coefficients of the different attributes, small positive or negative priors or zero priors (non-informative priors) were used to design the D-efficient design based on the following a priori hypotheses (Table 2).

• • People have equal preference for a nurse-led, general practitioner-led, and specialist-led screening programme. Therefore, we used non-informative (zero) priors.

• • People have a preference for quality and accurate screening tests.

• • People have a negative preference for out-of-pocket cost and wait-time.

• • People do not have a strong preference for different sources of information. Therefore, we used non-informative (zero) priors.

Table 2. Estimates used to design the choice tasks for the pilot and the final survey.

Attribute	Priors used in the Ngene design
	DCE design used for the pilot survey	DCE design used for the final survey
Screening conduct
Nurse at local community health clinic	0	Reference
General practinioner at your usual GP clinic	0	0.89071
Specialist in hospital outpatient clinic	Reference	0.44991
Quality and accuracy of the test results
75% accurate	0.000001 (Continuous scale)	Reference
85% accurate		0.97828
95% accurate		1.02893
Cost to the patient (per $100) – levels $0, $80, $250	-0.000001	-0.01173
Wait time (per week) – Levels 2, 8, 24 weeks	-0.000001	-0.50724
Source of information
Well trusted source	0	Reference
Moderate amount of trust	0	0
Limited familiarity or trust	Reference	-1.10438
Measures of efficiency
D-error	0.0083	0.0585
S estimate		7.434

Using moderately informative priors for two of the attributes means the 30 choices lean towards those with larger differences in the other attributes, so more information is gathered on these attributes.

Apart from the D-error, attribute level overlap and attribute level balance were used to assess the DCE design. Attribute level overlap is when the same level is present in both choice tasks, essentially eliminating this attribute from that choice. Attribute level balance is the distribution of the attribute levels across the two choice tasks. Lower attribute level overlap and equal distribution of levels indicate a better DCE design.

Results

Discussion

The main aim of this project was to design a DCE choice set that could elicit community preferences for health screening services for individuals with chronic diseases such as diabetes, cardiovascular and liver disease. The final set of attributes and levels for the DCE was based on a systematic review of the literature,⁷ qualitative interviews, a quantitative structured prioritisation exercise and an expert panel discussion.⁹ We followed a robust methodology to develop an efficient choice set that captures maximum information. The final choice set had 30 pair-wise choice tasks divided into three blocks, with minimum attribute level overlap and satisfactory attribute level balance.

Our study used the D-efficient criterion to design a fractional factorial design with 30 pair-wise choice tasks. The D-efficient criterion is probably the most common efficiency criterion in designing DCE choice tasks.²⁵ The number of pair-wise choice tasks (rows) in the design depends on the number of parameters in the utility specification. The minimum number of pair-wise choice tasks (rows) of the DCE design equals to or greater than the number of parameters, not including constants, plus one.²⁶ Our study had eight parameters, indicating that the minimum number of choice tasks would be nine. However, the number of choice tasks is often set to at least two or three times the minimum size to have sufficient degrees of freedom. Therefore, the use of 30 pair-wise choice tasks in the current DCE design would provide enough variation in the design matrix to estimate reliable parameter coefficients in the final DCE survey.

Our study used a heterogenous design, meaning each respondent responded only to a subset of the choice tasks. The choice set was divided into three blocks so that each respondent answered only ten choice tasks to reduce the burden on participants of answering all the choice tasks. Heterogeneous designs are generally considered better as they provide more information than homogenous designs.²⁷ The number of choice tasks each respondent receives depends on the complexity of each choice task and how many the analysts believe a respondent can handle without fatigue. Mixed evidence exists as to the impact the number of choice tasks has empirically upon choice experiments. Hensher et al. suggested using 4 to 16 choice tasks²⁸; however, few studies indicate that the number of choice tasks each respondent sees has the least influence on the error variance of choice data.^29,30 Our DCE design had only five attributes, and an expert panel validated the attributes and the levels not to be mentally demanding when put into a choice set. Furthermore, based on the completion rate, the pre-test indicated that a respondent could handle ten choice tasks without any fatigue.

Efficient designs have the potential to select a subset of choice tasks from the full factorial design that yields more information, estimate smaller standard errors and increase the reliability of the parameter estimates.²⁶ However, it is important to note that the efficiency of the design depends on the prior parameter estimates used in the model. If the priors are incorrect or close to actual behaviours, the design can become inefficient, leading to larger standard errors.³¹ Since no prior estimates were available in the literature, we conducted a pilot study to estimate the priors. This step has been recommended and could significantly improve the quality of the information in the final DCE survey through smart choice tasks with appropriate trade-offs across the attributes.^22,32 This means that the final DCE survey designed in this study can potentially estimate reliable parameter estimates at smaller sample sizes. However, several systematic reviews which have reviewed DCE studies report that most studies either fail to report the source of the priors or use non-informative (zero) or conservative (close to zero) priors for the DCE design.^7,33 This is a critical drawback as this limits the ability for critical appraisal and reproducibility of the survey. Furthermore, this leads to inefficient DCE designs that may require larger sample sizes to collect the same amount of information compared to a more efficient design.

The two DCE designs (for the pilot and the main study) developed in the study used constraints at the design stage to achieve attribute level balance, and the results indicate that the two designs achieved a satisfactory level of attribute balance. Imposing attribute balance constraints could have reduced the efficiency of the DCE design.²⁶ However, some degree of attribute level balance in the design ensured that all parameter levels were represented. This would ensure that the parameter coefficients in the main DCE survey could be estimated well on the whole range of levels instead of having data points at only one or a few attribute levels.

Conclusion

We followed a robust methodology to design a DCE choice set that could elicit community preferences for health screening services. The final DCE design had 30 pair-wise choice tasks and demonstrated satisfactory efficiency that will capture maximum information and best inform policy.

Declarations

Consent for publication

Not applicable.

Authors’ contributions

SK, SS, AB, and DB contributed to the design of the study, coordinated the collection of data, analysed the data, and drafted the manuscript. MA, EEP, JO’B, PV, and IH contributed to the development of the data analysis plan, interpretation of the results, and review of the manuscript. All authors have read and approved the final version of the manuscript.