Precision medicine implementation and research-practice partnerships: implications of measurement scale differential item functioning

Study design

This was a cross-sectional study that employed analytical approaches for quantifying factorial relationships. DIF analysis helped to reveal differences in perceptions towards PM implementation between two study participant subgroups: one affiliated to academia and the other affiliated to industry. Item response theory (IRT) is one of the widely used methods for assessing DIF. In this study, IRT allowed for a critical examination of the information that was harvested from the manner study participants endorsed and ranked the various items associated with PM implementation. As noted earlier, DIF examines the probability of endorsing an item conditioned on the latent trait. This method was deemed suitable for examining DIF in this study as IRT examines the monotonic relationship between responses and the latent trait¹⁷. Furthermore, as compared to classical test theory, IRT parameter estimates are not only as confounded by sample characteristics, but statistical properties of items can be expressed with greater precision, thereby increasing the interpretation accuracy of DIF between sub-groups¹⁸.

Members of “academia” were defined as individuals involved in biomedical research that related to translating newly discovered molecular or omics-based biomarkers (OBMs) for purposes of clinical or population health use. Members of “industry” were defined as individuals involved in the clinical use of the biomarkers or those involved in commercialization activities related to OBMs (e.g., Direct-To-Consumers Genetic Testing, DTC-GT)¹⁹. “Precision medicine implementation” was defined as the process of translating newly discovered omics-based biomarkers (OBMs) for clinical or population health use. “OBMs” were defined as candidate genetic biomarkers that are in the process of being clinically validated or those already validated and in clinical use.

Study participants

The snowball sampling method, a non-probability sampling method, was applied in identifying potential study participants from population of interest. Inclusion was based on a participant’s affiliation to either an academic institutions or a commercial entity (industry); the institution or organization had to be involved in molecular/genetic testing and/or omics-based biomarkers in Africa; a participant had to be involved in the field of precision medicine and/or biomarker-related biomedical research as indicated by contribution to the field through publications in peer-reviewed journals and/or attendance of related academic conferences. It is well known that snowball sampling poses significant recruitment “community bias” risk whereby recruitment of new members does not branch out from the sample subgroup of individuals initially accessed. In the present study, this risk of source bias was addressed by properly selecting the initial individuals to ensure proportional inclusivity of subgroups in the seed (initial) sample. Consequently, the seed sample was identified and selected from a precision medicine-related conference; an event we regarded as representative of our study population as it brought together diverse groups from academia and industry involved in precision medicine and its implementation in resource-constrained settings. Therefore, the probability of selection bias for subsequent in-link recruitment waves of study participants was negligible. Besides, it has been shown that with an appropriate bootstrap resampling, selection bias due to the initial seed sample (if known to be representative of the targeted population) is progressively attenuated as the sample expands wave by wave^20,21.

Guided by general principles of the Nuremberg Code, the Declaration of Helsinki and institutional review board permit obtained from the University of KwaZulu-Natal (BREC Permit Ref No BE513/18), a study package was distributed via email to potential participants between June and July 2019. Since this study contained negligible risk of potential embarrassment or other ethical dilemmas that are usually associated with snowball sampling in many other studies, initial participants were encouraged to forward the email containing the study package to their colleagues. The study package (available as Extended data)²² included an invitation letter with study description, consent form and a link to the online card-sort platform. A total of 31 study participants were recruited for the study. Although there is no definitive number for a pilot study sample size according to Hunt, Sparkman and Wilcox²³, to increase statistical power and degrees of freedom for our study, we applied a Monte Carlo simulation (MCS) approach as described by Mooney²⁴. MCS studies are computer-driven experimental investigations in which certain parameters, such as population means and standard deviations that are known a priori, are used to generate random (but plausible) sample data²⁴. This method of generating and analyzing data treats the collected research sample as a “population reservoir” from which a large number of random samples are drawn with continuous replacement such that the probability of selection for any given case remains equal over every random draw²⁵. We requested 2,000 iterations, drawn with replacement from the original data set of 31 cases (our empirical sample).

Procedure

A card-sort task approach²⁶ was used to obtain comparative item-rating data. Respondents were presented with a set of cards that contained constructs clearly defined using everyday language and a set of categories onto which the concepts on the cards were to be mapped, as explained by Angleitner, John, & Lohr²⁷. The respondents were asked to read each item and assign it to the construct or concept it best indicated, in their judgment. Participants were required to rank the cards according to their own understanding and preference, indicating their perception on concepts presented on each card. If participants deemed it necessary to make any changes as to where a card previously assigned should be reassigned to, they could do so. Although simple in practice, this exercise generated enough data that was used to assess the desired group perception levels, construct validity and scale reliability. The actual sorting of the items was, however, performed using an online platform hosted and supported by Optimal Workshop²⁸. The items are presented in Table 1. Besides offering convenience to study participants, the online platform enhanced data security and confidentiality. A total of 22 items were to be assigned into four categories initially provided for, with participants allowed to create own separate construct categories if they so desired. The results about number of items per category and the items’ ranking order (positioning) within their respective categories were then extracted and analyzed. Participants’ demographic data were also extracted, including sex, highest educational qualifications and age. Data was subsequently subjected to item response theory (IRT) analysis^29,30 with logistic regression modelling.

The data

In summary, the data are an output of an exercise where respondents were required to place items arranged as cards into specific categories and ranked in order of perceived importance. The cards related to specific attributes indicative of wider factors thought to influence precision medicine implementation at health systems level. This meant that if an item on the card was regarded as the most important in each category, it was ranked 1^st, and the next ranked 2^nd and so on. If a card was not placed in a category, it was scored zero in that same category. For compactness and statistical purposes, the data was formatted to have one observation per row (for 31 participants, there were 31*4 = 124 observations).

Statistical considerations

One of the assumptions in item response theory (IRT) model analysis is that factors are generally unidimensional³¹. Therefore, to examine differences in perception between the two sub-groups (academic and industry) related to each item in the PM implementation scale, DIF was tested within each subscale. Each of the four factors that were identified as influencing PM implementation correspond to a subscale. Thus, four sets of tests were conducted with each of the four factors (sub-scales) examined separately for both item response functions (IRFs) and unidimensionality. To determine and confirm sub-scale unidimensionality, we used R package “mokken” version 2.8.11³². In this study, scale reliability referred to the consistency of measurement, while construct validity refers to the extent to which a set of indicators that were devised to gauge the four constructs (OBM, OrG, PGA and ImO) really measured that construct. Differential item functioning (DIF) was defined as a property of an item that shows the extent to which the item might be measuring different dimensions of the variable it was supposed to measure for members of separate subgroups. We applied the ordinal logistic regression modelling function in the package “lordif” (logistic ordinal regression differential item function) using IRT, version 0.3-3)³³ in R version 3.6.1³⁴. The package “lordif” provides a logistic regression and IRT framework for detecting various types of differential item functioning³³. We preferred it because it could adequately handle the item-sort-and-rank data that we used. To analyze DIF in the scale, we devised three models. The first of the three models distinguish DIF and non-DIF items. The second model identifies uniform-DIF items that occur due to an item tapping a dimension of the attribute measured in the scale that manifests differently between the groups. That is, for uniform DIF, whereas the probability of endorsing an item at a certain ranking is greater for one group than the other, this is uniform over all levels of knowledge of the factors under consideration. Finally, the third model identifies non-uniform DIF; that is the probability of endorsing an item at a certain ranking is greater for one group than the other, but this is not uniform over all levels of knowledge of the factors under consideration. This may be because of distracting elements (not necessarily related to the scale) in the measurement process, such as different understandings of a word or phrase used on the item³⁵.

We used two sets of logistic regression-based criteria for detecting DIF: one set based on statistical significance (likelihood ratio χ² tests) and the other on Pseudo R² magnitude (Nagelkerke's and McFadden’s pseudo R²). Uniform DIF was examined by comparing the log likelihood of model 1 with 2 (degree of freedom (df)=1) and non-uniform DIF by comparing model 2 with 3 (df=1).The comparison between model 1 and model 3 (df=2) detected the total DIF effect—both uniform and non-uniform DIF^36,37. Items that revealed significance in any of the three likelihood ratio χ² tests with an alpha of 0.25 (Model 1 versus 2, Model 2 versus 3, and Model 1 versus 3) were flagged as having DIF. To examine the magnitude of DIF, we used various pseudo R^238,39 such as Nagelkerke's Pseudo R² and McFadden’s pseudo R² statistics for the defined model comparisons, and applied Zumbo et al.’s guideline to evaluate R²⁴⁰: effect size was regarded as negligible DIF (pseudo R² < 0.035), moderate DIF (0.035 ≥ pseudo R² < 0.070), and large DIF (pseudo-R² ≥ 0.070).

For individual participants, the difference between the DIF-adjusted (purified) subscale score and the initial unadjusted score for each subscale. Finally, we obtained item and subscale characteristic curves to show the impact of DIF for each group, i.e. item parameter estimates were computed and graphically examined via item characteristic curves (ICCs) and other graphs. R code used to calculate DIF is available as Extended data⁴¹.

Descriptive statistics that were obtained for the data included analysis of outliers, which was defined as any value that is greater than three standard deviations above or below the mean. Multivariate skewness and kurtosis statistics of the data were assessed using R package ‘psych’ version 1.8.12⁴².

Demographic characteristics of participants

Of the 35 participants invited and who attempted the item-sort exercise, 31 successfully completed it (88% response rate). There were three more female participants than males (10%). Those with research qualifications (PhDs and equivalent) were the majority (55%), while slightly a quarter of the participants were medical doctor. The rest of participants’ qualities varied greatly and are summarized in Figure 2.

Figure 2. Demographic characteristics of study participants.

Variable correlation

The correlogram in Figure 3 is an inter-item correlation plot indicating item and scale structure in the data. In the upper triangle, positive correlations are displayed in blue and negative correlations in red colored circles. Color intensity and the size of the circle are proportional to the correlation coefficients, helping to identify “groups” of variables that share a strong relationship with each other (hierarchical clustering). The lower triangular correlation matrix displays the actual correlation values.

Figure 3. Variable correlation matrix.

Each item seems to fit into one of the four well defined clusters (Figure 3), implying four factors defined the data as hypothesized. Items x16 (“Using this genetic/omics test has been regarded by practitioners as an appropriate mechanism for patient management (e.g. aid in drug dosage decisions, in carrying fetuses to term or carry out prophylactic surgery)”; and x20 (“Practitioners are more willing to order the genetic/omics test more often whenever deemed necessary”) are the only items not showing a strong correlation within the groups they belong.

To confirm the suggested four factors in the variable correlation matrix, we performed further scale analysis using functions in the “mokken”⁴³ R package. Four unidimensional scales were identified as dimensionality solutions via the package’s automated item selection algorithm (AISP) performed at factor loading ≥0.3 and Type I error level (α)=0.05. Table 2 shows the results of this procedure and various reliability associated with each factor (subscale).

Sub-scales were demarcated by means of scalability coefficients⁴⁴ as indicated by the coefficient H in Table 2. At the lowest scalability threshold of 0.3, four unidimensional subscales were identified through the AISP. AISP is a scaling method of item selection we applied to partition the 22 items into subscales. This implies that there were four latent factors (subscales) measured by the 22 items (i.e., four underlying variables explain the association between latent factors and responses to items). Unidimensionality ranges between H=0 and H=1: strong unidimensional scale has H ≥ 0.5, moderate 0.4 ≤ H < 0.5, and weak 0.3 ≤ H < 0.4⁴⁴. The four subscales’ coefficients H ranged between H ≥ 0.5 and H = 0.8, implying the subscales were strongly unidimensional. This was strongly supported by the reliability indices (Mokken’s rho, alpha and Guttman’s lambda-2 (λ2)) which indicated high internal consistency (internal validity/goodness-of-fit) for the four subscales. Cronbach's α, the expected correlation of two tests that measure the same construct, was separately obtained for the four constructs. All four subscales had coefficient alpha (α) ranging between 0.76 and 0.95 (with an average α =0.88). Besides, given that internal scale consistency (intercorrelations among items) is maximized when all items in a subscale measure the same construct, we concluded that the high Cronbach's alpha values for our subscales are an indirect but confirmatory indicator of the degree to which the items in the subscales measure corresponding unidimensional latent construct. However, Cronbach's alpha has been found to have limitations in measuring reliability in some instances^45–47. Due to this concern, we obtained other reliability indices for the same purpose as described below.

Guttman’s λ2 is a reliability estimate that is more robust than Cronbach’s alpha, i.e., more resistant to outliers. The range of λ2 values for the four subscales were: 0.81 ≤ λ2 ≥ 0.96. This implies that more than 80% of variance in each of the subscales is attributable to the true score and less than 20% to error, confirming that the internal construct validity and scale reliability is good. On the other hand, Mokken’s rho, considered to be an improvement in relation to Cronbach’s alpha and whose values should exceed 0.70 to show adequate scale reliability⁴⁸, was obtained. As shown in Table 2, all the four subscales had their rho (MS) coefficients above 0.78, confirming good scale reliability.

Table 2 also shows DIF detection outcomes using logistic regression approach³³. DIF items revealed from the likelihood ratio χ2 tests are indicated with asterisks in Table 2. Of the 22 items making up the PM implementation (PMI) scale, 13 were flagged as DIF items at α=0.25 with MCS set at 2,000. They all showed statistical significance for total DIF effect test when comparing Models 1 and 3. Of the 13 DIF items however, seven were indicated as mixed DIF items. Their likelihood ratio χ2 test comparisons between Model 1 and 2, and models 2 and 3 were statistically significant. Moreover, of the remaining six, five had non-uniform DIF (x1, x4, x9, x16 and x20); the comparisons were statistically significant for Models 2 vs 3 as compared to Models 1 vs 2. Only one item had uniform DIF (x19): it showed non-significant comparisons for models 2 vs 3 but significant comparisons for models 1 vs 2. However, using the same statistical significance test, all items were DIF flagged at α =0.5 and non at α =0.01. This finding indicates the effect of statistical significance levels in DIF detection.

However, in contrast to the likelihood ratio χ2 tests that detected 13 of the 22 items as DIF-positive, Nagelkerke's Pseudo R2 statistics flagged only five items as DIF (x1, x4, x7, x10 and x12). These were all moderate DIF (0.035 ≥ pseudo R2 < 0.070). To verify this result, we inspected other pseudo R2 statistics, such as McFadden’s and Cox and Snell’s R2. The additional pseudo R2 indices also yielded similar values.

Graphical Analyses of DIF Effects

From the above analyses, we generated various graphs. However, due to space constraints, we present only those graphs related to characteristics of omics biomarkers (OBM construct). Diagnostic plots are used below to describe the relationship between the perception latent trait and the probability of responding positively to an item as observed from the data (positively here means endorsing an item in any of the four scales). Participant’s perception trait level is signified by theta (θ).

The top left plot in Figure 4a shows item true score functions based on group-specific item parameter estimates. It shows the item characteristic curves (ICCs) for the item for industry vs academia affiliated participants represented with red dashed and black solid lines respectively. The slope of the function for industry group was substantially higher than that for academia group, indicating non-uniform DIF. The upper–right graph shows the absolute difference between the ICCs for the two groups, indicating that the difference is mainly at high levels of perception of omics biomarkers (especially between average (0) ≥ θ ≤ +3). The lower-left graph shows the item response functions for the two groups based on the affiliation–specific item parameter estimates (slope and category threshold values by group, as also printed on the graph). It juxtaposes the item response functions for academia and industry groups. The non-uniform component of DIF revealed by the LR χ2 test can also be observed in the difference of the slope parameter estimates (33.46 vs. 2.75). The lower-right graph shows the absolute difference between the ICCs (the upper-right graph) weighted by the score distribution (size) for the focal group, i.e., industry group, indicating minimal impact.

Figure 4. Graphical Analyses of DIF Effects.

(a) Graphical display of the item “X1:The omics test has been used among people similar to the present target population” which shows non–uniform DIF with respect to affiliation to either academia or industry. (b) Box-and-whisker plot and a scatter plot showing (individual-level) differences between initial and purified scale scores for each respondent. (c) Impact of DIF items on test characteristic curves (TCC) for all items and DIF items. (y-axis = sum of the item scores of the OBM domain; x axis (theta) = knowledge of OBMs). (d) Smoothed histogram showing perception differences towards omics biomarkers between those affiliated to academia and industry.

Figure 4b displays the boxplot and a scatter plot that show DIF items as unique to each group, accounting for DIF in trait estimates. It shows the difference in latent trait level between the initial IRT-based trait estimate with ignored DIF items and the purified trait estimate with DIF items accounted for. Each (black) circle and (red) triangle stand for the latent trait estimate of the OBM domain for a single participant. It also shows the expected amount of change in latent trait estimates when DIF is accounted for. Accounting for DIF items in a multi-item scale is commonly referred to as “scale purification”. In both graphs, the y-axis is the difference (initial – purified) and the x-axis on the right graph is the initial latent trait level. The Box-and-Whisker plot on the left shows that median difference was around -0.04, and the middle 50% of respondents, depicted as the box (i.e., the interquartile range)ranged from approximately -0.04 to 0.07, indicating a fairly normal but slight right-skewed distribution with possible outliers at high ends (θ above average). The scatter plot on the right shows the initial (no DIF) vs purified (DIF accounted for) difference against the initial latent trait level that ignored DIF, implying that across the entire latent trait continuum, those participants who were affiliated to industry (red triangles) show more positive difference, further suggesting that accounting for DIF leads to lower scores for them compared to initial (non DIF) scores; for those affiliated to academia, (dark circles), the pattern is in the opposite. Guidelines are placed at 0.0 (solid line), i.e. no difference, and the dotted horizontal reference line is drawn at the mean difference between the initial and purified estimates.

What is suggested in Figure 4b corresponds well with Figure 4c, the test characteristic curves (TCC). The y-axis of Figure 4c indicates all possible scores for domain “OBM”. Figure 4c indicates a clear difference in TCC curves, albeit in small margins, of DIF item responses between academia and industry groups. On the other hand, the left graph for all items displayed a much smaller difference than the right graph of Figure 4c. This can be explained as the DIF impact being diluted by the non-DIF items, resulting in the overall score difference between the two groups becoming minimal. It is evident that at average knowledge levels (θ = 0), the scale performs better among those affiliated to industry than academia. But at higher trait levels (θ ≤ +2), industry lags behind academia.

Finally, Figure 4d shows smoothed histograms of the omics biomarkers perception (knowledge) levels for those study participants affiliated to industry (dashed red line) and academia (solid black line) as measured by the OBM subscale (theta). Industry affiliated participants on average had lower mean scores than their academia-affiliated counterparts, although there is a broad overlap in the distributions.

Underlying data

Zenodo: Extended data for Manuscript: Precision medicine implementation and research-practice partnerships: implications of measurement scale differential item functioning (DIF). http://doi.org/10.5281/zenodo.3925276²².

File ‘DIF Analysis Raw.Extended Data’ (XLSX) contains the redacted survey responses from the card sort exercise and the results of the card sort exercise itself.

Extended data

Zenodo: Extended data for Manuscript: Precision medicine implementation and research-practice partnerships: implications of measurement scale differential item functioning (DIF). http://doi.org/10.5281/zenodo.3925276²².

File ‘DIF Study Package.Extended data’ (DOCX) contains the study package provided to participants.

Zenodo: R code for DIF. http://doi.org/10.5281/zenodo.3925336⁴¹.

This project contains the R code for this study in R and PDF formats.

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).