Methodological framework to identify possible adverse drug reactions using population-based administrative data

Purpose: We present a framework for detecting possible adverse drug reactions (ADRs) using the Utah Medicaid administrative data. We examined four classes of ADRs associated with treatment of dementia by acetylcholinesterase inhibitors (AChEIs): known reactions (gastrointestinal, psychological disturbances), potential reactions (respiratory disturbance), novel reactions (hepatic, hematological disturbances), and death. Methods: Our cohort design linked drug utilization data to medical claims from Utah Medicaid recipients. We restricted the analysis to 50 years-old and older beneficiaries diagnosed with dementia-related diseases. We compared patients treated with AChEI to patients untreated with anti-dementia medication therapy. We attempted to remove confounding by establishing propensity-score-matched cohorts for each outcome investigated; we then evaluated the effects of drug treatment by conditional multivariable Cox-proportional-hazard regression. Acute and transient effects were evaluated by a crossover design using conditional logistic regression. Results: Propensity-matched analysis of expected reactions revealed that AChEI treatment was associated with gastrointestinal episodes (Hazard Ratio [HR]: 2.02; 95%CI: 1.28-3.2), but not psychological episodes, respiratory disturbance, or death. Among the unexpected reactions, the risk of hematological episodes was higher (HR: 2.32; 95%CI: 1.47-3.6) in patients exposed to AChEI. AChEI exposure was not associated with an increase in hepatic episodes. We also noted a trend, identified in the case-crossover design, toward increase odds of experiencing acute hematological events during AChEI exposure (Odds Ratio: 3.0; 95% CI: 0.97 - 9.3). Conclusions: We observed an expected association between AChEIs treatment and gastrointestinal disturbances and detected a signal of possible hematological ADR after treatment with AChEIs in this pilot study. Using this analytic framework may raise awareness of potential ADEs and generate hypotheses for future investigations. Early findings, or signal detection, are considered hypothesis generating since confirmatory studies must be designed to determine if the signal represents a true drug safety problem.

. In many previous studies, administrative data sources have been used to estimate the extent of the problem or confirm safety signals identified from AERS 4,5 . However, fewer studies have demonstrated the potential of administrative data for first-line adverse drug reaction (ADR) surveillance 6 . In this pilot study, we present a framework for directed discovery of possible ADRs using population-based administrative data sources, an approach intended to complement the FDA's adverse reporting system. We describe our approach as directed because we target specific health outcomes of interest instead of simply mining the data for statistical associations.
We examined the associations between drug use and possible ADRs resulting from treatment of dementia with acetylcholinesterase inhibitors (AChEIs), namely, donepezil hydrochloride, rivastigmine tartrate, and galantamine hydrobromide. We measured associations for four classes of ADEs-established reactions (gastrointestinal and psychological disturbance), potential reactions based on drug pharmacology (respiratory disturbance), novel unexpected reactions (hepatic and hematological disturbance), and death. Hepatic and hematologic syndromes were evaluated because they are two examples of potentially fatal reactions that have been found in postmarketing surveillance of drug-induced disease 7 .

Methods
The directed discovery framework consists of clinical framing, data preparation, event detection, and hypothesis generating and testing. The first three components are described in the Methods; hypothesis generating and testing are explored in the Discussion.
Clinical framing and data preparation Clinical framing consisted of reviewing the medical literature and consulting clinical experts to define the treatment groups, inclusion criteria, drug courses, outcomes and covariates.
Sources. Data consisted of pharmacy and medical claims and enrollment status from Utah Medicaid recipients in the fee-forservice program between 1/01/2003 and 12/31/2005. We linked Utah death-certificate data to Medicaid recipients by a deterministic method using a social security number. To protect patients' privacy, all potentially traceable personal identifiers were removed. The University of Utah Institutional Review Board approved this study (IRB_00016984).

Subjects.
We studied Utah Medicaid recipients' aged 50 and older with a dementia-type diagnosis ( Table 1). As Medicaid enrollment occurs on a monthly basis, we tracked membership enrollment and de-enrollment and censored the patients whose enrollment was terminated and not re-established within the study period. Because of the relatively high rate of sustained enrollment, approximately 99% of the cohort was enrolled for at least 80% of the months from their first until their last month of eligibility or until the study period ended. We did not limit inclusion to continuously enrolled recipients.

Treatment Groups.
We inferred patient AChEI use by reconstructing courses of AChEI therapy from pharmacy claims data.
To achieve a greater homogeneity among users' disease stage and risk of adverse reactions 8 , we restricted the AChEIs cohort to the first incident course of AChEI therapy, which was defined as their first course with at least a 180-day drug-free period. To ensure that patients were receiving medical care during the 180-day drug-free period and were not receiving the drug elsewhere, recipients' had to be enrolled and to have at least one medical claim during the 180-day drug-free (baseline) period. We defined a course of AChEI therapy as beginning on the week the drug was first dispensed and ending on day 60 after a continuous gap in the drug supply of ≥ 60 days ( Figure 1).  The untreated comparison group consisted of Medicaid recipients 50 years and older with a dementia-like diagnosis who did not receive AChEI therapy. We established a 180-day baseline period during which recipients were enrolled and had at least one medical claim. The index date for individuals in the untreated group began at the first dementia-related outpatient visit that allowed for a 180-day baseline period. Starting time zero with a dementia-related outpatient visit established an indicated population that was engaging the health care system.
Outcomes. As noted earlier, our primary clinical outcomes were gastrointestinal, psychological, respiratory, hematological and hepatic conditions, and death. We identified health care visits related to each clinical outcome in professional and facility claims using Healthcare Cost and Utilization Project (HCUP) Clinical Classification Software (CCS) codes (documented in Table 1). As a primary diagnosis typically indicates the reason for seeking medical care or the most important problem at the visit, we limited the outcome detection to the primary diagnosis codes. We tailored outcome classifications for each study design (described under Event Detection). Our analysis also measured the association of AChEI use with death.
Potential confounding. We assessed demographic variables, comorbidities, drug therapy, and indicators of health care utilization as potential confounders. Comorbidity indices included HCUP comorbidity software version 3.2 and the modified RxRisk-V (RxRisk-Vm) score, which infers comorbidity using pharmacy claims 9 . We measured health care utilization by considering the number of outpatient visits, hospitalizations, and emergency department (ED) visits, and we also accounted for use of hospice services and nursing home care.
We considered specific classes of medications as potential confounders -specifically, antianxiolytics, anticonvulsants, Parkinson's treatment, antidepressants, antipsychotics, steroids, narcotics, respiratory agents, anticoagulants, corticosteroids, and sedatives. We treated the use of statin drugs as an indicator of health status because they are preferentially prescribed to healthier, less frail patients who are not at the end of life 10 .

Person time unit.
We constructed the final analytic table using 1-week discrete time intervals; i.e., changes in covariate status, medication use and outcomes are captured weekly. This interval maximizes efficiency without omitting clinically important changes in patient outcome and covariate status. All database manipulation was conducted in SAS 9.2.

Event detection Cohort design.
We used an open cohort design with propensity score matching to explore associations between data on drug utilization and possible ADRs. We used propensity scores to address covariate imbalance using logistic regression models to predict AChEI treatment. We included confounders and risk factors in the propensity score models 11 . Because we included risk factors along with confounders, we built separate propensity score models and matched cohorts for each study outcome. Two physicians who routinely treat patients with dementia independently selected variables to construct propensity score models. They discussed disagreements to arrive at consensus. Variables for each model are listed in Table 3.
Our analyses included propensity score matching followed by additional matching on key prognostic covariates 12 . For example, we performed propensity matching with covariate matching whether an individual had a gastrointestinal visit during the baseline period when evaluating the gastrointestinal outcome. Analysis of death consisted of propensity score matching and covariate matching for baseline age and hospice care.
Clinical endpoints were intended to measure increased health care utilization associated with specific diagnoses. We defined episodes of care to differentiate clusters of events and to reduce the impact of immediate clinical exuberance associated with a new episode of care. A 4-week gap in claims for each clinical outcome was required to initiate a new episode. For each study endpoint, we calculated the incidence densities per 100 patient-years.
We established matched untreated cohorts using Mahalanobis metric matching 13 . Baseline characteristics of patients in the AChEI-treated and matched untreated cohorts were compared using Student's t-tests and chi-square tests. We used conditional multivariable Coxproportional hazard models that allowed for recurrent events to assess the effect of AChEI on specific clinical endpoints 14 . All statistical analyses were performed with Stata MP 9.2 for Windows.
Case-crossover design. We established three 6-week time-windows (pre-treatment, first treatment, second treatment window) to assess acute and transient effects of AChEI treatment ( Figure 1). The index week for the pre-treatment window was the week following the most recent clinic visit for any condition during the baseline period.
To capture acute effects of AChEI treatment, we used the week the AChEI was first dispensed as the index week for the first treatment window. We compared the odds of experiencing an event during that window with the odds of experiencing an event during the pretreatment window to identify acute treatment effects. To evaluate the transience or stability of possible ADRs, we compared the odds of experiencing an event during the second treatment window to the odds of experiencing an event during the pre-treatment window. Patients were noted as having an event if they had a medical claim with the primary clinical diagnosis code of interest; we used only one event per time-window. Odds ratios between the referent and treatment windows were computed using conditional logistic regression. See Figure 2 for a summary of the two designs.

Description of study population
Of the 29,046 eligible patients in the study populations, 4,109 had a medical claim with a dementia diagnosis between 1/01/2003 and 12/31/2005. The AChEI-treated cohort consisted of 976 total users and 332 users with incident courses; of the latter, 224 were started on donepezil, 59 on rivastigmine, and 49 on galantamine. Because the numbers of incident users of specific AChEIs were small, we did not assess potential ADRs for individual drugs. In the AChEI-treated group the median duration of incident courses was 33.4 weeks with an interquartile range (IQR) from 15 to 68.5 weeks. The median proportion of weeks for which the AChEI-treated group was estimated to have access to the medication at least 1 day during the week was 100%, with an IQR of 95%-100%. The untreated cohort consisted of 2,968 patients who were diagnosed with dementia but did not receive medication to treat the disorder during the study period (Figure 3).
Basic characteristics of the study population during the 6-month baseline period are presented in Table 2. Compared with the   After propensity score matching for each clinical endpoint, the two groups were similar on all variables for each outcome-based cohort, except the average number of ED visits, which was slightly higher in the untreated matched groups for the evaluation of respiratory and hepatic episodes (Table 3). In general, the lack of statistically significant differences between the AChEI-treated and untreated groups on propensity-adjusted variables suggests a balance in measured covariates between treatment groups. Table 4 presents the incidence densities per 100-person years and 95% confidence intervals for the complete untreated population and propensity-matched comparisons. Table 5 presents the hazard rates for all unadjusted and matched comparisons.

Crude analyses.
In bivariate analysis (Table 5) we did not observe a higher rate of gastrointestinal episodes in the group treated with AChEIs compared to the untreated group. The rates of psychological episodes, respiratory episodes, hematological episodes, and hepatic episodes were slightly higher, but not statistically significantly, in the group treated with AChEIs compared to the untreated group. The rate of death in the group treated with AChEIs was significantly lower than in the untreated group.

Propensity-matched analyses.
We observed significantly higher rates of gastrointestinal episodes (Hazard Ratio [HR]: 2.02; 95% CI: 1.28 -3.2) and hematologic episodes (HR: 2.32; 95% CI: 1.47 -3.67) in the AChEI-treated group than in the propensity-matched untreated group (Table 5). For psychological episodes, respiratory episodes, and hepatic episodes, we observed higher, but not statistically significant, rates in the AChEI-treated group than in the propensitymatched untreated group. We observed a weak and non-significant association between AChEI treatment and mortality.
Case-crossover analysis. In crossover analysis we did not observe increased odds of experiencing gastrointestinal events during either the first or second treatment windows. We observed an acute, but non-significant, effect of AChEI treatment on the odds of experiencing a psychological event during the first-treatment window; this was not sustained during the second-treatment window. We observed acute, but non-significant, effects of AChEI treatment on the odds of experiencing respiratory event and hematological events during the first-treatment window; both rates appeared to decrease during the second-treatment window. The acute effect of AChEI treatment on the odds of experiencing a hepatic event during the first-treatment window was imprecise and appeared to decrease during the second-treatment window ( Table 6).

Discussion
We developed a cohort-based framework for using populationbased administrative data to identify known ADRs and to discover ADRs that may have gone unnoticed during clinical trials. We evaluated AChEI therapy in people with dementia, considering a composite of possible ADRs─ i.e., expected, suspected, unexpected reactions, and death ─to demonstrate that our analytic techniques produced expected results. We used propensity score matching and a within-subject design in an attempt to handle confounding. Our pilot study examined data from patients diagnosed with dementia for both cumulative effects of AChEI treatment and acute effects following initiation of AChEI therapy. We demonstrated this approach with Medicaid data from the state of Utah; nonetheless, the framework presented here can be transferred for use with other health insurer databases, including the Medicare Parts A, B, and D data now available.
A pervasive issue in pharmacoepidemiologic studies is confounding by indication 15 . This problem arises because factors that influence treatment choices made by clinicians also influence outcomes. Confounding by indication can bias the crude association between drug treatment and outcomes in either direction and with unknown magnitude. Propensity score models are one method used in pharmacoepidemiologic studies to balance measured confounders with the goal of making the treatment groups exchangeable.
In this study, we addressed confounding by indication by developing propensity score models for each study outcome. Theoretical confounders available in the data were included in each model to reduce bias. Before matching, the untreated group appeared to be frailer than the treated group; they had a higher proportion of hospice care, more comorbidity, and a lower proportion of statin users, which suggested less aggressive care because of poorer health. As one would expect, the unadjusted analysis made AChEI treatment appear protective against mortality when compared with the untreated group (HR: 0.66; 95% CI: 0.52 -0.82), which is not supported by clinical trials or other observation studies 16,17 . After propensity and covariate matching we found no difference between the AChEI-treated and untreated groups (HR: 1.07; 95% CI: 0.74 -1.54). This illustrates the importance of addressing confounding by indication when designing ADR surveillance systems.
An alternative approach to addressing confounding is to use inverse probability weighting (IPW) methods to model time-varying treatments and confounders. In simulation studies, these methods were less biased than conventional methods when time-varying confounding was present 18 . When allowing treatment to be time-varying, we observed gastrointestinal disturbance and discovered hematological disturbance; we noted the same findings as if follow-up began at initiation of drug treatment (data not shown). Future work should explore the presence of time-varying confounding and the benefits of using IPW methods to discover novel ADRs associated with drug treatments.
To evaluate possible acute and transient effects of AChEI treatments, we employed a type of case-crossover analyses. Typically in case-crossover analyses, events are compared between event and control time-windows for each individual. A major benefit of this within-subject design is that each person acts as his or her own control 19,20 . It also accounts for confounding by indication and other time-invariant and difficult-to-measure confounders. The drawback of such designs involves changes in treatment utilization that are influenced by health status or the study endpoints in question 21 .
For example, when day-level drug utilization data are inferred from dispensing history, determining whether adverse effects are truly transient or the result of a decrease or discontinuation of drug p-value 0.14 0.57 treatment is difficult. Ultimately, we deemed the within subject analysis to be an excellent complement to the propensity score approach because of its ability to discover acute and transient effects and for its simplicity and ability to remove time-invariant confounding by indication.

Hypothesis generating and confirmation
The framework described here provides a structured approach for confirming expectations by evaluating known ADRs and discovering new ADR safety signals, such as the association we found between AChEI use and hematological disturbance. In support of the analytical effectiveness of these procedures, our approach confirmed an association with an expected reaction, gastrointestinal disturbances. The findings from the two study designs, however, were not consistent. Our inability to find an acute increase of gastrointestinal events in the first-treatment time window may be attributable to insensitivity of claims-based coding to identify symptoms of gastrointestinal disturbance.
Despite the fact that our approach detected a significant association with one expected reaction, gastrointestinal disturbance, it failed to identify a strong positive association with the second expected reaction, psychological disturbance. We did show a higher rate of psychological episodes in the propensity-matched analysis; nevertheless, the association was not statistically significant. We did; nonetheless, observe higher odds of experiencing psychological events in the first-treatment time window than in the pretreatment time window using the within subject design. Even though the higher odds was expected, it was not statistically significance. This result can likely be attributed to a combination of factors. First, is the low power in the within subject design and second may be insensitivity of claims-based coding to identify symptoms of psychological disturbance.
We discovered no clear associations between AChEIs and respiratory disturbance or death. In a recent sequence symmetry analysis, the initiators of AChEI had no detectable increased rate of complications of chronic airway disorders 25 . We found no clear evidence of an increase or decrease in mortality associated with AChEI treatment in published studies or meta-analysis to which to compare our results 16 .
Our analysis of unexpected reactions discovered a statistically significant positive association between AChEI treatment and hemotological episodes. Hematological events also appeared to be positively associated with early AChEI treatment. A detailed review of results with hematological event subcategories (not reported here) found that the rate of anemia was much higher in the AChEItreated group than in the untreated group during the first 6 weeks of drug treatment. Further analysis is required to determine if this higher rate is causally associated with initiating anti-dementia drug treatment. At present, no known pharmacologic or empirical reasons can explain how AChEI drugs cause hematological toxicity.
The incidence of hepatic disturbance appeared to be higher in the treated group, although non-significant, in both the within subject and propensity matched design. Hepatotoxicity was a major safety concern with tacrine, which is the reason why it is no longer a commonly used; hepatotoxicity has not been reported for other AChEIs 26 . Larger observational studies are needed to determine whether an association between AChEIs and hepatotoxicity exists.

Limitations
The results from this study are considered hypothesis generating rather than identifying causal treatment effects. Causal studies require validation of treatments, outcomes, and covariate classifications. Furthermore, causal studies require a stronger theoretical understanding and explication of the underlying causal relationships between the treatment and outcomes.
We compared AChEI-treated patients with an incident AChEI course of therapy, to an untreated cohort of patients with a dementia diagnosis. Other options were to compare directly the safety of AChEI products with one another or to compare the safety of AChEI therapy with the safety of other classes of medications used to treat such patients' dementia. We were not able to compare individual drug products. Treatment with AChEIs is not directly comparable to treatment with memantine, a glutamaterginc N-methyl D-aspartate (NMDA) receptor antagonist, because memantine is typically not the first-line treatment for dementia; rather it is used in addition to an AChEI therapy, complicating any comparison.
In pharmacoepidemiologic studies, an untreated referent group can also be defined as patients with an incident course of a medication that is not associated with the indication or evaluated outcomes. This type of "active control group" is likely to be more similar to the treated group in regard to the activation of the health care system than the indicated but untreated group 17 . Drug dispensing indicates that the patient has activated the health system. In addition, prescription of a new medication is likely to result in closer monitoring and evaluation of an individual's health status. The primary concern when comparing treated with untreated groups is under-recording of health conditions, making the members of the comparison group seem healthier than they really are, which can lead to overestimation of the effects of drug treatment.
Because of the multiple outcomes in this study, we were unable to identify a single medication that could yield comparable cohorts for all events. Instead, we used a dementia-related visit, not drug dispensing, as the index date for the untreated group. For both cohorts, the median amount of time to a clinic visit following the index date was 3 weeks, and the longitudinal visit process was also similar. These patterns suggest that health care access and followup may have been similar for the two groups.
Another limitation of this study is the small number of subject in the AChEI treatment group. This markedly limited our ability to confirm the expected adverse effects of AChEI treatment and discover adverse events that may have gone undetected in clinical trials.

Future research
The discovery of an association between a drug treatment and a theoretical reaction, an idiosyncratic reaction, or death is considered hypothesis generating or signal detection. Confirmation requires additional observational and possibly experimental studies. Ideally, discovered associations would first be confirmed or further characterized in large, disparate data sources to reproduce evidence of the association across different populations. In May 2008, the FDA published The Sentinel Initiative report to present the national strategy for monitoring medical product safety 27 . Their approach primarily establishes a nationwide health information network for confirmation of safety signals across multiple large databases. Additional observational studies along with richer clinical information such as electronic health records or prospectively designed studies, however, may be needed to characterize the causal relationship between a drug treatment and the adverse outcome.

Data availability
The raw data are available upon request. IRB approval and signed Data Use Agreements with the Utah Department of Health. For more information, please contact the corresponding author Brian Sauer.

Ethical considerations
The primary ethical consideration is the privacy and confidentiality of patient data. Limited datasets were used that restricted the use of direct patient identifiers. Data are stored on secured servers and only shared according to IRB policy and state data use agreements. Alison Bourke Cegedim Strategic Data, London, UK This is an interesting paper, and as stated, it would be useful to see the methodology repeated on larger datasets to increase the power. One small point -the authors say that their inability to find an acute increase of gastrointestinal events in the first-treatment time window may be attributable to insensitivity of claims-based coding to identify symptoms of gastrointestinal disturbance. Could this also be because they were only looking for the outcome as a primary diagnosis, and GI disturbance may be consider less important? Would it have been possible to use prescribing of certain therapeutic agents as a proxy of GI events I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Open Peer Review
No competing interests were disclosed. Competing Interests: 01  Overall Comments: This study presents a framework for discovering ADRs using population based administrative data sources. The approach is exemplified by use of AChEIs treatment on dementia. Overall, the study is very well done. However, there is an issue that needs to be clarified as following: On page 4, paragraph 3, the authors state that they treated the use of statin drugs as indicators of health status because they are preferably prescribed to less frail who are not at the end of life. Even though the statement seems reasonable the reference they provided does not support it.
The results of these pilot study shows that use the framework for discovering ADRs using population based administrative data a source is promising as a hypothesis generating approach.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that identify symptoms in a disease state."
This statement was speculation on our part. The rationale is that prescribers expected GI and psychological disturbance and may have just viewed that as an expected reaction to treatment and not a treated condition and therefore was not coded. Chart review would be required to verify these statements and we didn't have access to the medical notes for this study.
none Competing Interests: