Assessing the development and implementation of the Global Trigger Tool method across a large health system in Sicily

We appreciate the review by Professor Btrent of our paper, the accuracy of his observations and the relevance of his suggestions. We are honored that he considers our paper “useful because it supplies empiric observation of one such use, across a large system of hospitals and an extended period of time.”
 
We are aware that some of the points require further clarity, which may have been lost as we limited the word count.
 
Comment
The Harvard Medical Practice Study (HMPS – cited by the authors) used post-discharge chart review to detect care-associated injuries (adverse events – AEs) that occurred during hospitalization. Under HMPS, trained nurses reviewed a random selection of charts. If those nurses discovered what they judged to be care-associated injuries, they flagged those events in the chart as a potential AE. Charts that contained one or more AEs were forwarded to 2 independent physician reviewers. If both physician reviewers judged, for each AE, that event had occurred, then it was reported as a confirmed AE. Those physicians also judged whether each AE was avoidable, and whether each could be considered negligent (substandard) care.
The IHI Global Trigger Tool (IHI GTT) built on the HMPS methodology. It attempted to improve the ability of initial nurse reviewers to detect potential AEs, by providing a set of 51 “review triggers,” falling into 7 major subcategories – explicit initial events that, if detected in the chart, chained to examination of other specific events. Initial assessments of the IHI GTT showed much higher detection rates for AEs than did the original HMPS methodology, which itself far exceeded typical voluntary reporting mechanisms.
Response
Harvard's medical practice study is a key pillar in patient safety and care research. We agree that the IHI Global trigger tool is an attempt to improve the Harvard Medical Practice Study methodology.
In our opinion, the IHI Global Trigger Tool has two significant differences from the Harvard medical practice study.
The first is that the IHI Global Trigger Tool is conducted on a random sample of clinical records that are analyzed only if triggers are present.
The second is that the IHI Global Trigger Tool "focuses on only those adverse events related to the active delivery of care (commission) and excludes, as much as possible, issues related to substandard care” (omission). (IHI Global Trigger Tool for Measuring Adverse Events. Second Edition 2009. http://www.ihi.org/resources/Pages/Tools/IHIGlobalTriggerToolforMeasuringAEs.aspx)
 
 
Comment
This report has at least 2 major differences from other work in the field. First, only 6.6% of records reviewed had at least one care-associated adverse event. Other studies have shown much higher AE rates. The authors note this in their text, and suggest it may reflect differences in local environments and chart review methods.
Response
The percentages of adverse events found in other studies are very different. In a recent systematic review of the literature, the percentage of adverse events is between 7% and 51%. (Hibbert PD, Molloy CJ, Hooper TD, et al .: The application of the Global Trigger Tool: a systematic review. Int J Qual Health Care. 2016; 28 (6): 640-649.) We think that the most important reasons, which can explain these wide differences, could be twofold.
The first could be due to the different care settings (medicine, surgery, obstetrics, ICU) and therefore to the different complexity of the patients studied. A risk adjustment system would be needed to evaluate the results homogeneously. In our experience, for example, there are significant differences in the AEs rate between apparently homogeneous wards belonging to different hospitals. We have not included these tables to be brief. Since it seems to us that this is very interesting, this aspect will be the subject of other publications.
The second reason could be due to changes in the detection protocol. The survey methods are, in fact, different in the different studies. We, for example, have turned our attention only to the triggers observed during admission and not to the triggers present at the time of admission. Therefore the adverse events found do not include those present at the time of admission. This could justify the differences in EAs observed compared to other studies.
 
Comment
Second, most other IHI GTT reviews show adverse drug events (medication-related events - overdoses, drug-drug interactions, and allergic or idiosyncratic reactions) as the dominant category of AEs detected. In this study, they are a distant number 3. Why?
Response
We do not know why the drug-related AEs observed in our study are infrequent compared to other studies.
In our study the most frequently detected AEs are Health care–associated Infection. In addition, Rutberg et al. Report that                Health care–associated Infection accounted for 39% of Aes. (Rutberg H, Borgstedt Risberg M, Sjödahl R, et al. Characterization of adverse events detected in a university hospital: a 4-year study using the Global Trigger Tool method. BMJ Open. 2014; 4: e004879.)
Von Plessen and Coll report that Infections, pressure ulcers procedures-related and gastrointestinal problems were common. (Von Plessen C, Kodal AM, Anhøj J. Experiences with Global Trigger Tool reviews in five Danish hospitals: an implementation study. BMJ Open. 2012; 2: e001324.
Schildmeijer et al report that "Overall, the level of agreement for detecting AEs is the level of harm for healthcare-related infections, that is, pneumonia, sepsis and urinary tract infection." (Schildmeijer K, Nilsson L, Årestedt K, et al. Assessment of adverse events in medical care: lack of consistency between experienced teams using the global trigger tool. BMJ Qual Saf. 2012; 21: 307-314.)
Landrigan et al. report that "Harms that were detected in procedures (186/588)" (Landrigan CP, Parry GJ, Bones CB, et al. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010; 363: 2124-2134.)
Mortaro et al. report that the most common types of AEs detected were related to surgical procedures (Mortaro A, Moretti F, Pascu D, et al .: Adverse Events Detection Through Global Trigger Tool Methodology: Results From a 5-Year Study in an Italian Hospital and Opportunities to Improve Interrater Reliability. J Patient Saf. 2017.)
 
Comment
It appears that the entire IHI GTT program assessed a total of 18,008 records from among 105 hospital wards from June 2015 through June 2018. However, this analysis examines only medicine, surgery, obstetrics, and intensive care wards. Those represented a total 89 wards, with a total of 14,706 records reviewed. This is unclear in both your abstract and your text. It would be very helpful to more clearly explain how you derived the CRs included in your analysis.
Response
We agree that this may be unclear. From June 2015 to June 2018, 18.008 CRs were examined. In this study, we analyzed 14,706 CRs relating to patients discharged from 89 medicines, surgery, obstetrics and intensive care wards. 16 wards and 3,302 CRs concerned pediatrics wards and emergency department.
 
Comment
You mention that you had a total of 199 individuals who participated in 71 3-person review teams. Obviously, some people participated in more than 1 team. It is not clear how the 2 initial reviewers shared their work, before their findings were submitted to a physician for final validation. Did they both separately review all records? Did they divide their assigned records between them? Please clarify.
Response
The total number of reviewers was 199 divided into 71 teams. In some teams, medical reviewers  from another team were the supervisors. The two primary record reviewers should each review all records independently.  The third reviewer was always a physician. The physician, who did not review the records, authenticated the consensus of the two primary record reviewers.
 
Comment
In your introduction, 2nd paragraph, you review alternative methods for detecting AEs, including incident reporting, CR review, and automated administrative data review (e.g., PSIs). Consider adding “prospective (concurrent) clinical trigger systems,” that track possible real-time clinical responses to AEs then track back to see if an AE actually occurred. Dr. R. Scott Evans at LDS Hospital in Salt Lake City, Utah; and Dr. David Bates at Brigham & Women’s Hospital in Boston, Massachusetts, developed and demonstrated such systems. Such approaches find AEs that never make their way into a traditional clinical record, and may detect far more events.
Response
This is very interesting.
In part, this concept is reported in the conclusions "The introduction of electronic medical records would allow a quicker process with the automation of the identification of triggers and the possibility of linking together in the identification of adverse events and near misses, especially where there has been more than one trigger "
We will review the work and attempt to explain this concept better.
 
Comment
Under Methods, please change the heading “Sampling selection” to say something like “Sample selection.” More importantly, the frame within which “10 inpatient CRs” were selected monthly is not clear. Working the total number of charts reviewed backward, you were probably sampling “10 inpatient CRs” each month for each of the 44 Sicilian public hospitals. Please clarify. How was record selection balanced across the types of wards (medicine, surgery, obstetrics, and ICUs) in each hospital?
Response
We will change the heading "Sampling selection" to "Sample selection."
We randomly selected 10 CRs for each department that participated, not for each hospital. The wards that participated were not homogeneously represented in the different hospitals. In some hospitals only the ICU CRs were analyzed. In some hospitals there were no medical, surgical or obstetrics or ICU wards. In others there were more than one medicine, surgery, obstetrics and ICU wards. In addition, hospitals were recruited at different times.
The CRs sample is therefore not representative. The aim of our study was not to provide a true representation of the patient population present in hospitals during the observation period. Coherently with as reported with the IHI GTT protocol, the main purpose of applying GTT methodology is to produce a sampling approach that is sufficient for the design of safety work in the hospital.
 
 
Comment
Under Results, please add clarity regarding the number of patients that had at least 1 AE during their index hospitalization (975); then break out the number of patients who experienced a single AE versus those who had more than 1 AE during their index hospitalization.
Response
We will indicate the number of patients who experienced a single AE versus those who had more than 1 AE during their index hospitalization.
 
Comment
Table 1: Please clarify – I can’t understand what you mean by the phrases “CRs with triggers per inpatient wards” versus “CRs with trigger isolated per wards”; or “CRs with trigger” versus “CRs with trigger isolated”.
Response
We will modify the table.

• The second line "CRs with triggers for inpatient wards" indicates the number and percentage of CRs with one or more triggers.
• The third line "CRs with trigger isolated for wards" indicates the number and percentage of CRs with only triggers.
• The fourth and fifth lines are repetitions. It it is a printing error.

 
Comment
Consider moving the first 2 paragraphs of the section entitled “Rates of triggers” from the Discussion section into the Results section.
Response
We will move the first 2 paragraphs of the section entitled "Rates of triggers" from the Discussion section into the Results section.
 
Comment
Clarify your text in the first paragraph of your “Rates of triggers” section. For example, it might read:
“In this study, 37.9% (n=5,574) of all CRs examined had at least 1 positive trigger. Of those, 2,778 CRs had a single positive trigger (49.8% of all CRs with positive triggers) while 2,796 CRs had more than one positive trigger (51.2% of all CRs with positive triggers).
Response
Thank you for this good advice. We will modify accordingly.
 
Comment
a) Label each of the subsections in Table 2. For example, in your text you cite the Table relative to “general care triggers”. What are “general care triggers”? Presumably you mean the subset of triggers with “C” labels. It would be quite helpful if you put headings on each of the subsections (for C, M, S, P, and I) that labelled the contents of each subsection.
Response
We will put headings on each of the subsections (for C, M, S, P, and I) that labelled the contents of each subsection.
 
Comment
b) You are (quite appropriately) approaching AE detection as a screening test with 2 main steps. Step 1 involves evaluation of the initial triggers. Step 2 involves following up on positive initial triggers, to see if a validated AE emerges. This is done in the context of a general review by a trained clinical expert, who can (and sometimes does) detect AEs that fall outside of the trigger system.
It would be very useful if you separated the total time spent on CR review into (a) initial assessment of the triggers; versus follow-up analysis of the positive triggers.
Response
We are unclear of the changes required on this point.
 
 
Comment
it would similarly be very useful if, in the 3rd column of Table 2 (labelled “Times associated with AEs, n (%)” you showed the proportion of positive triggers that yielded a confirmed AE related to/deriving from the initial positive trigger. The Table, as currently constructed, is quite confusing – in 10 instances (C03, C04, C08, C11, C12, C14, M02, M04, I01, I03) the count in 3rd column is larger than the count in 2nd column. Presumably, this is because you are not connecting the trigger to derivative AEs (but I can’t really tell, with any certainty).
Response
In the third column of table 2 we reported the number of times a trigger was found.
In the fourth column the number of times in which an adverse event was related to that trigger.
The title of the fourth column (Times associated, with AE)does not express this clearly.
Therefore in the third column the triggers are indicated, while in the fourth column the adverse events are indicated. We have found some CRswith more than one adverse event. In the third column of table 2 we intended to report that the C03 trigger was found in 279 CRs examined. Instead, in the fourth column of Table 2 we wanted to indicate that in 438 adverse events the C03 trigger was present.
 
Comment
The section entitled “Rates of AEs”, first paragraph, contains the sentence:
 “It would appear that the percentage of clinical records with triggers increases with the increase of percentage of CRs examined, compared to patients discharged, while the percentage of CRs with AEs remains more stable.”
I can’t translate quite what that means. Please simplify and clarify. It may relate to the 2 upper lines in Figure 2, which both appear to be increasing over time. However, you supply no statistical analysis to show that trends over time in the 2 lines are statistically associated, and you offer no reasoning as to why such a relationship may have meaning.
Response
We mean that we did not find a correlation between the number of CRs examined in the period and the number of EAs observed in the same period. We will try to verify if the differences are statistically significant.
 
Comment
Table 3 appears to identify high frequency triggers, then count the number of AEs of any sort that occur in CRs that had a high frequency trigger. It would be much more useful if you tracked each trigger to related or derivative AEs, rather than treating AEs generically.
Response
This would be very interesting to describe, but we do not know if it will be possible to provide this kind of detail due to editorial guidelines. It could be the subject of a new study to be published later.
 
Comment
In the section labeled "Rates of AEs" you have a bulleted list. The last item in that list says:
 "Intensive-care-related triggers have been identified 1,817 times with an AE in 1,924 (i.e., it is very common for triggers to identify more in the same patient)".
 I think I get the gist of what you mean, but (a) this sentence needs more clarity; and (b) the main purpose of identifying AEs is to move towards intervention and prevention (better, safer care). In that circumstance, treating AEs as generic items is not very useful. It is necessary to list them by specific causes. In that framing, associating triggers (which are cause specific) with a total generic AE count is not very informative or useful. You might want to rethink your framing.
Response
This would also be interesting to describe. We have been required to be fit in with te word count and   to comply with the editorial guidelines. We will clarify this section