Triage science in the emergency department (ED) has evolved from sorting patients according to the severity and urgency of their medical conditions into an effective tool to ensure patient safety by managing appropriate emergency response time and patient flow. ^{1 , 2} Current triage systems have been developed based on the unique conditions of each country. ³

The Cipto Triage Method (CTM) is a three-tiered emergency department triage assessment tool based on patients’ characteristics and the public healthcare system in Indonesia. ⁴ This is a structured, non-scoring method developed in the Dr. Cipto Mangunkusumo Hospital to assess the clinical acuity performed in the ED by emergency medicine specialists, Emergency Medical Officers (EMO) or emergency nurses with comparable accuracy. ⁴

Previous criterion and construct validity evaluation of CTM for emergency nurses using retrospective chart review (prepared by an expert panel) and in-hospital mortality as the outcome showed that this method had good screening performance. ⁴ This further promotes the policy of nurse-based triage assessment for routine ED service as being simpler yet effective. ⁵

After five years of implementation, we wanted to further evaluate the accuracy of the CTM. This study compared triage decisions using the CTM, performed by triage nurses as the index test, to the EMO as the reference standard. Adult non-trauma patients were the target population because they had the highest proportion of ED cases. No sex differences were expected from the CTM assessment based on the previous study. ⁴ The null hypotheses were developed based on minimal acceptable criteria: index test sensitivity for resuscitation and urgent triage categories were <0.8 and index test specificity for non-urgent triage category was 0.8.

For the inter-rater reliability analysis using case vignettes, the response rate for both groups (nurses and EMO) was 100%. The work experience as a professional for the ED practitioner among the nurse group was longer than that of the EMO group. ²⁰ Agreement in the nurse and EMO groups was substantial (0.68; 95%CI 0.67–0.7) and very good (0.81; 95%CI 0.78–0.83), respectively, indicating that both nurses and EMO were consistent in triage decisions, although the EMO group had better performance agreement ( Table 1).

Table 2 shows that female participants were 53.9% of the sample, and the proportion of male and females were equal in resuscitation and non-urgent triage category. Based on the reference standard, the prevalence of participants in the resuscitation, urgent, and non-urgent triage categories was 23.7%, 62.6%, and 13.7%, respectively. Most participants were alert (94.9%), and more than half had pain during triage assessment (54.2%). The mean heart and respiratory rates were higher, whereas the mean systole and peripheral oxygen saturation were lower in the resuscitation category than in the other triage categories. Almost one-third had malignancy (29.3%), and most participants self-attended to the ED (non-referral cases). The discharges from the ED were 40.9% in all triage categories; only 2.5% were admitted to the intensive care unit, and 5.3% died in the ED. Among participants in the resuscitation and urgent categories, half of them were admitted to the ward. Furthermore, 13% of the patients in the non-urgent category were also admitted, and one patient in the non-urgent category subsequently died in the ED during emergency care.

The overall accuracy of the triage assessment of the index test was 84.7%. The highest proportion of mis-triage was overtriage of patients who had an urgent triage decision based on the index test, however, the actual condition based on the reference standard was non-urgent (7.1%).

Table 3 shows the diagnostic features of each triage decision based on the index test compared with the reference standard. ¹² The sensitivity of the index test for resuscitation and urgent and non-urgent was 84.6% and 92.7%, respectively, while the specificity of non-urgent was 97%. Based on the results of this study, the null hypothesis is rejected. The triage assessment using CTM performed by a nurse was acceptable for screening patient acuity in the ED. The PPV of resuscitation and urgent triage were very good (90.5% and 84.5%, respectively). In the non-urgent category, the PPV was acceptable (72%), but the NPV was very good (92.2%). ²⁰

Stratification based on age showed an increased PPV and positive likelihood ratio in the <60 years age group compared to the ≥60 years age group in the resuscitation and non-urgent categories. No statistically significant difference (<10%) in the PPV, NPV, and likelihood ratio was found in the urgent category after age stratification. ²⁰

Reliability assessment for the nurses and EMO groups showed that both achieved the minimum target of agreement (>0.67). ¹³ The reliability among nurses is similar to another study using the Manchester Triage Scale (MTS) method (weighted kappa 0.62; 95%CI 0.6–0.65) but lower compared to the Emergency Severity Index (ESI) (weighted kappa 0.79; 95%CI 0.74–0.84) ¹⁴ and the other three triage categories. ¹⁵ Meanwhile, the inter-rater agreement among EMO was comparable with that among physicians using the ESI method (weighted kappa 0.84; 95%CI 0.48–0.96). ¹⁴

Accuracy assessment was focused on the PPV, NPV, and positive likelihood ratio because of the importance of this information in daily practice. ¹⁶ Higher sensitivity was expected in the resuscitation and urgent categories because CTM acts as a screening tool for the risk of clinical severity and urgency. ¹⁶ Higher specificity was expected for the non-urgent triage category, because for the less severe conditions, the triage decision would be considered reliable when the triage officer is able to be more specific and prevent under-triage decisions. The lower PPV of non-urgent triage categories implied that the index test had excessive over-triage by putting non-urgent cases into higher triage categories. This also causes the specificity of the urgent category to be lower. Overtriage of non-urgent to the urgent category occurs because triage officers often detect initial symptoms that appear more prominent, and patients have many comorbidities.

The accuracy of CTM performed by the nurse was 84.7%, higher than the accuracy of triage assessment using ESI (59.2%). ¹⁷ Being a single-centre study, lower triage category (three categories) and previous triage training before the study conducted could be attributed to higher accuracy compared to the other triage methods.

The current data showed a consistent PPV in the resuscitation and lower PPV in the non-urgent categories (90.5% and 72%, respectively), which is comparable to a previous validation study of CTM performed on the general population attending the ED (99% and 94.1%, respectively). ⁴

Another triage validation study using MTS (five categories) showed relatively lower sensitivity (47%–87%) of immediate and urgency categories compared to the reference standard (three categories) in the general adult population. ¹⁸ Meanwhile validation of ESI in adult ED patients showed sensitivity (83%) and specificity (82%) in detecting higher-urgency categories. ¹⁹ A systematic review by Zachariasse et al. reported 33 types of triage systems with large variations in accuracy, and concluded that The Canadian Triage and Acuity Scale (CTAS), ESI, and MTS had reasonable validity, established standardised guidelines and training programmes. ³

The different proportions of PPV in the resuscitation and non-urgent triage categories after age stratification ( Table 3) indicated that age should always be measured during the estimation of clinical acuity in ED triage. Lower PPV and positive likelihood ratio of resuscitation category in older participants indicated that vague signs and symptoms in geriatric patients could reduce the triage nurses’ ability to detect the risk of critical condition in new ED patients. Sex-based analysis was not performed because we already excluded pregnancy in the eligible criteria, thus all eligible medical condition that included in the study had similar triage assessment method for males and females.

Based on hospital characteristics, the results of this study can be interpolated to other referral hospitals in Indonesia. Triage assessment by nurses using CTM could also be implemented in other public or private hospitals in Indonesia because it has already incorporated a method to collect information regarding patients’ potential severity and urgency with simple triage categories.

This study had several limitations, including a review bias due to non-blinding of the reference standard of the index test results, thereby leading to overestimation of sensitivity and specificity. Spectrum bias might have occurred due to the hospital’s status as a referral centre, leading to higher prevalence of patients in the resuscitation triage category than that in the non-referral hospital; this could have resulted in overestimation of PPV of the resuscitation category performed by the index test. Fewer participants in the non-urgent triage categories might have contributed to the wider confidence interval of sensitivity, specificity, PPV, NPV, and likelihood ratio of non-urgent triage category. Lack of sex-based stratification might underestimate the effect of sex on the clinical presentation in adult.

In summary, nurse-based triage assessment using the CTM in the ED had very good accuracy in determining the patient’s severity level. Therefore, the development of formal triage training to improve the triage performance of nurses and physicians in the ED is recommended. To improve consistency, further research could focus on developing a clinical decision analysis for ED triage based on various clinical scenarios.

This study had ethical approval from Ethical Committee of Cipto Mangukusumo Hospital/Faculty of Medicine Universitas Indonesia protocol number 21-11-1250, all study participants or their representative had given written consent. All methods were carried out in accordance with relevant guidelines and regulations.