The Modified Early Warning Score as a Predictive Tool During Unplanned Surgical Intensive Care Unit Admission

Background: The Modified Early Warning Score (MEWS) has been proposed to warn healthcare providers of potentially serious adverse events. We evaluated this scoring system during unplanned escalation of care in hospitalized surgical patients during a 1-year period. Methods: Following institutional review board approval, all consecutive, unplanned surgical admissions into the surgical intensive care unit (SICU) during 2016 were entered into this study. MEWS and patient demographics during bedside evaluation for SICU admission were extracted from electronic medical records. Logistic regression was used to analyze the association of MEWS with the incidence of future mortality. P values were set at <0.01 for statistical significance. Results: In this series of 263 consecutive patients, the incidence of mortality following unplanned escalation of care was 29.3% (confidence interval [CI] 24.1% to 35.0%), ranging from 22% to 57%, with all positive MEWS values. The association of MEWS with future mortality was not statistically significant (P=0.0107). A misclassification rate of 0.29 (CI 0.24 to 0.35) was observed with this association. Conclusion: MEWS provided no clinical benefit as an early warning system, as mortality was elevated throughout the MEWS scale in this clinical setting. The high misclassification rate indicates MEWS does not provide discriminatory support for patients at risk for mortality.


INTRODUCTION
The Modified Early Warning Score (MEWS) has been proposed to warn healthcare providers of the potential development of serious adverse events, including unplanned escalation of care. [1][2][3][4][5][6] MEWS is composed of bedside measurements of heart rate, respiratory rate, systolic blood pressure, temperature, and level of consciousness (alert, responsive to voice, responsive to pain, and unresponsive). 6,7 The values of these measurements are scored and ranked, with a clinical response initiated once predetermined threshold scores are exceeded. 6,7 Two studies have suggested that the use of early warning systems, such as MEWS, might be beneficial in reducing mortality in hospitalized patients. 8,9 Pittard developed an outreach monitoring service in 3 surgical wards to assess the benefits of this new service on unplanned admissions to intensive care units (ICUs), length of stay, and mortality rates. 10 However, McGaughey and colleagues expressed concerns that the implementation of the early warning system was not based upon robust, evidence-based research. 11 Le Lagadec and Dwyer observed that although aggregated weighted scoring systems are frequently used, the efficiency of the specific early warning system appears to be dependent upon the patient cohort, facilities available, and staff training and attitude. 12 We evaluated this scoring system when used as a component of bedside evaluation during unplanned escalation of care in hospitalized surgical patients following systemwide implementation of the electronic warning system available through the Epic electronic medical record.

METHODS
Following institutional review board approval, all adult (ࣙ18 years of age), consecutive, unplanned surgical admissions to the surgical intensive care unit (SICU) during 2016 were entered into this study. The dataset for MEWS analysis was 100% complete. Calculated MEWS values during bedside evaluation before unplanned SICU admission were extracted from electronic medical records. Data are expressed either as counts and percentages or as medians with 25%-75% interquartile and full ranges. Logistic regression was used to analyze the association of the bedside MEWS values on the incidence of future mortality. Key analyses have associated 95% confidence intervals (CI). 13,14 The effect size for this model was analyzed with likelihood odds ratios. 15,16 The discriminative ability of this model was analyzed with C-statistics. 17,18 The predictive accuracy of this model was analyzed with misclassification rates. [19][20][21] Internal model validation was conducted with the statistical technique of bootstrapping (1,000 cycles) to confirm that the calculated CIs provided a range of probable population values that were consistent with our data analysis in this clinical setting. 14,22,23 P values were set for statistical significance at <0.01 to minimize the risk of false discovery rates or in declaring associations significant by chance alone. 24,25 The program JMP 13.2 (SAS Institute) was used for the statistical analysis of the dataset.

RESULTS
For this series of 263 consecutive patients, demographics and comorbidities during evaluation for unplanned escalation of care are shown in Table 1. The etiologies for unplanned SICU admission are shown in Table 2. The incidence of mortality following unplanned escalation of care was 29.3% (CI 24.1% to 35.0%). MEWS values ranging from 0 to 8, when plotted against future mortality rates, were not statistically significant (chi-square [χ 2 ]=6.5, P=0.0107), with an unadjusted odds ratio per unit change of 1.2 (CI 1.1 to 1.5) and a C-index value of 0.60 (CI 0.54 to 0.66) ( Figure). Probabilities and associated predictive modeling calculations across the range of MEWS values are shown in Table  3. Probability for mortality rates were observed with all positive MEWS values and ranged from 22% to 57% (Figure and  Table 3). Large numbers of false positive and false negative values were observed with all MEWS values. A cut-point of 3 was calculated by the statistical program based upon highest Sensitivity -(1-Specificity) percentile (Table 3).
We developed a confusion matrix with our dataset to analyze the performance of this early warning system in which

Modified Early Warning Score
Unplanned SICU admission contributes to morbidity and mortality. 26 The incidence of mortality following unplanned escalation of care in this study was 29.3% (CI 24.1% to 35.0%). This observation is similar to the reported incidences of mortality ranging from 17% to 76% in clinical studies of unplanned escalation of care. [26][27][28][29][30][31] MEWS was originally proposed as a tool to inform healthcare providers of the potential for development of critical illness in emergency rooms and in high-care units, but it quickly became adopted for use on hospital wards. [2][3][4]6,7 Subsequent studies have evaluated the benefits of early warning systems such as MEWS but have reported conflicting responses, with some studies showing benefits, 7,32 whereas other studies have not shown beneficial effects. 33 Cuthbertson et al observed that some physiologic measures and MEWS were predictive in surgical patients requiring ICU admission, but they acknowledged that their study was limited by missing data and that MEWS required prospective validation. 32 In our study, increasing MEWS values were clinically, but not statistically, associated with prognosis.
As this study may be the first to contain a complete dataset in this clinical setting, we conducted internal validation with bootstrapping to determine probable population values, 14,22,23 but our findings need external validation. The kappa and Youden J values were close to zero, suggesting that MEWS as an early warning system was unusable. In addition, the misclassification rate of 29% suggests poor calibration using MEWS as a tool in predicting mortality. [19][20][21] Within our clinical setting, MEWS was not useful as an early warning system, and our findings suggest that an unmeasured confounder exists that triggered nursing services to notify the rapid response team.
Early studies identified that predictability with MEWS may have been limited because of incomplete datasets and inadequate healthcare personnel education. 1,2,34 Ludikhuize et al were able to improve nursing identification of deteriorating patients following introduction of MEWS but noted that this early warning system was rarely used. 34 However, our institution provided nursing education during development and implementation of MEWS, and this scoring system automatically, rather than manually, calculated and recorded scores in the electronic medical record with a color-coded alert system displayed on computers or smartphones for all patient healthcare providers. In the study by Ludikhuize et al, the need to manually calculate scores may have played a role in low adherence with MEWS. 34 In our study, the statistical program calculated the optimum cut-point value of 3, and using this cut-point, we calculated an unadjusted odds ratio of 2.1 (CI 1.2 to 3.5) and a relative risk probability for death of 1.  Table 3), the key finding in our study is an unacceptable misclassification rate of 29%, wherein our group of patients with MEWS values of 1 and 2 had unacceptable mortality rates. We find that MEWS  is an unacceptable predictive tool under escalation-of-care conditions.

Statistical Analysis
In predictive modeling, forecasting adverse events is highly desirable when the potential prognosis is severe or if consequences increase with delayed diagnosis. 21 The discriminative power of a model can be calculated by several mathematical processes to assess predictive accuracy. 21,35 Sensitivity and specificity calculations provide estimates of illness probability, and predictive values provide additional assessments that patients with a positive test do have the condition or patients with a negative test do not have the condition (Table 4). The use of various odds ratios, especially the use of likelihood odds ratios, provides a measure of effect size (Table 4), and the use of C-statistics (Figure) provides a measure of discrimination. [15][16][17][18]35 However, these test statistics may not perform well in lowprevalence conditions 7,36,37 and may overestimate their benefits or underestimate the costs of clinical resources. 21,35,38 Clinicians need a testing tool to limit the potential for negative consequences on patient health and on medical care expenditures. 35 Misclassification rates support that answer (Table 3). Misclassification rates identify how often the model is wrong and account for the prevalence of the condition in question. [19][20][21] In our study, the high misclassification rate for MEWS and the number needed to misdiagnose of 1 in 3.46 ( Table 4) strongly suggest that this scoring system, when evaluated under these clinical conditions, is not effective as an early warning system. [19][20][21]

Limitations and Strengths
One limitation of this study is that the dataset is not representative of all patients on general wards but of a clinical setting during an unplanned escalation of care, so these results should only be interpreted in this setting. Another limitation is the need for these results to be confirmed by other centers under similar clinical settings. However, as already noted, this model underwent bootstrapping to provide internal validation to allow inferences about similar clinical populations and to check the stability of the results.
One strength of this study is the robust, complete set of MEWS values from bedside evaluation of patients during escalation of care, the time when MEWS should be most useful. Measures of effect size and C-statistics for discriminative ability were provided, in which these values were poor. The major strength of this study is the use of misclassification rates, a valuable mathematical calculation that identifies when a tool incorrectly classifies patients, leading to misjudgments in clinical care.

CONCLUSION
The key finding in this study is that MEWS is not an effective early warning system in surgical patients undergoing bedside evaluation during escalation of care. The misclassification rate in this model is high, and hence, MEWS could serve to misinform clinicians as to the nature of the patient's condition, with the result of either undertreatments, leading to severe consequences, or overtreatments, with attendant risks from unnecessary therapies and associated costs. Further, MEWS did not provide discriminatory support for patients at risk for mortality.