Abstract
Background: The Modified Early Warning Score (MEWS) has been proposed to warn healthcare providers of potentially serious adverse events. We evaluated this scoring system during unplanned escalation of care in hospitalized surgical patients during a 1-year period.
Methods: Following institutional review board approval, all consecutive, unplanned surgical admissions into the surgical intensive care unit (SICU) during 2016 were entered into this study. MEWS and patient demographics during bedside evaluation for SICU admission were extracted from electronic medical records. Logistic regression was used to analyze the association of MEWS with the incidence of future mortality. P values were set at <0.01 for statistical significance.
Results: In this series of 263 consecutive patients, the incidence of mortality following unplanned escalation of care was 29.3% (confidence interval [CI] 24.1% to 35.0%), ranging from 22% to 57%, with all positive MEWS values. The association of MEWS with future mortality was not statistically significant (P=0.0107). A misclassification rate of 0.29 (CI 0.24 to 0.35) was observed with this association.
Conclusion: MEWS provided no clinical benefit as an early warning system, as mortality was elevated throughout the MEWS scale in this clinical setting. The high misclassification rate indicates MEWS does not provide discriminatory support for patients at risk for mortality.
INTRODUCTION
The Modified Early Warning Score (MEWS) has been proposed to warn healthcare providers of the potential development of serious adverse events, including unplanned escalation of care.1-6 MEWS is composed of bedside measurements of heart rate, respiratory rate, systolic blood pressure, temperature, and level of consciousness (alert, responsive to voice, responsive to pain, and unresponsive).6,7 The values of these measurements are scored and ranked, with a clinical response initiated once predetermined threshold scores are exceeded.6,7 Two studies have suggested that the use of early warning systems, such as MEWS, might be beneficial in reducing mortality in hospitalized patients.8,9 Pittard developed an outreach monitoring service in 3 surgical wards to assess the benefits of this new service on unplanned admissions to intensive care units (ICUs), length of stay, and mortality rates.10 However, McGaughey and colleagues expressed concerns that the implementation of the early warning system was not based upon robust, evidence-based research.11 Le Lagadec and Dwyer observed that although aggregated weighted scoring systems are frequently used, the efficiency of the specific early warning system appears to be dependent upon the patient cohort, facilities available, and staff training and attitude.12
We evaluated this scoring system when used as a component of bedside evaluation during unplanned escalation of care in hospitalized surgical patients following systemwide implementation of the electronic warning system available through the Epic electronic medical record.
METHODS
Following institutional review board approval, all adult (≥18 years of age), consecutive, unplanned surgical admissions to the surgical intensive care unit (SICU) during 2016 were entered into this study. The dataset for MEWS analysis was 100% complete. Calculated MEWS values during bedside evaluation before unplanned SICU admission were extracted from electronic medical records. Data are expressed either as counts and percentages or as medians with 25%-75% interquartile and full ranges. Logistic regression was used to analyze the association of the bedside MEWS values on the incidence of future mortality. Key analyses have associated 95% confidence intervals (CI).13,14 The effect size for this model was analyzed with likelihood odds ratios.15,16 The discriminative ability of this model was analyzed with C-statistics.17,18 The predictive accuracy of this model was analyzed with misclassification rates.19-21 Internal model validation was conducted with the statistical technique of bootstrapping (1,000 cycles) to confirm that the calculated CIs provided a range of probable population values that were consistent with our data analysis in this clinical setting.14,22,23 P values were set for statistical significance at <0.01 to minimize the risk of false discovery rates or in declaring associations significant by chance alone.24,25 The program JMP 13.2 (SAS Institute) was used for the statistical analysis of the dataset.
RESULTS
For this series of 263 consecutive patients, demographics and comorbidities during evaluation for unplanned escalation of care are shown in Table 1. The etiologies for unplanned SICU admission are shown in Table 2. The incidence of mortality following unplanned escalation of care was 29.3% (CI 24.1% to 35.0%). MEWS values ranging from 0 to 8, when plotted against future mortality rates, were not statistically significant (chi-square [χ2]=6.5, P=0.0107), with an unadjusted odds ratio per unit change of 1.2 (CI 1.1 to 1.5) and a C-index value of 0.60 (CI 0.54 to 0.66) (Figure).
Demographics and Comorbidities in Patients With Unplanned Escalation of Care
Admission Etiologies in Patients With Unplanned Escalation of Care
![Association of Modified Early Warning Scores (MEWS) on prognosis during unplanned escalation of care. The line plots the probability of prognosis by MEWS values. Points below the line identify deceased patients. Points above the line identify alive patients. The whole-model statistic is χ2=6.5, P=0.0107; C-index=0.60 (confidence interval [CI] 0.54 to 0.66). Following bootstrapping of the model (1,000 cycles), the whole-model statistic was within the CI range (0.56 to 18.3) of probable population values for this clinical setting.](https://www.ochsnerjournal.org/content/ochjnl/20/2/176/F1.large.jpg)
Association of Modified Early Warning Scores (MEWS) on prognosis during unplanned escalation of care. The line plots the probability of prognosis by MEWS values. Points below the line identify deceased patients. Points above the line identify alive patients. The whole-model statistic is χ2=6.5, P=0.0107; C-index=0.60 (confidence interval [CI] 0.54 to 0.66). Following bootstrapping of the model (1,000 cycles), the whole-model statistic was within the CI range (0.56 to 18.3) of probable population values for this clinical setting.
Probabilities and associated predictive modeling calculations across the range of MEWS values are shown in Table 3. Probability for mortality rates were observed with all positive MEWS values and ranged from 22% to 57% (Figure and Table 3). Large numbers of false positive and false negative values were observed with all MEWS values. A cut-point of 3 was calculated by the statistical program based upon highest Sensitivity – (1–Specificity) percentile (Table 3).
Probabilities, Associated Calculations, and Cross-Classifications for Testing Across Modified Early Warning Scores (MEWS) in Patients With Unplanned Escalation of Care
We developed a confusion matrix with our dataset to analyze the performance of this early warning system in which the true outcomes are known (Table 4). This model had an accuracy of 0.711 (CI 0.69 to 0.73), with a prevalence for survival of 0.707 (71%, CI 0.65 to 0.76) and a prevalence for fatality of 0.293 (29%, CI 0.241 to 0.35). The sensitivity of the model was 0.984 (CI 0.97 to 0.996), with a specificity of 0.052 (CI 0.019 to 0.080). The positive predictive value was 0.715 (CI 0.71 to 0.72), and the negative predictive value was 0.571 (CI 0.20 to 0.88). The likelihood odds ratio for a positive test was 1.038 (CI 0.99 to 1.08), and the likelihood odds ratio for a negative test was 0.310 (CI 0.06 to 1.61). The number needed to diagnose was 27.9 (CI 13.23 to 88.03), and the number needed to misdiagnose was 3.46 (CI 3.24 to 3.67). The kappa value was 0.049 (CI –0.015 to 0.103), and the Youden J value was 0.036 (CI –0.01 to 0.08). Finally, the misclassification rate observed in this study was 0.289 (29%, CI 0.27 to 0.31) (Table 4). Bootstrapping of this model calculated that the whole-model statistic (χ2=6.5) was within the CI range (0.56 to 18.3) of probable population values for this clinical setting.
Confusion Matrix for Modified Early Warning Score During Bedside Evaluation in Unplanned Escalation of Care
DISCUSSION
Modified Early Warning Score
Unplanned SICU admission contributes to morbidity and mortality.26 The incidence of mortality following unplanned escalation of care in this study was 29.3% (CI 24.1% to 35.0%). This observation is similar to the reported incidences of mortality ranging from 17% to 76% in clinical studies of unplanned escalation of care.26-31
MEWS was originally proposed as a tool to inform healthcare providers of the potential for development of critical illness in emergency rooms and in high-care units, but it quickly became adopted for use on hospital wards.2-4,6,7 Subsequent studies have evaluated the benefits of early warning systems such as MEWS but have reported conflicting responses, with some studies showing benefits,7,32 whereas other studies have not shown beneficial effects.33 Cuthbertson et al observed that some physiologic measures and MEWS were predictive in surgical patients requiring ICU admission, but they acknowledged that their study was limited by missing data and that MEWS required prospective validation.32 In our study, increasing MEWS values were clinically, but not statistically, associated with prognosis.
As this study may be the first to contain a complete dataset in this clinical setting, we conducted internal validation with bootstrapping to determine probable population values,14,22,23 but our findings need external validation. The kappa and Youden J values were close to zero, suggesting that MEWS as an early warning system was unusable. In addition, the misclassification rate of 29% suggests poor calibration using MEWS as a tool in predicting mortality.19-21 Within our clinical setting, MEWS was not useful as an early warning system, and our findings suggest that an unmeasured confounder exists that triggered nursing services to notify the rapid response team.
Early studies identified that predictability with MEWS may have been limited because of incomplete datasets and inadequate healthcare personnel education.1,2,34 Ludikhuize et al were able to improve nursing identification of deteriorating patients following introduction of MEWS but noted that this early warning system was rarely used.34 However, our institution provided nursing education during development and implementation of MEWS, and this scoring system automatically, rather than manually, calculated and recorded scores in the electronic medical record with a color-coded alert system displayed on computers or smartphones for all patient healthcare providers. In the study by Ludikhuize et al, the need to manually calculate scores may have played a role in low adherence with MEWS.34
In our study, the statistical program calculated the optimum cut-point value of 3, and using this cut-point, we calculated an unadjusted odds ratio of 2.1 (CI 1.2 to 3.5) and a relative risk probability for death of 1.7 (CI 1.1 to 2.4) in patients with MEWS ≥3 compared to patients with MEWS values <3 (whole-model statistic χ2=7.0, P=0.0080), with a misclassification rate of 0.40 (CI 0.35 to 0.46). Although our results show that increasing mortality rates are associated with increasing MEWS values (Figure and Table 3), the key finding in our study is an unacceptable misclassification rate of 29%, wherein our group of patients with MEWS values of 1 and 2 had unacceptable mortality rates. We find that MEWS is an unacceptable predictive tool under escalation-of-care conditions.
Statistical Analysis
In predictive modeling, forecasting adverse events is highly desirable when the potential prognosis is severe or if consequences increase with delayed diagnosis.21 The discriminative power of a model can be calculated by several mathematical processes to assess predictive accuracy.21,35 Sensitivity and specificity calculations provide estimates of illness probability, and predictive values provide additional assessments that patients with a positive test do have the condition or patients with a negative test do not have the condition (Table 4). The use of various odds ratios, especially the use of likelihood odds ratios, provides a measure of effect size (Table 4), and the use of C-statistics (Figure) provides a measure of discrimination.15-18,35 However, these test statistics may not perform well in low-prevalence conditions7,36,37 and may overestimate their benefits or underestimate the costs of clinical resources.21,35,38 Clinicians need a testing tool to limit the potential for negative consequences on patient health and on medical care expenditures.35 Misclassification rates support that answer (Table 3). Misclassification rates identify how often the model is wrong and account for the prevalence of the condition in question.19-21 In our study, the high misclassification rate for MEWS and the number needed to misdiagnose of 1 in 3.46 (Table 4) strongly suggest that this scoring system, when evaluated under these clinical conditions, is not effective as an early warning system.19-21
Limitations and Strengths
One limitation of this study is that the dataset is not representative of all patients on general wards but of a clinical setting during an unplanned escalation of care, so these results should only be interpreted in this setting. Another limitation is the need for these results to be confirmed by other centers under similar clinical settings. However, as already noted, this model underwent bootstrapping to provide internal validation to allow inferences about similar clinical populations and to check the stability of the results.
One strength of this study is the robust, complete set of MEWS values from bedside evaluation of patients during escalation of care, the time when MEWS should be most useful. Measures of effect size and C-statistics for discriminative ability were provided, in which these values were poor. The major strength of this study is the use of misclassification rates, a valuable mathematical calculation that identifies when a tool incorrectly classifies patients, leading to misjudgments in clinical care.
CONCLUSION
The key finding in this study is that MEWS is not an effective early warning system in surgical patients undergoing bedside evaluation during escalation of care. The misclassification rate in this model is high, and hence, MEWS could serve to misinform clinicians as to the nature of the patient's condition, with the result of either undertreatments, leading to severe consequences, or overtreatments, with attendant risks from unnecessary therapies and associated costs. Further, MEWS did not provide discriminatory support for patients at risk for mortality.
This article meets the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties Maintenance of Certification competencies for Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement.
ACKNOWLEDGMENTS
This study was internally funded by Ochsner Clinic Foundation, but the institution had no role in the design or analysis of this study. Results of this study were presented in part at the Louisiana Society of Anesthesiologists Annual Meeting, New Orleans, LA in April 2019 and at the 15th Annual Ochsner Research Day, New Orleans, LA in May 2018. The authors acknowledge the information technology department at Ochsner Health for implementation of the Modified Early Warning Score system into the electronic medical record, with real-time integration of a color-coded early warning system for local and remote access. The authors have no financial or proprietary interest in the subject matter of this article.
- ©2020 by the author(s); Creative Commons Attribution License (CC BY)
©2020 by the author(s); licensee Ochsner Journal, Ochsner Clinic Foundation, New Orleans, LA. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (creativecommons.org/licenses/by/4.0/legalcode) that permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
REFERENCES
- 1.↵MitchellIA, McKayH, Van LeuvanC, A prospective controlled trial of the effect of a multi-faceted intervention on early recognition and intervention in deteriorating hospital patients. Resuscitation. 2010 Jun;81(6):658-666. doi: 10.1016/j.resuscitation.2010.03.001.
- 2.↵HammondNE, SpoonerAJ, BarnettAG, CorleyA, BrownP, FraserJF. The effect of implementing a modified early warning scoring (MEWS) system on the adequacy of vital sign documentation. Aust Crit Care. 2013 Feb;26(1):18-22. doi: 10.1016/j.aucc.2012.05.001.
- 3.LudikhuizeJ, SmorenburgSM, de RooijSE, de JongeE. Identification of deteriorating patients on general wards; measurement of vital parameters and potential effectiveness of the modified early warning score. J Crit Care. 2012 Aug;27(4):424.e7-13. doi: 10.1016/j.jcrc.2012.01.003.
- 4.↵van GalenLS, StruikPW, DriesenBE, Delayed recognition of deterioration of patients in general wards is mostly caused by human related monitoring failures: a root cause analysis of unplanned ICU admissions. PLoS One. 2016 Aug 18;11(8):e0161393. doi: 10.1371/journal.pone.0161393.
- 5.ChurpekMM, YuenTC, ParkSY, GibbonsR, EdelsonDP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards. Crit Care Med. 2014 Apr;42(4):841-848. doi: 10.1097/CCM.0000000000000038.
- 6.↵MorganR, Lloyd-WilliamsF, WrightM, Morgan-WarrenRJ. An early warning scoring system for detecting developing critical illness. Clin Intensive Care. 1997;8(2):S100.
- 7.↵van RooijenCR, de RuijterW, van DamB. Evaluation of the threshold value for the early warning score on general wards. Neth J Med. 2013 Jan;71(1):38-43.
- 8.↵BuistMD, JarmolowskiE, BurtonPR, BernardSA, WaxmanBP, AndersonJ. Recognising clinical instability in hospital patients before cardiac arrest or unplanned admission to intensive care. A pilot study in a tertiary-care hospital. Med J Aust. 1999 Jul 5;171(1):22-25.
- 9.↵SubbeCP, KrugerM, RutherfordP, GemmelL. Validation of a modified early warning score in medical admissions. QJM. 2001 Oct;94(10):521-526. doi: 10.1093/qjmed/94.10.521.
- 10.↵PittardAJ. Out of our reach? Assessing the impact of introducing a critical care outreach service. Anaesthesia. 2003 Sep;58(9):882-885. doi: 10.1046/j.1365-2044.2003.03331.x.
- 11.↵McGaugheyJ, AlderdiceF, FowlerR, KapilaA, MayhewA, MoutrayM. Outreach and early warning systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007 Jul 18;(3):CD005529. doi: 10.1002/14651858.CD005529.pub2.
- 12.↵Le LagadecMD, DwyerT. Scoping review: the use of early warning systems for the identification of in-hospital patients at risk of deterioration. Aust Crit Care. 2017 Jul;30(4):211-218. doi: 10.1016/j.aucc.2016.10.003.
- 13.↵BaghiH, NoorbaloochiS, MooreJB. Statistical and nonstatistical significance: implications for health care researchers. Qual Manag Health Care. 2007 Apr-Jun;16(2):104-112.
- 14.↵VisintainerPF, TejaniN. Understanding and using confidence intervals in clinical research. J Matern Fetal Med. 1998 Jul-Aug;7(4):201-206. doi: 10.1002/(SICI)1520-6661(199807/08)7:4<201::AID-MFM8>3.0.CO;2-M.
- 15.↵KimHY. Statistical notes for clinical researchers: effect size. Restor Dent Endod. 2015 Nov;40(4):328-331. doi: 10.5395/rde.2015.40.4.328.
- 16.↵SullivanGM, FeinnR. Using effect size-or why the P value is not enough. J Grad Med Educ. 2012;4(3):279-282. doi: 10.4300/JGME-D-12-00156.1.
- 17.↵MerkowRP, HallBL, CohenME, Relevance of the C-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg. 2012 May;214(5):822-830. doi: 10.1016/j.jamcollsurg.2011.12.041.
- 18.↵AltmanDG, BlandJM. Diagnostic tests 3: receiver operating characteristic plots. BMJ. 1994 Jul 16;309(6948):188. doi: 10.1136/bmj.309.6948.188.
- 19.↵CopelandKT, CheckowayH, McMichaelAJ, HolbrookRH. Bias due to misclassification in the estimation of relative risk. Am J Epidemiol. 1977 May;105(5):488-495. doi: 10.1093/oxfordjournals.aje.a112408.
- 20.LylesRH, TangL, SuperakHM, Validation data-based adjustment for outcome misclassification in logistic regression: an illustration. Epidemiology. 2011 Jul;22(4):589-597. doi: 10.1097/EDE.0b013e3182117c85.
- 21.↵Romero-BrufauS, HuddlestonJM, EscobarGJ, LiebowM. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015 Aug 13;19:285. doi: 10.1186/s13054-015-0999-1.
- 22.↵BlandJM, AltmanDG. Statistics notes: bootstrap resampling methods. BMJ. 2015 Jun 2;350:h2622. doi: 10.1136/bmj.h2622.
- 23.↵SteyerbergEW, HarrellFEJr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016 Jan;69:245-247. doi: 10.1016/j.jclinepi.2015.04.005.
- 24.↵ColquhounD. An investigation of the false discovery rate and the misinterpretation of p-values. R Soc Open Sci. 2014 Nov 19;1(3):140216. doi: 10.1098/rsos.140216.
- 25.↵GlickmanME, RaoSR, SchultzMR. False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies. J Clin Epidemiol. 2014 Aug;67(8):850-857. doi: 10.1016/j.jclinepi.2014.03.012.
- 26.↵ChanKS, TanCK, FangCS, Readmission to the intensive care unit: an indicator that reflects the potential risks of morbidity and mortality of surgical patients in the intensive care unit. Surg Today. 2009;39(4):295-299. doi: 10.1007/s00595-008-3876-6.
- 27.PavoniV, GianeselloL, PaparellaL, BuoninsegniLT, MoriE, GoriG. Outcome and quality of life of elderly critically ill patients: an Italian prospective observational study. Arch Gerontol Geriatr. 2012 Mar-Apr;54(2):e193-198. doi: 10.1016/j.archger.2011.11.013.
- 28.AlbanRF, NisimAA, HoJ, NishiGK, ShabotMM. Readmission to surgical intensive care increases severity-adjusted patient mortality. J Trauma. 2006 May;60(5):1027-1031. doi: 10.1097/01.ta.0000218217.42861.b7.
- 29.KabenA, CorreaF, ReinhartK, SettmacherU, GummertJ, KalffR, Readmission to a surgical intensive care unit: incidence, outcome and risk factors. Crit Care. 2008;12(5):R123. doi: 10.1186/cc7023.
- 30.LissauerME, DiazJJ, NarayanM, ShahPK, HannaNN. Surgical intensive care unit admission variables predict subsequent readmission. Am Surg. 2013 Jun;79(6):583-588.
- 31.↵BrunelliA, FergusonMK, RoccoG, A scoring system predicting the risk for intensive care unit admission for complications after major lung resection: a multicenter analysis. Ann Thorac Surg. 2008 Jul;86(1):213-218. doi: 10.1016/j.athoracsur.2008.03.063.
- 32.↵CuthbertsonBH, BoroujerdiM, McKieL, AucottL, PrescottG. Can physiological variables and early warning scoring systems allow early recognition of the deteriorating surgical patient? Crit Care Med. 2007 Feb;35(2):402-409. doi: 10.1097/01.CCM.0000254826.10520.87.
- 33.↵LudikhuizeJ, Brunsveld-ReindersAH, et al; Cost and Outcomes of Medical Emergency Teams Study Group. Outcomes associated with the nationwide introduction of rapid response systems in the Netherlands. 2015 Dec;43(12):2544-2551. doi: 10.1097/CCM.0000000000001272.
- 34.↵LudikhuizeJ, de JongeE, GoossensA. Measuring adherence among nurses one year after training in applying the modified early warning score and situation-background-assessment-recommendation instruments. Resuscitation. 2011 Nov;82(11):1428-1433. doi: 10.1016/j.resuscitation.2011.05.026.
- 35.↵EusebiP. Diagnostic accuracy measures. Cerebrovasc Dis. 2013;36(4):267-272. doi: 10.1159/000353863.
- 36.↵KellettJ, WangF, WoodworthS, HuangW. Changes and their prognostic implications in the abbreviated VitalPAC early warning score (ViEWS) after admission to hospital of 18,827 surgical patients. Resuscitation. 2014 Apr;85(4):544-548. doi: 10.1016/j.resuscitation.2012.12.002.
- 37.↵EscobarGJ, LaGuardiaJC, TurkBJ, RaginsA, KipnisP, DraperD. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012 May-Jun;7(5):388-395. doi: 10.1002/jhm.1929.
- 38.↵SackettDL, DeeksJJ, AltmanDG. Down with odds ratios! BMJ Evid Based Med. 1996 Sept/Oct;1(6):164-146.
