Review ArticleReviews and Commentaries

Clinical Decision Support Alert Appropriateness: A Review and Proposal for Improvement

Allison B. McCoy, Eric J. Thomas, Marie Krousel-Wood and Dean F. Sittig
Ochsner Journal June 2014, 14 (2) 195-202;

Abstract

Background Many healthcare providers are adopting clinical decision support (CDS) systems to improve patient safety and meet meaningful use requirements. Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drug-disease warnings or provide dosing guidance are most commonly implemented. Alert overrides, which occur when clinicians do not follow the guidance presented by the alert, can hinder improved patient outcomes.

Methods We present a review of CDS alerts and describe a proposal to develop novel methods for evaluating and improving CDS alerts that builds upon traditional informatics approaches. Our proposal incorporates previously described models for predicting alert overrides that utilize retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable.

Results Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Our proposal expands the use of web-based monitoring tools with an interactive dashboard for evaluating CDS alert and response appropriateness that incorporates the predictive models. The dashboard provides 2 views, an alert detail view and a patient detail view, to provide a full history of alerts and help put the patient's events in context.

Conclusion The proposed research introduces several innovations to address the challenges and gaps in alert evaluations. This research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety.

Keywords

INTRODUCTION

Many healthcare providers are adopting electronic health records (EHRs) that incorporate clinical decision support (CDS) to improve patient safety and meet Medicare and Medicaid Stage 1 meaningful use requirements.1,2 Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drug-disease warnings or provide dosing guidance are most common.3,4 Initial research reported that adverse drug events (ADEs) were potentially preventable by alerts and other CDS systems.5 Despite such promise, CDS implementations in diverse settings have not consistently improved patient outcomes.6-9 Alert overrides occur when clinicians do not follow the guidance presented by the alert. For example, an alert may appear when a clinician orders amoxicillin, warning that the patient is allergic to penicillin-class medications. The clinician may accept the alert and cancel the order. Or the clinician may override the alert and order amoxicillin, either because he or she failed to read the alert or because the benefit to the patient outweighs the risk. In most organizations, the majority of daily alerts displayed to providers during the ordering process are overridden, and such overrides may be a barrier to improved patient and process outcomes.10 Both justifiable and nonjustifiable overrides occur, and detailed evaluation of the alerts and provider responses is necessary to determine appropriateness.11,12 However, these evaluation methods are labor intensive and difficult to replicate for every alert implemented at individual institutions. More efficient approaches to effectively evaluate alert appropriateness are necessary for optimizing patient safety. This article reviews CDS alerts and proposes novel methods for evaluating and improving CDS alerts that build upon traditional informatics approaches.

CLINICAL DECISION SUPPORT ALERTS

Early reports of ADEs among hospitalized patients indicating that about 28% of ADEs were preventable have elicited substantial research into the use of CDS to prevent patient harm.5 Studies have since shown that medication errors, which occur in 4%-6% of orders, can be prevented by computerized provider order entry (CPOE) and CDS.5,13-18 In one study, the use of CPOE and CDS decreased the rate of medication errors by 81%.18 Although improved patient safety is a leading motivation for CDS adoption, financial incentives also exist, as CDS has reportedly contributed to substantial savings.19 Finally, Stage 1 of meaningful use requires institutions to implement drug-drug and drug-allergy interaction checks, implement 1 high-priority condition CDS rule, and track CDS compliance.1 Multiple CDS approaches exist, including alerts, simple guided-dosing algorithms, order sets, and complex ordering advisors.20,21 Alerts are implemented in 61%-78% of hospitals and included in all major commercial EHRs to notify clinicians of interactions, changing laboratory values, or other information.3,4,21,22 On average, as reported in 1 study, clinicians received 56 alerts per day and spent 49 minutes per day processing them, making the alerts a substantial component of the daily care workflow.23

EVALUATION OF CLINICAL DECISION SUPPORT ALERTS

Alert Overrides

Despite initial reports of CDS success, evaluations of CDS systems have not always demonstrated improved patient outcomes.6-9 Nonadherence to the alerts by clinicians, also referred to as alert overrides, occurs for 49%-96% of alerts and is a potential barrier to such success.10,24-33 Although the CDS system may be designed to improve patient safety, it cannot be effective if the alerts are poorly implemented or the clinicians do not change their behavior in response to relevant alerts. Excess alerts, those that are repeated (eg, for each refill of a long-term medication) or not relevant, cause alert fatigue and contribute to alert overrides.10 Studies examining overrides have used chart review or user feedback to conclude that many overrides are clinically justifiable because of the clinical irrelevance of an alert, known patient tolerance for a drug, or documented clinician intention to monitor the patient, indicating a need for institutions to evaluate alerts to prevent alert fatigue.24,25,27,29,32-36 Researchers have used statistical modeling to evaluate possible predictors of alert overrides, including human factors (eg, workflow integration, prioritization), patient and clinician characteristics, triggering substance, alert frequency, response type required, and perceived severity and value.37,38 Many of these factors significantly contributed to the alert acceptance rate in multivariable analysis, indicating that the modeling approach may be a viable alternative to extensive chart reviews. However, existing predictive models have not yet been shown to distinguish between inappropriate and justifiable overrides.

Alert and Response Appropriateness

Although alert overrides by providers have been the focus of many evaluations, some overrides are justifiable because of clinical irrelevance, patient tolerance, or the provider's documented intention to monitor the patient.11,12 Likewise, some alerts are inappropriate, and adhering to the alert advice could cause harm to the patient.12 Detailed evaluations of alert appropriateness are necessary to identify such undesirable, unintended consequences and to institute efforts to mitigate resulting errors.39-41 In the evaluation framework from Ong and Coiera,42 signal detection theory is applied, classifying alerts as hits, misses, false alarms, and true negatives. In another report, Ancker et al described “The Triangle Model,” emphasizing simultaneous, interconnected evaluation of the patient, technology, and organization in conjunction with evaluation of providers' interactions.11 A more relevant framework categorizes alerts as successes, justifiable overrides, provider nonadherence, and unintended adverse consequences through retrospective chart review based on alert and response appropriateness.12 This approach is advantageous because it accounts for inappropriate alerts that result in justifiable overrides (ie, the clinician correctly disregards the alert advice) or unintended adverse consequences (ie, the clinician follows the alert advice and potentially harms the patient). Although these evaluation methods are necessary for determining the true effectiveness of alerts, they are labor intensive and difficult to replicate for every alert implemented at individual institutions. More efficient, semiautomated evaluation approaches are necessary to understand alert responses and overrides and ultimately to improve patient safety.

Surveillance Tools for Alert Evaluation

To facilitate alert evaluations, institutions have implemented CDS surveillance systems. Zimmerman et al43 displayed retrospective CDS data in a spreadsheet-based dashboard, and Reynolds et al44 developed a web-based, graphic dashboard to allow monitoring of order and alert volume by patient location, prescriber type, and alert type. In previous work by McCoy et al, review by an alerts committee or physician-led informatics group provided opportunities to identify poorly performing alerts and make system improvements. A real-time surveillance dashboard displayed lists of patients receiving high-risk medications, CDS interactions, and detailed patient views to clinical pharmacists to augment decision making.45 The tool allowed informatics personnel to identify and correct inappropriate triggering criteria in existing alerts through aggregate evaluation of the appropriateness of responses by pharmacists during routine clinical duties.

These studies indicate that web-based surveillance tools can be increasingly useful in the evaluation and improvement of alerts. The surveillance tools may also be beneficial to clinicians, allowing them to review their alert and response histories and empowering them to change their behavior if necessary.

Methods for Improving Alerts

Several projects have attempted to improve CDS alerts and reduce override rates by turning off frequently overridden alerts.46-49 Duke and Bolchini50 developed a model for creating context-aware drug-drug interaction alerts that allowed tailoring alert displays based on relevant patient-specific information, resulting in improved acceptance of the alerts. However, alerts deemed inappropriate in some clinical scenarios (eg, increased international normalized ratio values following administration of warfarin, which may be acceptable for mechanical valve recipients) should also be suppressed. No consistent method exists to avoid false positive alerts (which divert clinician time and attention) and false negative alerts (which silently leave patients at risk) that is generalizable across systems and clinical domains.

A PROPOSAL TO EVALUATE AND IMPROVE THE APPROPRIATENESS OF ALERTS

Predicting Inappropriate Alerts and Responses

To better evaluate and improve CDS alert appropriateness, we first propose the use of the alert evaluation framework developed and evaluated in prior research that utilizes retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable.12 The framework classifies alert and clinician response appropriateness, identifying successes, justifiable overrides, provider nonadherence, and unintended consequences (Table 1). This approach aims first to identify predictors of alert and response inappropriateness to eliminate the need for manual reviews, and second to validate our findings in both ambulatory and community hospital settings.

View this table:
Table 1.

Alert Evaluation Framework

Multivariable binary logistic regression has been applied in several studies to evaluate the association between various clinician, patient, and alert characteristics and overrides.24,30,37,38,51,52 High-level characteristics frequently included as covariates in prior studies are listed in Table 2; other studies have evaluated provider-entered override reasons, but these explanations are not routinely collected across institutions. However, as demonstrated in the previously described evaluation framework, effective alert evaluations should assess alert and response appropriateness, not merely alert overrides. By identifying factors that predict inappropriate alerts and responses, informatics personnel can improve alert logic to account for these factors, increasing the specificity of the alerts. As a result of the improved specificity, clinicians may experience less alert fatigue, override fewer alerts, and provide better care for patients with conditions that warrant serious alerts.

View this table:
Table 2.

Multivariable Alert Evaluation Covariates

Through independent chart review by 3 different clinicians (eg, physicians, pharmacists) using explicit and implicit review criteria and assessing inter-rater reliability using Cohen's kappa statistic, we plan to develop a gold standard for the appropriateness of each alert and clinician response (Table 1).12 For each alert type (eg, drug-drug, drug-allergy), we then will assess its predictive power to identify inappropriate alerts and responses for each characteristic identified through literature review (Table 2), investigator experience, and collaboration with a human-factors expert. We will explore the use of different predictive models with variable selection, including multinomial logistic regression, using 10-fold cross-validation to split the data into training and test sets.

Novel Metrics for Predicting Inappropriate Alerts and Responses

Although variables traditionally included in the evaluation of alerts have been significantly associated with alert overrides, additional predictors of alert responses may improve the models. Substantial evidence demonstrates that integrating clinical context can increase alert appropriateness and improve alert acceptance.10,50 The first variable that we will incorporate into the models is the indication of an alerted medication, whether entered manually by the clinician during e-prescribing or inferred from a medication indication knowledge base developed in our prior work.53-55 The algorithms and back-end knowledge required to drive such integration or allow exceptions in simple rule-based logic are difficult to develop and maintain. We have previously explored and validated several complementary methods for developing this knowledge for use in patient summaries.53-57 Additional variables derived from these knowledge bases will be included in the predictive models to determine if additional data improve detection of inappropriate alerts.

Prior work also has described methods for determining the reputation of users generating content, most often in the setting of e-commerce ratings, in which the reputation is computed as the proportion of ratings from a specific user that are the same as ratings submitted by other users.58 In previous work, we developed a clinician reputation metric to evaluate crowdsourced knowledge about links between prescribed medications and indicated problems that we found to have a specificity of 99.5% and an improved sensitivity (66.3%) compared to alternative measures.59 This method can be applied to alert override evaluations: a clinician's response to a specific alert is compared to other clinicians' responses to the same alert, given a similar patient scenario. By considering clinicians as users and alert responses as user-generated content, alert evaluators may adopt similar reputation metrics to identify inappropriate alerts that can be used in the previously developed predictive models.

Designing and Implementing an Interactive Alert Evaluation Dashboard

During previous research, we developed a condition-specific, web-based surveillance tool that allowed clinical pharmacists, informatics personnel, and clinicians to review CDS alert responses in the context of patients at high risk for ADEs.45,60 Figure 1 depicts the surveillance workflow that is designed to improve patient safety. Although a randomized, controlled trial in which clinical pharmacists used the tool did not reduce ADEs in patients with acute kidney injury, the technology assisted informatics personnel in refining logic to improve the specificity of the CDS alerts.45

Figure 1.
Figure 1.

Surveillance workflow for improving patient safety. CPOE, computerized provider order entry; EHR, electronic health record.

We propose to develop and implement InSPECt (Interactive Surveillance Portal for Evaluating Clinical decision support), an open-source, EHR-independent dashboard that will incorporate the medication indication and reputation metrics developed in the first phase of the project and will permit further assessment of the use of surveillance in evaluating CDS implementations. InSPECt will consist of 2 view types: the alert detail and the patient detail. The alert detail view displays all logged alert instances and allows reviewers to identify inappropriate alerts at risk of harm, showing details such as alert time; triggering medication(s), laboratory value, or allergy; patient demographics; and clinician name and service. The display can be filtered or sorted on any column. This view will also display a graph of alert rates over time, including total and overridden alerts, and will report the estimated rate of inappropriate alerts using the metrics developed in the first phase of the project. Also within the alert detail view, users will be able to select alternate triggering criteria for the alerts, resulting in updated estimates for alert display, override, appropriateness, and response appropriateness rates. Figure 2 depicts a mockup of the alert detail view.

Figure 2.
Figure 2.

Alert detail view mockup.

A mockup of the patient detail view is shown in Figure 3. This view displays a graph of events, such as relevant laboratory values and medications, and a detailed timeline for a patient in reverse chronological order to provide context for the alerts. The timeline will include all orders, problems, laboratory results, and alert interactions documented in the patient's EHR and can be sorted on any column. Reviewers can use the patient detail view to understand clinician actions and patient condition changes occurring in conjunction with alert overrides without having to search a patient's EHR independently.

Figure 3.
Figure 3.

Patient detail view mockup.

After development and validation of InSPECt are complete, we will collaborate with CDS managers and clinicians at study sites to review alerts. We will take advantage of the InSPECt interactivity to identify poorly performing alerts and evaluate alternate-triggering criteria that may improve the rate of appropriateness and potentially reduce the rates of overrides and inappropriate responses. We will then work with other information technology staff, clinician leaders, and informatics investigators to design an intervention to improve the alerts and evaluate the effect of the improved alerts on patient, provider, and process outcomes.

CONCLUSION

Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Prior work is also limited by evaluations from single institutions with locally developed systems that restrict generalizability. Our proposed research introduces several innovations to address the challenges and gaps in alert evaluations; it builds upon the alert appropriateness framework developed previously, adopting predictive models and introducing metrics novel to the biomedical informatics domain that have proven successful in other domains. Expanding prior surveillance methods, we also aim to develop an EHR-independent application that is deployable by any institution using open-source, readily available technologies, validating the results in both ambulatory and community hospital settings utilizing commercial EHRs. Combined, this research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety.

This article meets the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties Maintenance of Certification competencies for Patient Care, Medical Knowledge, and Systems-Based Practice.

ACKNOWLEDGMENT

This project was supported in part by NLM Grant 1K22LM011430-01A1, a UTHealth Young Clinical and Translational Sciences Investigator Award (KL2 TR 000370-06A1), Contract No. 10510592 for Patient-Centered Cognitive Support under the Strategic Health IT Advanced Research Projects Program (SHARP) from the Office of the National Coordinator for Health Information Technology, and NCRR Grant 3UL1RR024148.

Footnotes

  • The authors have no financial or proprietary interest in the subject matter of this article.

REFERENCES

  1. ↵
    1. Blumenthal D,
    2. Tavenner M
    (8 5, 2010) The “meaningful use” regulation for electronic health records. N Engl J Med 363(6):501–504, pmid:20647183, Epub 2010 Jul 13.
    OpenUrlCrossRefPubMed
  2. ↵
    1. Bates DW,
    2. O'Neil AC,
    3. Boyle D,
    4. et al.
    (Sep-Oct 1994) Potential identifiability and preventability of adverse events using information systems. J Am Med Inform Assoc 1(5):404–411, pmid:7850564.
    OpenUrlCrossRefPubMed
  3. ↵
    1. Jha AK,
    2. DesRoches CM,
    3. Campbell EG,
    4. et al.
    (4 16, 2009) Use of electronic health records in U.S. hospitals. N Engl J Med 360(16):1628–1638, pmid:19321858, Epub 2009 Mar 25.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Kuperman GJ,
    2. Bobb A,
    3. Payne TH,
    4. et al.
    (Jan-Feb 2007) Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 14(1):29–40, pmid:17068355, Epub 2006 Oct 26.
    OpenUrlCrossRefPubMed
  5. ↵
    1. Bates DW,
    2. Cullen DJ,
    3. Laird N,
    4. et al.
    (7 5, 1995) Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 274(1):29–34, pmid:7791255.
    OpenUrlCrossRefPubMed
  6. ↵
    1. Gurwitz JH,
    2. Field TS,
    3. Rochon P,
    4. et al.
    (12, 2008) Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc 56(12):2225–2233, pmid:19093922.
    OpenUrlCrossRefPubMed
    1. Strom BL,
    2. Schinnar R,
    3. Bilker W,
    4. Hennessy S,
    5. Leonard CE,
    6. Pifer E
    (Jul-Aug 2010) Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co-prescribing as a test case. J Am Med Inform Assoc 17(4):411–415, pmid:20595308.
    OpenUrlCrossRefPubMed
    1. Tamblyn R,
    2. Reidel K,
    3. Huang A,
    4. et al.
    (Mar-Apr 2010) Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial. Med Decis Making 30(2):176–188, pmid:19675319, Epub 2009 Aug 12.
    OpenUrlCrossRefPubMed
  9. ↵
    1. Strom BL,
    2. Schinnar R,
    3. Aberra F,
    4. et al.
    (9 27, 2010) Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 170(17):1578–1583, pmid:20876410.
    OpenUrlCrossRefPubMed
  10. ↵
    1. van der Sijs H,
    2. Aarts J,
    3. Vulto A,
    4. Berg M
    (Mar-Apr 2006) Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 13(2):138–147, pmid:16357358, Epub 2005 Dec 15.
    OpenUrlCrossRefPubMed
  11. ↵
    1. Ancker JS,
    2. Kern LM,
    3. Abramson E,
    4. Kaushal R
    (Jan-Feb 2012) The Triangle Model for evaluating the effect of health information technology on healthcare quality and safety. J Am Med Inform Assoc 19(1):61–65, pmid:21857023, Epub 2011 Aug 20.
    OpenUrlCrossRefPubMed
  12. ↵
    1. McCoy AB,
    2. Waitman LR,
    3. Lewis JB,
    4. et al.
    (May-Jun 2012) A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 19(3):346–352, pmid:21849334, Epub 2011 Aug 17.
    OpenUrlCrossRefPubMed
  13. ↵
    1. Lesar TS,
    2. Briceland L,
    3. Stein DS
    (1 22-29, 1997) Factors related to errors in medication prescribing. JAMA 277(4):312–317, pmid:9002494.
    OpenUrlCrossRefPubMed
    1. Bobb A,
    2. Gleason K,
    3. Husch M,
    4. Feinglass J,
    5. Yarnold PR,
    6. Noskin GA
    (4 12, 2004) The epidemiology of prescribing errors: the potential impact of computerized prescriber order entry. Arch Intern Med 164(7):785–792, pmid:15078649.
    OpenUrlCrossRefPubMed
    1. Reckmann MH,
    2. Westbrook JI,
    3. Koh Y,
    4. Lo C,
    5. Day RO
    (Sep-Oct 2009) Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc 16(5):613–623, pmid:19567798, Epub 2009 Jun 30.
    OpenUrlCrossRefPubMed
    1. Ammenwerth E,
    2. Schnell-Inderst P,
    3. Machan C,
    4. Siebert U
    (Sep-Oct 2008) The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 15(5):585–600, pmid:18579832, Epub 2008 Jun 25.
    OpenUrlPubMed
    1. van Rosse F,
    2. Maat B,
    3. Rademaker CM,
    4. van Vught AJ,
    5. Egberts AC,
    6. Bollen CW
    (4, 2009) The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 123(4):1184–1190, pmid:19336379.
    OpenUrlAbstract/FREE Full Text
  18. ↵
    1. Bates DW,
    2. Teich JM,
    3. Lee J,
    4. et al.
    (Jul-Aug 1999) The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 6(4):313–321, pmid:10428004.
    OpenUrlCrossRefPubMed
  19. ↵
    1. Kaushal R,
    2. Jha AK,
    3. Franz C,
    4. et al.
    (May-Jun 2006) Brigham and Women's Hospital CPOE Working Group. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc 13(3):261–266, pmid:16501178, Epub 2006 Feb 24.
    OpenUrlCrossRefPubMed
  20. ↵
    1. Miller RA,
    2. Waitman LR,
    3. Chen S,
    4. Rosenbloom ST
    (12, 2005) The anatomy of decision support during inpatient care provider order entry (CPOE): empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform 38(6):469–485, pmid:16290243, Epub 2005 Oct 21.
    OpenUrlCrossRefPubMed
  21. ↵
    1. Wright A,
    2. Sittig DF,
    3. Ash JS,
    4. et al.
    (5 1, 2011) Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 18(3):232–242, pmid:21415065, Epub 2011 Mar 17.
    OpenUrlCrossRefPubMed
  22. ↵
    1. Wright A,
    2. Sittig DF,
    3. Ash JS,
    4. Sharma S,
    5. Pang JE,
    6. Middleton B
    (Sep-Oct 2009) Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc 16(5):637–644, pmid:19567796, Epub 2009 Jun 30.
    OpenUrlCrossRefPubMed
  23. ↵
    1. Murphy DR,
    2. Reis B,
    3. Sittig DF,
    4. Singh H
    (2, 2012) Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 125(2):209.e1–e7, pmid:22269625.
    OpenUrlCrossRefPubMed
  24. ↵
    1. Weingart SN,
    2. Toth M,
    3. Sands DZ,
    4. Aronson MD,
    5. Davis RB,
    6. Phillips RS
    (11 24, 2003) Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 163(21):2625–2631, pmid:14638563.
    OpenUrlCrossRefPubMed
  25. ↵
    1. Hsieh TC,
    2. Kuperman GJ,
    3. Jaggi T,
    4. et al.
    (Nov-Dec 2004) Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 11(6):482–491, pmid:15298998, Epub 2004 Aug 6.
    OpenUrlPubMed
    1. Persell SD,
    2. Dolan NC,
    3. Friesema EM,
    4. Thompson JA,
    5. Kaiser D,
    6. Baker DW
    (2 16, 2010) Frequency of inappropriate medical exceptions to quality measures. Ann Intern Med 152(4):225–231, pmid:20157137.
    OpenUrlCrossRefPubMed
  27. ↵
    1. Grizzle AJ,
    2. Mahmood MH,
    3. Ko Y,
    4. et al.
    (10, 2007) Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 13(10):573–578, pmid:17927462.
    OpenUrlPubMed
    1. van der Sijs H,
    2. Mulder A,
    3. van Gelder T,
    4. Aarts J,
    5. Berg M,
    6. Vulto A
    (10, 2009) Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 18(10):941–947, pmid:19579216.
    OpenUrlCrossRefPubMed
  29. ↵
    1. Ko Y,
    2. Abarca J,
    3. Malone DC,
    4. et al.
    (Jan-Feb 2007) Practitioners' views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 14(1):56–64, pmid:17068346, Epub 2006 Oct 26.
    OpenUrlCrossRefPubMed
  30. ↵
    1. Isaac T,
    2. Weissman JS,
    3. Davis RB,
    4. et al.
    (2 9, 2009) Overrides of medication alerts in ambulatory care. Arch Intern Med 169(3):305–311, pmid:19204222.
    OpenUrlCrossRefPubMed
    1. Shah NR,
    2. Seger AC,
    3. Seger DL,
    4. et al.
    (Jan-Feb 2006) Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 13(1):5–11, pmid:16221941, Epub 2005 Oct 12.
    OpenUrlPubMed
  32. ↵
    1. Swiderski SM,
    2. Pedersen CA,
    3. Schneider PJ,
    4. Miller AS
    (6, 2007) A study of the frequency and rationale for overriding allergy warnings in a computerized prescriber order entry system. J Patient Saf 3(2):91–96.
    OpenUrlCrossRef
  33. ↵
    1. Litzelman DK,
    2. Tierney WM
    (8, 1996) Physicians' reasons for failing to comply with computerized preventive care guidelines. J Gen Intern Med 11(8):497–499, pmid:8872790.
    OpenUrlCrossRefPubMed
    1. van der Sijs H,
    2. van Gelder T,
    3. Vulto A,
    4. Berg M,
    5. Aarts J
    (5, 2010) Understanding handling of drug safety alerts: a simulation study. Int J Med Inform 79(5):361–369, pmid:20171929, Epub 2010 Feb 19.
    OpenUrlCrossRefPubMed
    1. Sittig DF,
    2. Krall MA,
    3. Dykstra RH,
    4. Russell A,
    5. Chin HL
    (2 1, 2006) A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Inform Decis Mak 6:6, pmid:16451720.
    OpenUrlCrossRefPubMed
  36. ↵
    1. Russ AL,
    2. Zillich AJ,
    3. McManus MS,
    4. Doebbeling BN,
    5. Saleem JJ
    (4, 2012) Prescribers' interactions with medication alerts at the point of prescribing: A multi-method, in situ investigation of the human-computer interaction. Int J Med Inform 81(4):232–243, pmid:22296761, Epub 2012 Jan 31.
    OpenUrlCrossRefPubMed
  37. ↵
    1. Seidling HM,
    2. Phansalkar S,
    3. Seger DL,
    4. et al.
    (Jul-Aug 2011) Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 18(4):479–484, pmid:21571746, Epub 2011 May 12.
    OpenUrlCrossRefPubMed
  38. ↵
    1. Weingart SN,
    2. Seger AC,
    3. Feola N,
    4. Heffernan J,
    5. Schiff G,
    6. Isaac T
    (7 1, 2011) Electronic drug interaction alerts in ambulatory care: the value and acceptance of high-value alerts in US medical practices as assessed by an expert clinical panel. Drug Saf 34(7):587–593, pmid:21663334.
    OpenUrlCrossRefPubMed
  39. ↵
    1. Ash JS,
    2. Sittig DF,
    3. Campbell EM,
    4. Guappone KP,
    5. Dykstra RH
    (10, 2007) Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 11:26–30, pmid:18693791.
    OpenUrlPubMed
    1. Koppel R,
    2. Metlay JP,
    3. Cohen A,
    4. et al.
    (3 9, 2005) Role of computerized physician order entry systems in facilitating medication errors. JAMA 293(10):1197–1203, pmid:15755942.
    OpenUrlCrossRefPubMed
  41. ↵
    1. Bloomrosen M,
    2. Starren J,
    3. Lorenzi NM,
    4. Ash JS,
    5. Patel VL,
    6. Shortliffe EH
    (Jan-Feb 2011) Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 18(1):82–90, pmid:21169620.
    OpenUrlCrossRefPubMed
  42. ↵
    1. Ong MS,
    2. Coiera E
    (2011) Evaluating the effectiveness of clinical alerts: a signal detection approach. AMIA Annu Symp Proc. 2011; 1036–1044, Epub 2011 Oct 22.
  43. ↵
    1. Zimmerman CR,
    2. Jackson A,
    3. Chaffee B,
    4. O'Reilly M
    (10, 2007) A dashboard model for monitoring alert effectiveness and bandwidth. AMIA Annu Symp Proc 11:1176, pmid:18694272.
    OpenUrlPubMed
  44. ↵
    1. Reynolds G,
    2. Boyer D,
    3. Mackey K,
    4. Povondra L,
    5. Cummings A
    (11, 2008) Alerting strategies in computerized physician order entry: a novel use of a dashboard-style analytics tool in a children's hospital. AMIA Annu Symp Proc 6:1108, pmid:18999063.
    OpenUrlPubMed
  45. ↵
    1. McCoy AB,
    2. Cox ZL,
    3. Neal EB,
    4. et al.
    (1 1, 2012) Real-time pharmacy surveillance and clinical decision support to reduce adverse drug events in acute kidney injury: a randomized, controlled trial. Appl Clin Inform 3(2):221–238, pmid:22719796, Epub 2012 Jun 13.
    OpenUrlCrossRefPubMed
  46. ↵
    1. van der Sijs H,
    2. Aarts J,
    3. van Gelder T,
    4. Berg M,
    5. Vulto A
    (Jul-Aug 2008) Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 15(4):439–448, pmid:18436915, Epub 2008 Apr 24.
    OpenUrlCrossRefPubMed
    1. van der Sijs H,
    2. Kowlesar R,
    3. Aarts J,
    4. Berg M,
    5. Vulto A,
    6. van Gelder T
    (9, 2009) Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 65(9):919–925, pmid:19415251, Epub 2009 May 5.
    OpenUrlCrossRefPubMed
    1. Beccaro MA,
    2. Villanueva R,
    3. Knudson KM,
    4. Harvey EM,
    5. Langle JM,
    6. Paul W
    (9 29, 2010) Decision Support Alerts for Medication Ordering in a Computerized Provider Order Entry (CPOE) System: A systematic approach to decrease alerts. Appl Clin Inform 1(3):346–362, pmid:23616845.
    OpenUrlPubMed
  49. ↵
    1. Lee EK,
    2. Mejia AF,
    3. Senior T,
    4. Jose J
    (11 13, 2010) Improving patient safety through medical alert management: An automated decision tool to reduce alert fatigue. AMIA Annu Symp Proc 2010:417–421, pmid:21347012.
    OpenUrlPubMed
  50. ↵
    1. Duke JD,
    2. Bolchini D
    (2011) A successful model and visual design for creating context-aware drug-drug interaction alerts. AMIA Annu Symp Proc. 2011; 339–348, Epub 2011 Oct 22.
  51. ↵
    1. Tamblyn R,
    2. Huang A,
    3. Taylor L,
    4. et al.
    (Jul-Aug 2008) A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc 15(4):430–438, pmid:18436904, Epub 2008 Apr 24.
    OpenUrlCrossRefPubMed
  52. ↵
    1. Galanter WL,
    2. Didomenico RJ,
    3. Polikaitis A
    (May-Jun 2005) A trial of automated decision support alerts for contraindicated medications using computerized physician order entry. J Am Med Inform Assoc 12(3):269–274, pmid:15684124, Epub 2005 Jan 31.
    OpenUrlPubMed
  53. ↵
    1. Wright A,
    2. Chen ES,
    3. Maloney FL
    (12, 2010) An automated technique for identifying associations between medications, laboratory results and problems. J Biomed Inform 43(6):891–901, pmid:20884377, Epub 2010 Sep 25.
    OpenUrlCrossRefPubMed
    1. McCoy AB,
    2. Wright A,
    3. Laxmisan A,
    4. Singh H,
    5. Sittig DF
    (2011) A prototype knowledge base and SMART app to facilitate organization of patient medications by clinical problems. AMIA Annu Symp Proc. 2011; 888–894, Epub 2011 Oct 22.
  55. ↵
    1. McCoy AB,
    2. Wright A,
    3. Laxmisan A,
    4. et al.
    (Sep-Oct 2012) Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications. J Am Med Inform Assoc 19(5):713–718, pmid:22582202, Epub 2012 May 12.
    OpenUrlCrossRefPubMed
    1. McCoy AB,
    2. Sittig DF,
    3. Wright A
    (2012) Comparison of Association Rule Mining and Crowdsourcing for Automated Generation of a Problem-Medication Knowledge Base. In: 2012 IEEE Second International Conference on Healthcare Informatics, Imaging and Systems Biology (HISB). Sept, 125.
  57. ↵
    1. Wright A,
    2. McCoy A,
    3. Henkin S,
    4. Flaherty M,
    5. Sittig D
    (3 6, 2013) Validation of an association rule mining-based method to infer associations between medications and problems. Appl Clin Inform 4(1):100–109, pmid:23650491.
    OpenUrlCrossRefPubMed
  58. ↵
    1. Miller N,
    2. Resnick P,
    3. Zeckhauser R
    (9, 2005) Eliciting informative feedback: The peer-prediction method. Manag Sci 51(9):1359–1373.
    OpenUrl
  59. ↵
    1. McCoy AB,
    2. Wright A,
    3. Rogith D,
    4. Fathiamini S,
    5. Ottenbacher AJ,
    6. Sittig DF
    (12 7, 2013) Development of a clinician reputation metric to identify appropriate problem-medication pairs in a crowdsourced knowledge base. J Biomed Inform, Epub ahead of print.
  60. ↵
    1. Waitman LR,
    2. Phillips IE,
    3. McCoy AB,
    4. et al.
    (7, 2011) Adopting real-time surveillance dashboards as a component of an enterprisewide medication safety strategy. Jt Comm J Qual Patient Saf 37(7):326–332, pmid:21819031.
    OpenUrlPubMed
Back to top

In this issue

Print
Download PDF
Email Article
Citation Tools

Jump to section

Keywords