Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study

Adam Wright, Joan S. Ash, Skye Aaron, Angela Ai, Thu Trang T. Hickman, Jane F. Wiesen, William Galanter, Allison B. McCoy, Richard Schreiber, Christopher A. Longhurst, Dean F. Sittig

Research output: Contribution to journalArticlepeer-review

16 Scopus citations


Objective: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. Materials and methods: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. Results: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. Discussion: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. Conclusion: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

Original languageEnglish (US)
Pages (from-to)78-85
Number of pages8
JournalInternational Journal of Medical Informatics
StatePublished - Oct 2018


  • Best practices
  • Clinical decision support
  • Electronic health records
  • Safety

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study'. Together they form a unique fingerprint.

Cite this