Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study

Adam Wright, Joan Ash, Skye Aaron, Angela Ai, Thu Trang T. Hickman, Jane Wiesen, William Galanter, Allison B. McCoy, Richard Schreiber, Christopher A. Longhurst, Dean F. Sittig

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Objective: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. Materials and methods: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. Results: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. Discussion: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. Conclusion: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

Original languageEnglish (US)
Pages (from-to)78-85
Number of pages8
JournalInternational Journal of Medical Informatics
Volume118
DOIs
StatePublished - Oct 1 2018

Fingerprint

Clinical Decision Support Systems
Delphi Technique
Practice Guidelines
Knowledge Management
Electronic Health Records
Research
Interviews
Delivery of Health Care

Keywords

  • Best practices
  • Clinical decision support
  • Electronic health records
  • Safety

ASJC Scopus subject areas

  • Health Informatics

Cite this

Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders : Results of a Delphi study. / Wright, Adam; Ash, Joan; Aaron, Skye; Ai, Angela; Hickman, Thu Trang T.; Wiesen, Jane; Galanter, William; McCoy, Allison B.; Schreiber, Richard; Longhurst, Christopher A.; Sittig, Dean F.

In: International Journal of Medical Informatics, Vol. 118, 01.10.2018, p. 78-85.

Research output: Contribution to journalArticle

Wright, Adam ; Ash, Joan ; Aaron, Skye ; Ai, Angela ; Hickman, Thu Trang T. ; Wiesen, Jane ; Galanter, William ; McCoy, Allison B. ; Schreiber, Richard ; Longhurst, Christopher A. ; Sittig, Dean F. / Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders : Results of a Delphi study. In: International Journal of Medical Informatics. 2018 ; Vol. 118. pp. 78-85.
@article{6c81b4230f6947f88e27a03ed488c456,
title = "Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study",
abstract = "Objective: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. Materials and methods: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. Results: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. Discussion: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. Conclusion: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.",
keywords = "Best practices, Clinical decision support, Electronic health records, Safety",
author = "Adam Wright and Joan Ash and Skye Aaron and Angela Ai and Hickman, {Thu Trang T.} and Jane Wiesen and William Galanter and McCoy, {Allison B.} and Richard Schreiber and Longhurst, {Christopher A.} and Sittig, {Dean F.}",
year = "2018",
month = "10",
day = "1",
doi = "10.1016/j.ijmedinf.2018.08.001",
language = "English (US)",
volume = "118",
pages = "78--85",
journal = "International Journal of Medical Informatics",
issn = "1386-5056",
publisher = "Elsevier Ireland Ltd",

}

TY - JOUR

T1 - Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders

T2 - Results of a Delphi study

AU - Wright, Adam

AU - Ash, Joan

AU - Aaron, Skye

AU - Ai, Angela

AU - Hickman, Thu Trang T.

AU - Wiesen, Jane

AU - Galanter, William

AU - McCoy, Allison B.

AU - Schreiber, Richard

AU - Longhurst, Christopher A.

AU - Sittig, Dean F.

PY - 2018/10/1

Y1 - 2018/10/1

N2 - Objective: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. Materials and methods: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. Results: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. Discussion: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. Conclusion: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

AB - Objective: Developing effective and reliable rule-based clinical decision support (CDS) alerts and reminders is challenging. Using a previously developed taxonomy for alert malfunctions, we identified best practices for developing, testing, implementing, and maintaining alerts and avoiding malfunctions. Materials and methods: We identified 72 initial practices from the literature, interviews with subject matter experts, and prior research. To refine, enrich, and prioritize the list of practices, we used the Delphi method with two rounds of consensus-building and refinement. We used a larger than normal panel of experts to include a wide representation of CDS subject matter experts from various disciplines. Results: 28 experts completed Round 1 and 25 completed Round 2. Round 1 narrowed the list to 47 best practices in 7 categories: knowledge management, designing and specifying, building, testing, deployment, monitoring and feedback, and people and governance. Round 2 developed consensus on the importance and feasibility of each best practice. Discussion: The Delphi panel identified a range of best practices that may help to improve implementation of rule-based CDS and avert malfunctions. Due to limitations on resources and personnel, not everyone can implement all best practices. The most robust processes require investing in a data warehouse. Experts also pointed to the issue of shared responsibility between the healthcare organization and the electronic health record vendor. Conclusion: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

KW - Best practices

KW - Clinical decision support

KW - Electronic health records

KW - Safety

UR - http://www.scopus.com/inward/record.url?scp=85051395128&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051395128&partnerID=8YFLogxK

U2 - 10.1016/j.ijmedinf.2018.08.001

DO - 10.1016/j.ijmedinf.2018.08.001

M3 - Article

C2 - 30153926

AN - SCOPUS:85051395128

VL - 118

SP - 78

EP - 85

JO - International Journal of Medical Informatics

JF - International Journal of Medical Informatics

SN - 1386-5056

ER -