Clinical decision support alert malfunctions: Analysis and empirically derived taxonomy

Adam Wright, Angela Ai, Joan Ash, Jane FWiesen, Thu Trang T. Hickman, Skye Aaron, Dustin McEvoy, Shane Borkowsky, Pavithra I. Dissanayake, Peter Embi, William Galanter, Jeremy Harper, Steven Kassakian, Rachel Ramoni, Richard Schreiber, Anwar Sirajuddin, David WBates, Dean F. Sittig

Research output: Contribution to journalArticle

18 Citations (Scopus)

Abstract

Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Original languageEnglish (US)
Pages (from-to)496-506
Number of pages11
JournalJournal of the American Medical Informatics Association
Volume25
Issue number5
DOIs
StatePublished - Jan 1 2018

Fingerprint

Clinical Decision Support Systems
Medical Informatics
Electronic Health Records

Keywords

  • Anomaly detection
  • Clinical decision support
  • Electronic health records
  • Safety

ASJC Scopus subject areas

  • Health Informatics

Cite this

Clinical decision support alert malfunctions : Analysis and empirically derived taxonomy. / Wright, Adam; Ai, Angela; Ash, Joan; FWiesen, Jane; Hickman, Thu Trang T.; Aaron, Skye; McEvoy, Dustin; Borkowsky, Shane; Dissanayake, Pavithra I.; Embi, Peter; Galanter, William; Harper, Jeremy; Kassakian, Steven; Ramoni, Rachel; Schreiber, Richard; Sirajuddin, Anwar; WBates, David; Sittig, Dean F.

In: Journal of the American Medical Informatics Association, Vol. 25, No. 5, 01.01.2018, p. 496-506.

Research output: Contribution to journalArticle

Wright, A, Ai, A, Ash, J, FWiesen, J, Hickman, TTT, Aaron, S, McEvoy, D, Borkowsky, S, Dissanayake, PI, Embi, P, Galanter, W, Harper, J, Kassakian, S, Ramoni, R, Schreiber, R, Sirajuddin, A, WBates, D & Sittig, DF 2018, 'Clinical decision support alert malfunctions: Analysis and empirically derived taxonomy', Journal of the American Medical Informatics Association, vol. 25, no. 5, pp. 496-506. https://doi.org/10.1093/JAMIA/OCX106
Wright, Adam ; Ai, Angela ; Ash, Joan ; FWiesen, Jane ; Hickman, Thu Trang T. ; Aaron, Skye ; McEvoy, Dustin ; Borkowsky, Shane ; Dissanayake, Pavithra I. ; Embi, Peter ; Galanter, William ; Harper, Jeremy ; Kassakian, Steven ; Ramoni, Rachel ; Schreiber, Richard ; Sirajuddin, Anwar ; WBates, David ; Sittig, Dean F. / Clinical decision support alert malfunctions : Analysis and empirically derived taxonomy. In: Journal of the American Medical Informatics Association. 2018 ; Vol. 25, No. 5. pp. 496-506.
@article{643704cf07924eca8fff3ea9d068d345,
title = "Clinical decision support alert malfunctions: Analysis and empirically derived taxonomy",
abstract = "Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.",
keywords = "Anomaly detection, Clinical decision support, Electronic health records, Safety",
author = "Adam Wright and Angela Ai and Joan Ash and Jane FWiesen and Hickman, {Thu Trang T.} and Skye Aaron and Dustin McEvoy and Shane Borkowsky and Dissanayake, {Pavithra I.} and Peter Embi and William Galanter and Jeremy Harper and Steven Kassakian and Rachel Ramoni and Richard Schreiber and Anwar Sirajuddin and David WBates and Sittig, {Dean F.}",
year = "2018",
month = "1",
day = "1",
doi = "10.1093/JAMIA/OCX106",
language = "English (US)",
volume = "25",
pages = "496--506",
journal = "Journal of the American Medical Informatics Association",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "5",

}

TY - JOUR

T1 - Clinical decision support alert malfunctions

T2 - Analysis and empirically derived taxonomy

AU - Wright, Adam

AU - Ai, Angela

AU - Ash, Joan

AU - FWiesen, Jane

AU - Hickman, Thu Trang T.

AU - Aaron, Skye

AU - McEvoy, Dustin

AU - Borkowsky, Shane

AU - Dissanayake, Pavithra I.

AU - Embi, Peter

AU - Galanter, William

AU - Harper, Jeremy

AU - Kassakian, Steven

AU - Ramoni, Rachel

AU - Schreiber, Richard

AU - Sirajuddin, Anwar

AU - WBates, David

AU - Sittig, Dean F.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

AB - Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

KW - Anomaly detection

KW - Clinical decision support

KW - Electronic health records

KW - Safety

UR - http://www.scopus.com/inward/record.url?scp=85042698794&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85042698794&partnerID=8YFLogxK

U2 - 10.1093/JAMIA/OCX106

DO - 10.1093/JAMIA/OCX106

M3 - Article

AN - SCOPUS:85042698794

VL - 25

SP - 496

EP - 506

JO - Journal of the American Medical Informatics Association

JF - Journal of the American Medical Informatics Association

SN - 1067-5027

IS - 5

ER -