Clinical decisions support malfunctions in a commercial electronic health record

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Objectives: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. Methods: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. Results: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44. Discussion: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. Conclusion: CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.

Original languageEnglish (US)
Pages (from-to)910-923
Number of pages14
JournalApplied Clinical Informatics
Volume8
Issue number3
StatePublished - Sep 6 2017

Fingerprint

Clinical Decision Support Systems
Statistical process control
Electronic Health Records
Health
Knowledge management
Visualization
Knowledge Management
Data structures
Formularies

Keywords

  • Alerts
  • Clinical decision support
  • Electronic health record
  • Electronic medical record
  • Errors
  • Malfunction

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics
  • Health Information Management

Cite this

Clinical decisions support malfunctions in a commercial electronic health record. / Kassakian, Steven; Yackel, Thomas; Gorman, Paul; Dorr, David.

In: Applied Clinical Informatics, Vol. 8, No. 3, 06.09.2017, p. 910-923.

Research output: Contribution to journalArticle

@article{e65b2ad62f4f4003b1c3e73de849f8a8,
title = "Clinical decisions support malfunctions in a commercial electronic health record",
abstract = "Objectives: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. Methods: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. Results: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68{\%}) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95{\%}, positive predictive value of 29{\%} and F-measure 0.44. Discussion: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. Conclusion: CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.",
keywords = "Alerts, Clinical decision support, Electronic health record, Electronic medical record, Errors, Malfunction",
author = "Steven Kassakian and Thomas Yackel and Paul Gorman and David Dorr",
year = "2017",
month = "9",
day = "6",
language = "English (US)",
volume = "8",
pages = "910--923",
journal = "Applied Clinical Informatics",
issn = "1869-0327",
publisher = "Schattauer GmbH",
number = "3",

}

TY - JOUR

T1 - Clinical decisions support malfunctions in a commercial electronic health record

AU - Kassakian, Steven

AU - Yackel, Thomas

AU - Gorman, Paul

AU - Dorr, David

PY - 2017/9/6

Y1 - 2017/9/6

N2 - Objectives: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. Methods: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. Results: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44. Discussion: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. Conclusion: CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.

AB - Objectives: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. Methods: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. Results: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44. Discussion: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. Conclusion: CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.

KW - Alerts

KW - Clinical decision support

KW - Electronic health record

KW - Electronic medical record

KW - Errors

KW - Malfunction

UR - http://www.scopus.com/inward/record.url?scp=85029209458&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029209458&partnerID=8YFLogxK

M3 - Article

C2 - 28880046

AN - SCOPUS:85029209458

VL - 8

SP - 910

EP - 923

JO - Applied Clinical Informatics

JF - Applied Clinical Informatics

SN - 1869-0327

IS - 3

ER -