Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology

Michael C. Ryan, Susan Ostmo, Karyn Jonas, Audina Berrocal, Kimberly Drenser, Jason Horowitz, Thomas C. Lee, Charles Simmons, Maria Ana Martinez-Castellanos, R. V Paul Chan, Michael Chiang

Research output: Contribution to journalArticle

26 Citations (Scopus)

Abstract

Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.

Original languageEnglish (US)
Pages (from-to)1902-1910
Number of pages9
JournalAMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium
Volume2014
StatePublished - 2014

Fingerprint

Telemedicine
Ophthalmology
Research
Reading
Clinical Studies
Information Systems
Physicians

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology. / Ryan, Michael C.; Ostmo, Susan; Jonas, Karyn; Berrocal, Audina; Drenser, Kimberly; Horowitz, Jason; Lee, Thomas C.; Simmons, Charles; Martinez-Castellanos, Maria Ana; Chan, R. V Paul; Chiang, Michael.

In: AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, Vol. 2014, 2014, p. 1902-1910.

Research output: Contribution to journalArticle

Ryan, MC, Ostmo, S, Jonas, K, Berrocal, A, Drenser, K, Horowitz, J, Lee, TC, Simmons, C, Martinez-Castellanos, MA, Chan, RVP & Chiang, M 2014, 'Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology', AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, vol. 2014, pp. 1902-1910.
Ryan, Michael C. ; Ostmo, Susan ; Jonas, Karyn ; Berrocal, Audina ; Drenser, Kimberly ; Horowitz, Jason ; Lee, Thomas C. ; Simmons, Charles ; Martinez-Castellanos, Maria Ana ; Chan, R. V Paul ; Chiang, Michael. / Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology. In: AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2014 ; Vol. 2014. pp. 1902-1910.
@article{d66aef434f9c4d708bed211f50301f1e,
title = "Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology",
abstract = "Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32{\%}] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13{\%} discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10{\%} discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.",
author = "Ryan, {Michael C.} and Susan Ostmo and Karyn Jonas and Audina Berrocal and Kimberly Drenser and Jason Horowitz and Lee, {Thomas C.} and Charles Simmons and Martinez-Castellanos, {Maria Ana} and Chan, {R. V Paul} and Michael Chiang",
year = "2014",
language = "English (US)",
volume = "2014",
pages = "1902--1910",
journal = "AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium",
issn = "1559-4076",
publisher = "American Medical Informatics Association",

}

TY - JOUR

T1 - Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology

AU - Ryan, Michael C.

AU - Ostmo, Susan

AU - Jonas, Karyn

AU - Berrocal, Audina

AU - Drenser, Kimberly

AU - Horowitz, Jason

AU - Lee, Thomas C.

AU - Simmons, Charles

AU - Martinez-Castellanos, Maria Ana

AU - Chan, R. V Paul

AU - Chiang, Michael

PY - 2014

Y1 - 2014

N2 - Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.

AB - Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.

UR - http://www.scopus.com/inward/record.url?scp=84927567328&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84927567328&partnerID=8YFLogxK

M3 - Article

C2 - 25954463

AN - SCOPUS:84927567328

VL - 2014

SP - 1902

EP - 1910

JO - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium

JF - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium

SN - 1559-4076

ER -