Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology

Michael C. Ryan, Susan Ostmo, Karyn Jonas, Audina Berrocal, Kimberly Drenser, Jason Horowitz, Thomas C. Lee, Charles Simmons, Maria Ana Martinez-Castellanos, R. V.Paul Chan, Michael F. Chiang

Research output: Contribution to journalArticle

30 Scopus citations

Abstract

Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.

Original languageEnglish (US)
Pages (from-to)1902-1910
Number of pages9
JournalAMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium
Volume2014
StatePublished - Jan 1 2014

    Fingerprint

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Ryan, M. C., Ostmo, S., Jonas, K., Berrocal, A., Drenser, K., Horowitz, J., Lee, T. C., Simmons, C., Martinez-Castellanos, M. A., Chan, R. V. P., & Chiang, M. F. (2014). Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014, 1902-1910.