Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks

James M. Brown, John Campbell, Andrew Beers, Ken Chang, Susan Ostmo, R. V.Paul Chan, Jennifer Dy, Deniz Erdogmus, Stratis Ioannidis, Jayashree Kalpathy-Cramer, Michael Chiang

Research output: Contribution to journalArticle

53 Citations (Scopus)

Abstract

IMPORTANCE Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable. OBJECTIVE To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs. DESIGN, SETTING, AND PARTICIPANTS A deep convolutional neural networkwas trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017. EXPOSURES A deep learning algorithm trained on retinal photographs. MAIN OUTCOMES AND MEASURES Receiver operating characteristic analysiswas performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts. RESULTS Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre-plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93%with 94%specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts. CONCLUSIONS AND RELEVANCE This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

Original languageEnglish (US)
Pages (from-to)803-810
Number of pages8
JournalJAMA Ophthalmology
Volume136
Issue number7
DOIs
StatePublished - Jul 1 2018

Fingerprint

Retinopathy of Prematurity
Learning
ROC Curve
Retinal Diseases
Retinal Vessels
Informatics
Blindness

ASJC Scopus subject areas

  • Ophthalmology

Cite this

Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. / Brown, James M.; Campbell, John; Beers, Andrew; Chang, Ken; Ostmo, Susan; Chan, R. V.Paul; Dy, Jennifer; Erdogmus, Deniz; Ioannidis, Stratis; Kalpathy-Cramer, Jayashree; Chiang, Michael.

In: JAMA Ophthalmology, Vol. 136, No. 7, 01.07.2018, p. 803-810.

Research output: Contribution to journalArticle

Brown, JM, Campbell, J, Beers, A, Chang, K, Ostmo, S, Chan, RVP, Dy, J, Erdogmus, D, Ioannidis, S, Kalpathy-Cramer, J & Chiang, M 2018, 'Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks', JAMA Ophthalmology, vol. 136, no. 7, pp. 803-810. https://doi.org/10.1001/jamaophthalmol.2018.1934
Brown, James M. ; Campbell, John ; Beers, Andrew ; Chang, Ken ; Ostmo, Susan ; Chan, R. V.Paul ; Dy, Jennifer ; Erdogmus, Deniz ; Ioannidis, Stratis ; Kalpathy-Cramer, Jayashree ; Chiang, Michael. / Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. In: JAMA Ophthalmology. 2018 ; Vol. 136, No. 7. pp. 803-810.
@article{d9dd77473de74cf7a8907f9d8d354d07,
title = "Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks",
abstract = "IMPORTANCE Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable. OBJECTIVE To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs. DESIGN, SETTING, AND PARTICIPANTS A deep convolutional neural networkwas trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017. EXPOSURES A deep learning algorithm trained on retinal photographs. MAIN OUTCOMES AND MEASURES Receiver operating characteristic analysiswas performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts. RESULTS Of the 5511 included retinal photographs, 4535 (82.3{\%}) were graded as normal, 805 (14.6{\%}) as pre-plus disease, and 172 (3.1{\%}) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93{\%}with 94{\%}specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100{\%} and 94{\%}, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts. CONCLUSIONS AND RELEVANCE This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.",
author = "Brown, {James M.} and John Campbell and Andrew Beers and Ken Chang and Susan Ostmo and Chan, {R. V.Paul} and Jennifer Dy and Deniz Erdogmus and Stratis Ioannidis and Jayashree Kalpathy-Cramer and Michael Chiang",
year = "2018",
month = "7",
day = "1",
doi = "10.1001/jamaophthalmol.2018.1934",
language = "English (US)",
volume = "136",
pages = "803--810",
journal = "JAMA Ophthalmology",
issn = "2168-6165",
publisher = "American Medical Association",
number = "7",

}

TY - JOUR

T1 - Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks

AU - Brown, James M.

AU - Campbell, John

AU - Beers, Andrew

AU - Chang, Ken

AU - Ostmo, Susan

AU - Chan, R. V.Paul

AU - Dy, Jennifer

AU - Erdogmus, Deniz

AU - Ioannidis, Stratis

AU - Kalpathy-Cramer, Jayashree

AU - Chiang, Michael

PY - 2018/7/1

Y1 - 2018/7/1

N2 - IMPORTANCE Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable. OBJECTIVE To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs. DESIGN, SETTING, AND PARTICIPANTS A deep convolutional neural networkwas trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017. EXPOSURES A deep learning algorithm trained on retinal photographs. MAIN OUTCOMES AND MEASURES Receiver operating characteristic analysiswas performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts. RESULTS Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre-plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93%with 94%specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts. CONCLUSIONS AND RELEVANCE This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

AB - IMPORTANCE Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable. OBJECTIVE To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs. DESIGN, SETTING, AND PARTICIPANTS A deep convolutional neural networkwas trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017. EXPOSURES A deep learning algorithm trained on retinal photographs. MAIN OUTCOMES AND MEASURES Receiver operating characteristic analysiswas performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts. RESULTS Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre-plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93%with 94%specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts. CONCLUSIONS AND RELEVANCE This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

UR - http://www.scopus.com/inward/record.url?scp=85049693038&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049693038&partnerID=8YFLogxK

U2 - 10.1001/jamaophthalmol.2018.1934

DO - 10.1001/jamaophthalmol.2018.1934

M3 - Article

C2 - 29801159

AN - SCOPUS:85049693038

VL - 136

SP - 803

EP - 810

JO - JAMA Ophthalmology

JF - JAMA Ophthalmology

SN - 2168-6165

IS - 7

ER -