Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity

Travis K. Redd, John Campbell, James M. Brown, Sang Jin Kim, Susan Ostmo, Robison Vernon Paul Chan, Jennifer Dy, Deniz Erdogmus, Stratis Ioannidis, Jayashree Kalpathy-Cramer, Michael Chiang

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Background: Prior work has demonstrated the near-perfect accuracy of a deep learning retinal image analysis system for diagnosing plus disease in retinopathy of prematurity (ROP). Here we assess the screening potential of this scoring system by determining its ability to detect all components of ROP diagnosis. Methods: Clinical examination and fundus photography were performed at seven participating centres. A deep learning system was trained to detect plus disease, generating a quantitative assessment of retinal vascular abnormality (the i-ROP plus score) on a 1-9 scale. Overall ROP disease category was established using a consensus reference standard diagnosis combining clinical and image-based diagnosis. Experts then ranked ordered a second data set of 100 posterior images according to overall ROP severity. Results: 4861 examinations from 870 infants were analysed. 155 examinations (3%) had a reference standard diagnosis of type 1 ROP. The i-ROP deep learning (DL) vascular severity score had an area under the receiver operating curve of 0.960 for detecting type 1 ROP. Establishing a threshold i-ROP DL score of 3 conferred 94% sensitivity, 79% specificity, 13% positive predictive value and 99.7% negative predictive value for type 1 ROP. There was strong correlation between expert rank ordering of overall ROP severity and the i-ROP DL vascular severity score (Spearman correlation coefficient=0.93; p<0.0001). Conclusion: The i-ROP DL system accurately identifies diagnostic categories and overall disease severity in an automated fashion, after being trained only on posterior pole vascular morphology. These data provide proof of concept that a deep learning screening platform could improve objectivity of ROP diagnosis and accessibility of screening.

Original languageEnglish (US)
JournalBritish Journal of Ophthalmology
DOIs
StateAccepted/In press - Jan 1 2018

Fingerprint

Retinopathy of Prematurity
Learning
Blood Vessels
Retinal Vessels
Aptitude
Photography

Keywords

  • child health (paediatrics)
  • public health
  • retina
  • telemedicine

ASJC Scopus subject areas

  • Ophthalmology
  • Sensory Systems
  • Cellular and Molecular Neuroscience

Cite this

Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity. / Redd, Travis K.; Campbell, John; Brown, James M.; Kim, Sang Jin; Ostmo, Susan; Chan, Robison Vernon Paul; Dy, Jennifer; Erdogmus, Deniz; Ioannidis, Stratis; Kalpathy-Cramer, Jayashree; Chiang, Michael.

In: British Journal of Ophthalmology, 01.01.2018.

Research output: Contribution to journalArticle

Redd, Travis K. ; Campbell, John ; Brown, James M. ; Kim, Sang Jin ; Ostmo, Susan ; Chan, Robison Vernon Paul ; Dy, Jennifer ; Erdogmus, Deniz ; Ioannidis, Stratis ; Kalpathy-Cramer, Jayashree ; Chiang, Michael. / Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity. In: British Journal of Ophthalmology. 2018.
@article{6124650b36804a398998735436327bf7,
title = "Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity",
abstract = "Background: Prior work has demonstrated the near-perfect accuracy of a deep learning retinal image analysis system for diagnosing plus disease in retinopathy of prematurity (ROP). Here we assess the screening potential of this scoring system by determining its ability to detect all components of ROP diagnosis. Methods: Clinical examination and fundus photography were performed at seven participating centres. A deep learning system was trained to detect plus disease, generating a quantitative assessment of retinal vascular abnormality (the i-ROP plus score) on a 1-9 scale. Overall ROP disease category was established using a consensus reference standard diagnosis combining clinical and image-based diagnosis. Experts then ranked ordered a second data set of 100 posterior images according to overall ROP severity. Results: 4861 examinations from 870 infants were analysed. 155 examinations (3{\%}) had a reference standard diagnosis of type 1 ROP. The i-ROP deep learning (DL) vascular severity score had an area under the receiver operating curve of 0.960 for detecting type 1 ROP. Establishing a threshold i-ROP DL score of 3 conferred 94{\%} sensitivity, 79{\%} specificity, 13{\%} positive predictive value and 99.7{\%} negative predictive value for type 1 ROP. There was strong correlation between expert rank ordering of overall ROP severity and the i-ROP DL vascular severity score (Spearman correlation coefficient=0.93; p<0.0001). Conclusion: The i-ROP DL system accurately identifies diagnostic categories and overall disease severity in an automated fashion, after being trained only on posterior pole vascular morphology. These data provide proof of concept that a deep learning screening platform could improve objectivity of ROP diagnosis and accessibility of screening.",
keywords = "child health (paediatrics), public health, retina, telemedicine",
author = "Redd, {Travis K.} and John Campbell and Brown, {James M.} and Kim, {Sang Jin} and Susan Ostmo and Chan, {Robison Vernon Paul} and Jennifer Dy and Deniz Erdogmus and Stratis Ioannidis and Jayashree Kalpathy-Cramer and Michael Chiang",
year = "2018",
month = "1",
day = "1",
doi = "10.1136/bjophthalmol-2018-313156",
language = "English (US)",
journal = "British Journal of Ophthalmology",
issn = "0007-1161",
publisher = "BMJ Publishing Group",

}

TY - JOUR

T1 - Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity

AU - Redd, Travis K.

AU - Campbell, John

AU - Brown, James M.

AU - Kim, Sang Jin

AU - Ostmo, Susan

AU - Chan, Robison Vernon Paul

AU - Dy, Jennifer

AU - Erdogmus, Deniz

AU - Ioannidis, Stratis

AU - Kalpathy-Cramer, Jayashree

AU - Chiang, Michael

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Background: Prior work has demonstrated the near-perfect accuracy of a deep learning retinal image analysis system for diagnosing plus disease in retinopathy of prematurity (ROP). Here we assess the screening potential of this scoring system by determining its ability to detect all components of ROP diagnosis. Methods: Clinical examination and fundus photography were performed at seven participating centres. A deep learning system was trained to detect plus disease, generating a quantitative assessment of retinal vascular abnormality (the i-ROP plus score) on a 1-9 scale. Overall ROP disease category was established using a consensus reference standard diagnosis combining clinical and image-based diagnosis. Experts then ranked ordered a second data set of 100 posterior images according to overall ROP severity. Results: 4861 examinations from 870 infants were analysed. 155 examinations (3%) had a reference standard diagnosis of type 1 ROP. The i-ROP deep learning (DL) vascular severity score had an area under the receiver operating curve of 0.960 for detecting type 1 ROP. Establishing a threshold i-ROP DL score of 3 conferred 94% sensitivity, 79% specificity, 13% positive predictive value and 99.7% negative predictive value for type 1 ROP. There was strong correlation between expert rank ordering of overall ROP severity and the i-ROP DL vascular severity score (Spearman correlation coefficient=0.93; p<0.0001). Conclusion: The i-ROP DL system accurately identifies diagnostic categories and overall disease severity in an automated fashion, after being trained only on posterior pole vascular morphology. These data provide proof of concept that a deep learning screening platform could improve objectivity of ROP diagnosis and accessibility of screening.

AB - Background: Prior work has demonstrated the near-perfect accuracy of a deep learning retinal image analysis system for diagnosing plus disease in retinopathy of prematurity (ROP). Here we assess the screening potential of this scoring system by determining its ability to detect all components of ROP diagnosis. Methods: Clinical examination and fundus photography were performed at seven participating centres. A deep learning system was trained to detect plus disease, generating a quantitative assessment of retinal vascular abnormality (the i-ROP plus score) on a 1-9 scale. Overall ROP disease category was established using a consensus reference standard diagnosis combining clinical and image-based diagnosis. Experts then ranked ordered a second data set of 100 posterior images according to overall ROP severity. Results: 4861 examinations from 870 infants were analysed. 155 examinations (3%) had a reference standard diagnosis of type 1 ROP. The i-ROP deep learning (DL) vascular severity score had an area under the receiver operating curve of 0.960 for detecting type 1 ROP. Establishing a threshold i-ROP DL score of 3 conferred 94% sensitivity, 79% specificity, 13% positive predictive value and 99.7% negative predictive value for type 1 ROP. There was strong correlation between expert rank ordering of overall ROP severity and the i-ROP DL vascular severity score (Spearman correlation coefficient=0.93; p<0.0001). Conclusion: The i-ROP DL system accurately identifies diagnostic categories and overall disease severity in an automated fashion, after being trained only on posterior pole vascular morphology. These data provide proof of concept that a deep learning screening platform could improve objectivity of ROP diagnosis and accessibility of screening.

KW - child health (paediatrics)

KW - public health

KW - retina

KW - telemedicine

UR - http://www.scopus.com/inward/record.url?scp=85057264647&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057264647&partnerID=8YFLogxK

U2 - 10.1136/bjophthalmol-2018-313156

DO - 10.1136/bjophthalmol-2018-313156

M3 - Article

JO - British Journal of Ophthalmology

JF - British Journal of Ophthalmology

SN - 0007-1161

ER -