TY - JOUR
T1 - Plus Disease in Retinopathy of Prematurity
T2 - Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis
AU - Imaging and Informatics in Retinopathy of Prematurity Research Consortium
AU - Kalpathy-Cramer, Jayashree
AU - Campbell, J. Peter
AU - Erdogmus, Deniz
AU - Tian, Peng
AU - Kedarisetti, Dharanish
AU - Moleta, Chace
AU - Reynolds, James D.
AU - Hutcheson, Kelly
AU - Shapiro, Michael J.
AU - Repka, Michael X.
AU - Ferrone, Philip
AU - Drenser, Kimberly
AU - Horowitz, Jason
AU - Sonmez, Kemal
AU - Swan, Ryan
AU - Ostmo, Susan
AU - Jonas, Karyn E.
AU - Chan, R. V.Paul
AU - Chiang, Michael F.
AU - Chiang, Michael F.
AU - Ostmo, Susan
AU - Sonmez, Kemal
AU - Campbell, J. Peter
AU - Jonas, Karyn
AU - Horowitz, Jason
AU - Coki, Osode
AU - Eccles, Cheryl Ann
AU - Sarna, Leora
AU - Berrocal, Audina
AU - Negron, Catherin
AU - Denser, Kimberly
AU - Cumming, Kristi
AU - Osentoski, Tammy
AU - Check, Tammy
AU - Zajechowski, Mary
AU - Lee, Thomas
AU - Kruger, Evan
AU - McGovern, Kathryn
AU - Simmons, Charles
AU - Murthy, Raghu
AU - Galvis, Sharon
AU - Rotter, Jerome
AU - Chen, Ida
AU - Li, Xiaohui
AU - Taylor, Kent
AU - Roll, Kaye
AU - Kalpathy-Cramer, Jayashree
AU - Erdogmus, Deniz
AU - Martinez-Castellanos, Maria Ana
AU - Salinas-Longoria, Samantha
N1 - Publisher Copyright:
© 2016 American Academy of Ophthalmology
PY - 2016/11/1
Y1 - 2016/11/1
N2 - Purpose To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. Design We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. Participants Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. Methods Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. Main Outcome Measures Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. Results There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06–0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74–0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95–0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). Conclusions Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.
AB - Purpose To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. Design We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. Participants Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. Methods Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. Main Outcome Measures Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. Results There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06–0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74–0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95–0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). Conclusions Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.
UR - http://www.scopus.com/inward/record.url?scp=84994123709&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84994123709&partnerID=8YFLogxK
U2 - 10.1016/j.ophtha.2016.07.020
DO - 10.1016/j.ophtha.2016.07.020
M3 - Article
C2 - 27566853
AN - SCOPUS:84994123709
SN - 0161-6420
VL - 123
SP - 2345
EP - 2351
JO - Ophthalmology
JF - Ophthalmology
IS - 11
ER -