Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity

Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium, Sang Jin Kim, John Campbell, Jayashree Kalpathy-Cramer, Susan Ostmo, Karyn E. Jonas, Dongseok Choi, R. V. Paul Chan, Michael Chiang, Mustafa (Kemal) Sonmez, Jason Horowitz, Osode Coki, Cheryl Ann Eccles, Leora Sarna, Anton Orlin, Audina Berrocal, Catherin Negron, Kimberly Denser, Kristi Cumming, Tammy Osentoski & 23 others Tammy Check, Mary Zajechowski, Thomas Lee, Evan Kruger, Kathryn McGovern, Charles Simmons, Raghu Murthy, Sharon Galvis, Jerome Rotter, Ida Chen, Xiaohui Li, Kent Taylor, Kaye Roll, Deniz Erdogmus, Stratis Ioannidis, Maria Ana Martinez-Castellanos, Samantha Salinas-Longoria, Rafael Romero, Andrea Arriola, Francisco Olguin-Manriquez, Miroslava Meraz-Gutierrez, Carlos M. Dulanto-Reinoso, Cristina Montero-Mendoza

Research output: Contribution to journalArticle

  • 2 Citations

Abstract

IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

LanguageEnglish (US)
Pages648-655
Number of pages8
JournalJAMA Ophthalmology
Volume136
Issue number6
DOIs
StatePublished - Jun 1 2018

Fingerprint

Retinopathy of Prematurity
Premature Infants

ASJC Scopus subject areas

  • Ophthalmology

Cite this

Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity. / Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium.

In: JAMA Ophthalmology, Vol. 136, No. 6, 01.06.2018, p. 648-655.

Research output: Contribution to journalArticle

Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium 2018, 'Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity' JAMA Ophthalmology, vol. 136, no. 6, pp. 648-655. https://doi.org/10.1001/jamaophthalmol.2018.1195
Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium. / Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity. In: JAMA Ophthalmology. 2018 ; Vol. 136, No. 6. pp. 648-655.
@article{53a9069aab964726b3ad05366454d592,
title = "Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity",
abstract = "IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1{\%}) were female and 116 (82.3{\%}) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95{\%} CI, 0.71-0.78] vs 0.55 [95{\%} CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.",
author = "{Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium} and Kim, {Sang Jin} and John Campbell and Jayashree Kalpathy-Cramer and Susan Ostmo and Jonas, {Karyn E.} and Dongseok Choi and {Paul Chan}, {R. V.} and Michael Chiang and Sonmez, {Mustafa (Kemal)} and Jason Horowitz and Osode Coki and Eccles, {Cheryl Ann} and Leora Sarna and Anton Orlin and Audina Berrocal and Catherin Negron and Kimberly Denser and Kristi Cumming and Tammy Osentoski and Tammy Check and Mary Zajechowski and Thomas Lee and Evan Kruger and Kathryn McGovern and Charles Simmons and Raghu Murthy and Sharon Galvis and Jerome Rotter and Ida Chen and Xiaohui Li and Kent Taylor and Kaye Roll and Deniz Erdogmus and Stratis Ioannidis and Martinez-Castellanos, {Maria Ana} and Samantha Salinas-Longoria and Rafael Romero and Andrea Arriola and Francisco Olguin-Manriquez and Miroslava Meraz-Gutierrez and Dulanto-Reinoso, {Carlos M.} and Cristina Montero-Mendoza",
year = "2018",
month = "6",
day = "1",
doi = "10.1001/jamaophthalmol.2018.1195",
language = "English (US)",
volume = "136",
pages = "648--655",
journal = "JAMA Ophthalmology",
issn = "2168-6165",
publisher = "American Medical Association",
number = "6",

}

TY - JOUR

T1 - Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity

AU - Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium

AU - Kim, Sang Jin

AU - Campbell, John

AU - Kalpathy-Cramer, Jayashree

AU - Ostmo, Susan

AU - Jonas, Karyn E.

AU - Choi, Dongseok

AU - Paul Chan, R. V.

AU - Chiang, Michael

AU - Sonmez, Mustafa (Kemal)

AU - Horowitz, Jason

AU - Coki, Osode

AU - Eccles, Cheryl Ann

AU - Sarna, Leora

AU - Orlin, Anton

AU - Berrocal, Audina

AU - Negron, Catherin

AU - Denser, Kimberly

AU - Cumming, Kristi

AU - Osentoski, Tammy

AU - Check, Tammy

AU - Zajechowski, Mary

AU - Lee, Thomas

AU - Kruger, Evan

AU - McGovern, Kathryn

AU - Simmons, Charles

AU - Murthy, Raghu

AU - Galvis, Sharon

AU - Rotter, Jerome

AU - Chen, Ida

AU - Li, Xiaohui

AU - Taylor, Kent

AU - Roll, Kaye

AU - Erdogmus, Deniz

AU - Ioannidis, Stratis

AU - Martinez-Castellanos, Maria Ana

AU - Salinas-Longoria, Samantha

AU - Romero, Rafael

AU - Arriola, Andrea

AU - Olguin-Manriquez, Francisco

AU - Meraz-Gutierrez, Miroslava

AU - Dulanto-Reinoso, Carlos M.

AU - Montero-Mendoza, Cristina

PY - 2018/6/1

Y1 - 2018/6/1

N2 - IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

AB - IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

UR - http://www.scopus.com/inward/record.url?scp=85048787689&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048787689&partnerID=8YFLogxK

U2 - 10.1001/jamaophthalmol.2018.1195

DO - 10.1001/jamaophthalmol.2018.1195

M3 - Article

VL - 136

SP - 648

EP - 655

JO - JAMA Ophthalmology

T2 - JAMA Ophthalmology

JF - JAMA Ophthalmology

SN - 2168-6165

IS - 6

ER -