Plus Disease in Retinopathy of Prematurity

A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability

John Campbell, Jayashree Kalpathy-Cramer, Deniz Erdogmus, Peng Tian, Dharanish Kedarisetti, Chace Moleta, James D. Reynolds, Kelly Hutcheson, Michael J. Shapiro, Michael X. Repka, Philip Ferrone, Kimberly Drenser, Jason Horowitz, Mustafa (Kemal) Sonmez, Ryan Swan, Susan Ostmo, Karyn E. Jonas, R. V Paul Chan, Michael Chiang, Michael F. Chiang & 37 others Susan Ostmo, Kemal Sonmez, J. Peter Campbell, R. V Paul Chan, Karyn Jonas, Jason Horowitz, Osode Coki, Cheryl Ann Eccles, Leora Sarna, Audina Berrocal, Catherin Negron, Kimberly Denser, Kristi Cumming, Tammy Osentoski, Tammy Check, Mary Zajechowski, Thomas Lee, Evan Kruger, Kathryn McGovern, Charles Simmons, Raghu Murthy, Sharon Galvis, Jerome Rotter, Ida Chen, Xiaohui Li, Kent Taylor, Kaye Roll, Jayashree Kalpathy-Cramer, Deniz Erdogmus, Maria Ana Martinez-Castellanos, Samantha Salinas-Longoria, Rafael Romero, Andrea Arriola, Francisco Olguin-Manriquez, Miroslava Meraz-Gutierrez, Carlos M. Dulanto-Reinoso, Cristina Montero-Mendoza

Research output: Contribution to journalArticle

23 Citations (Scopus)

Abstract

Purpose To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). Design We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Participants Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. Methods Expert classification of images of plus disease in ROP. Main Outcome Measures Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). Results There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0–0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615). Conclusions There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

Original languageEnglish (US)
Pages (from-to)2338-2344
Number of pages7
JournalOphthalmology
Volume123
Issue number11
DOIs
StatePublished - Nov 1 2016

Fingerprint

Retinopathy of Prematurity
Blood Vessels
Analysis of Variance
Informatics
Neonatal Intensive Care Units
Publications
Patient Care
Teaching
Outcome Assessment (Health Care)
Datasets

ASJC Scopus subject areas

  • Ophthalmology

Cite this

Campbell, J., Kalpathy-Cramer, J., Erdogmus, D., Tian, P., Kedarisetti, D., Moleta, C., ... Montero-Mendoza, C. (2016). Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability. Ophthalmology, 123(11), 2338-2344. https://doi.org/10.1016/j.ophtha.2016.07.026

Plus Disease in Retinopathy of Prematurity : A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability. / Campbell, John; Kalpathy-Cramer, Jayashree; Erdogmus, Deniz; Tian, Peng; Kedarisetti, Dharanish; Moleta, Chace; Reynolds, James D.; Hutcheson, Kelly; Shapiro, Michael J.; Repka, Michael X.; Ferrone, Philip; Drenser, Kimberly; Horowitz, Jason; Sonmez, Mustafa (Kemal); Swan, Ryan; Ostmo, Susan; Jonas, Karyn E.; Chan, R. V Paul; Chiang, Michael; Chiang, Michael F.; Ostmo, Susan; Sonmez, Kemal; Campbell, J. Peter; Chan, R. V Paul; Jonas, Karyn; Horowitz, Jason; Coki, Osode; Eccles, Cheryl Ann; Sarna, Leora; Berrocal, Audina; Negron, Catherin; Denser, Kimberly; Cumming, Kristi; Osentoski, Tammy; Check, Tammy; Zajechowski, Mary; Lee, Thomas; Kruger, Evan; McGovern, Kathryn; Simmons, Charles; Murthy, Raghu; Galvis, Sharon; Rotter, Jerome; Chen, Ida; Li, Xiaohui; Taylor, Kent; Roll, Kaye; Kalpathy-Cramer, Jayashree; Erdogmus, Deniz; Martinez-Castellanos, Maria Ana; Salinas-Longoria, Samantha; Romero, Rafael; Arriola, Andrea; Olguin-Manriquez, Francisco; Meraz-Gutierrez, Miroslava; Dulanto-Reinoso, Carlos M.; Montero-Mendoza, Cristina.

In: Ophthalmology, Vol. 123, No. 11, 01.11.2016, p. 2338-2344.

Research output: Contribution to journalArticle

Campbell, J, Kalpathy-Cramer, J, Erdogmus, D, Tian, P, Kedarisetti, D, Moleta, C, Reynolds, JD, Hutcheson, K, Shapiro, MJ, Repka, MX, Ferrone, P, Drenser, K, Horowitz, J, Sonmez, MK, Swan, R, Ostmo, S, Jonas, KE, Chan, RVP, Chiang, M, Chiang, MF, Ostmo, S, Sonmez, K, Campbell, JP, Chan, RVP, Jonas, K, Horowitz, J, Coki, O, Eccles, CA, Sarna, L, Berrocal, A, Negron, C, Denser, K, Cumming, K, Osentoski, T, Check, T, Zajechowski, M, Lee, T, Kruger, E, McGovern, K, Simmons, C, Murthy, R, Galvis, S, Rotter, J, Chen, I, Li, X, Taylor, K, Roll, K, Kalpathy-Cramer, J, Erdogmus, D, Martinez-Castellanos, MA, Salinas-Longoria, S, Romero, R, Arriola, A, Olguin-Manriquez, F, Meraz-Gutierrez, M, Dulanto-Reinoso, CM & Montero-Mendoza, C 2016, 'Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability', Ophthalmology, vol. 123, no. 11, pp. 2338-2344. https://doi.org/10.1016/j.ophtha.2016.07.026
Campbell, John ; Kalpathy-Cramer, Jayashree ; Erdogmus, Deniz ; Tian, Peng ; Kedarisetti, Dharanish ; Moleta, Chace ; Reynolds, James D. ; Hutcheson, Kelly ; Shapiro, Michael J. ; Repka, Michael X. ; Ferrone, Philip ; Drenser, Kimberly ; Horowitz, Jason ; Sonmez, Mustafa (Kemal) ; Swan, Ryan ; Ostmo, Susan ; Jonas, Karyn E. ; Chan, R. V Paul ; Chiang, Michael ; Chiang, Michael F. ; Ostmo, Susan ; Sonmez, Kemal ; Campbell, J. Peter ; Chan, R. V Paul ; Jonas, Karyn ; Horowitz, Jason ; Coki, Osode ; Eccles, Cheryl Ann ; Sarna, Leora ; Berrocal, Audina ; Negron, Catherin ; Denser, Kimberly ; Cumming, Kristi ; Osentoski, Tammy ; Check, Tammy ; Zajechowski, Mary ; Lee, Thomas ; Kruger, Evan ; McGovern, Kathryn ; Simmons, Charles ; Murthy, Raghu ; Galvis, Sharon ; Rotter, Jerome ; Chen, Ida ; Li, Xiaohui ; Taylor, Kent ; Roll, Kaye ; Kalpathy-Cramer, Jayashree ; Erdogmus, Deniz ; Martinez-Castellanos, Maria Ana ; Salinas-Longoria, Samantha ; Romero, Rafael ; Arriola, Andrea ; Olguin-Manriquez, Francisco ; Meraz-Gutierrez, Miroslava ; Dulanto-Reinoso, Carlos M. ; Montero-Mendoza, Cristina. / Plus Disease in Retinopathy of Prematurity : A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability. In: Ophthalmology. 2016 ; Vol. 123, No. 11. pp. 2338-2344.
@article{98c338959aef4e7b92be4ab00e702c14,
title = "Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability",
abstract = "Purpose To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). Design We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Participants Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. Methods Expert classification of images of plus disease in ROP. Main Outcome Measures Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). Results There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0–0.75; mean, 0.30). The RSD agreement ranged from 80{\%} to 94{\%} for the dataset of 100 images and from 29{\%} to 79{\%} for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615). Conclusions There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.",
author = "John Campbell and Jayashree Kalpathy-Cramer and Deniz Erdogmus and Peng Tian and Dharanish Kedarisetti and Chace Moleta and Reynolds, {James D.} and Kelly Hutcheson and Shapiro, {Michael J.} and Repka, {Michael X.} and Philip Ferrone and Kimberly Drenser and Jason Horowitz and Sonmez, {Mustafa (Kemal)} and Ryan Swan and Susan Ostmo and Jonas, {Karyn E.} and Chan, {R. V Paul} and Michael Chiang and Chiang, {Michael F.} and Susan Ostmo and Kemal Sonmez and Campbell, {J. Peter} and Chan, {R. V Paul} and Karyn Jonas and Jason Horowitz and Osode Coki and Eccles, {Cheryl Ann} and Leora Sarna and Audina Berrocal and Catherin Negron and Kimberly Denser and Kristi Cumming and Tammy Osentoski and Tammy Check and Mary Zajechowski and Thomas Lee and Evan Kruger and Kathryn McGovern and Charles Simmons and Raghu Murthy and Sharon Galvis and Jerome Rotter and Ida Chen and Xiaohui Li and Kent Taylor and Kaye Roll and Jayashree Kalpathy-Cramer and Deniz Erdogmus and Martinez-Castellanos, {Maria Ana} and Samantha Salinas-Longoria and Rafael Romero and Andrea Arriola and Francisco Olguin-Manriquez and Miroslava Meraz-Gutierrez and Dulanto-Reinoso, {Carlos M.} and Cristina Montero-Mendoza",
year = "2016",
month = "11",
day = "1",
doi = "10.1016/j.ophtha.2016.07.026",
language = "English (US)",
volume = "123",
pages = "2338--2344",
journal = "Ophthalmology",
issn = "0161-6420",
publisher = "Elsevier Inc.",
number = "11",

}

TY - JOUR

T1 - Plus Disease in Retinopathy of Prematurity

T2 - A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability

AU - Campbell, John

AU - Kalpathy-Cramer, Jayashree

AU - Erdogmus, Deniz

AU - Tian, Peng

AU - Kedarisetti, Dharanish

AU - Moleta, Chace

AU - Reynolds, James D.

AU - Hutcheson, Kelly

AU - Shapiro, Michael J.

AU - Repka, Michael X.

AU - Ferrone, Philip

AU - Drenser, Kimberly

AU - Horowitz, Jason

AU - Sonmez, Mustafa (Kemal)

AU - Swan, Ryan

AU - Ostmo, Susan

AU - Jonas, Karyn E.

AU - Chan, R. V Paul

AU - Chiang, Michael

AU - Chiang, Michael F.

AU - Ostmo, Susan

AU - Sonmez, Kemal

AU - Campbell, J. Peter

AU - Chan, R. V Paul

AU - Jonas, Karyn

AU - Horowitz, Jason

AU - Coki, Osode

AU - Eccles, Cheryl Ann

AU - Sarna, Leora

AU - Berrocal, Audina

AU - Negron, Catherin

AU - Denser, Kimberly

AU - Cumming, Kristi

AU - Osentoski, Tammy

AU - Check, Tammy

AU - Zajechowski, Mary

AU - Lee, Thomas

AU - Kruger, Evan

AU - McGovern, Kathryn

AU - Simmons, Charles

AU - Murthy, Raghu

AU - Galvis, Sharon

AU - Rotter, Jerome

AU - Chen, Ida

AU - Li, Xiaohui

AU - Taylor, Kent

AU - Roll, Kaye

AU - Kalpathy-Cramer, Jayashree

AU - Erdogmus, Deniz

AU - Martinez-Castellanos, Maria Ana

AU - Salinas-Longoria, Samantha

AU - Romero, Rafael

AU - Arriola, Andrea

AU - Olguin-Manriquez, Francisco

AU - Meraz-Gutierrez, Miroslava

AU - Dulanto-Reinoso, Carlos M.

AU - Montero-Mendoza, Cristina

PY - 2016/11/1

Y1 - 2016/11/1

N2 - Purpose To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). Design We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Participants Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. Methods Expert classification of images of plus disease in ROP. Main Outcome Measures Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). Results There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0–0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615). Conclusions There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

AB - Purpose To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). Design We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Participants Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. Methods Expert classification of images of plus disease in ROP. Main Outcome Measures Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). Results There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0–0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615). Conclusions There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

UR - http://www.scopus.com/inward/record.url?scp=84994891814&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994891814&partnerID=8YFLogxK

U2 - 10.1016/j.ophtha.2016.07.026

DO - 10.1016/j.ophtha.2016.07.026

M3 - Article

VL - 123

SP - 2338

EP - 2344

JO - Ophthalmology

JF - Ophthalmology

SN - 0161-6420

IS - 11

ER -