Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis

Application of machine learning principles

E. Ataer-Cansizoglu, Jayashree Kalpathy-Cramer, S. You, K. Keck, D. Erdogmus, Michael Chiang

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

Objective: Inter-expert variability in imagebased clinical diagnosis has been demonstrated in many diseases including retinopa -thy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.

Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observer

Results: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.

Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

Original languageEnglish (US)
Pages (from-to)93-102
Number of pages10
JournalMethods of Information in Medicine
Volume54
Issue number1
DOIs
StatePublished - 2015

Fingerprint

Spatial Analysis
Decision Support Techniques
Low Birth Weight Infant
Blindness
Machine Learning
Datasets

Keywords

  • Feature selection
  • Inter-expert disagreement
  • Kernel density estimation
  • Retinopathy of prematurity

ASJC Scopus subject areas

  • Health Informatics
  • Health Information Management
  • Advanced and Specialized Nursing

Cite this

Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis : Application of machine learning principles. / Ataer-Cansizoglu, E.; Kalpathy-Cramer, Jayashree; You, S.; Keck, K.; Erdogmus, D.; Chiang, Michael.

In: Methods of Information in Medicine, Vol. 54, No. 1, 2015, p. 93-102.

Research output: Contribution to journalArticle

Ataer-Cansizoglu, E. ; Kalpathy-Cramer, Jayashree ; You, S. ; Keck, K. ; Erdogmus, D. ; Chiang, Michael. / Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis : Application of machine learning principles. In: Methods of Information in Medicine. 2015 ; Vol. 54, No. 1. pp. 93-102.
@article{8afa0394636b4a47a5646852654ef56e,
title = "Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis: Application of machine learning principles",
abstract = "Objective: Inter-expert variability in imagebased clinical diagnosis has been demonstrated in many diseases including retinopa -thy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observerResults: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.",
keywords = "Feature selection, Inter-expert disagreement, Kernel density estimation, Retinopathy of prematurity",
author = "E. Ataer-Cansizoglu and Jayashree Kalpathy-Cramer and S. You and K. Keck and D. Erdogmus and Michael Chiang",
year = "2015",
doi = "10.3414/ME13-01-0081",
language = "English (US)",
volume = "54",
pages = "93--102",
journal = "Methods of Information in Medicine",
issn = "0026-1270",
publisher = "Schattauer GmbH",
number = "1",

}

TY - JOUR

T1 - Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis

T2 - Application of machine learning principles

AU - Ataer-Cansizoglu, E.

AU - Kalpathy-Cramer, Jayashree

AU - You, S.

AU - Keck, K.

AU - Erdogmus, D.

AU - Chiang, Michael

PY - 2015

Y1 - 2015

N2 - Objective: Inter-expert variability in imagebased clinical diagnosis has been demonstrated in many diseases including retinopa -thy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observerResults: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

AB - Objective: Inter-expert variability in imagebased clinical diagnosis has been demonstrated in many diseases including retinopa -thy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observerResults: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

KW - Feature selection

KW - Inter-expert disagreement

KW - Kernel density estimation

KW - Retinopathy of prematurity

UR - http://www.scopus.com/inward/record.url?scp=84921286003&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84921286003&partnerID=8YFLogxK

U2 - 10.3414/ME13-01-0081

DO - 10.3414/ME13-01-0081

M3 - Article

VL - 54

SP - 93

EP - 102

JO - Methods of Information in Medicine

JF - Methods of Information in Medicine

SN - 0026-1270

IS - 1

ER -