Dealing with inter-expert variability in retinopathy of prematurity: A machine learning approach

V. Bolón-Canedo, E. Ataer-Cansizoglu, D. Erdogmus, Jayashree Kalpathy-Cramer, O. Fontenla-Romero, A. Alonso-Betanzos, Michael Chiang

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

Background and objective: Understanding the causes of disagreement among experts in clinical decision making has been a challenge for decades. In particular, a high amount of variability exists in diagnosis of retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and a major cause of childhood blindness. A possible cause of variability, that has been mostly neglected in the literature, is related to discrepancies in the sets of important features considered by different experts. In this paper we propose a methodology which makes use of machine learning techniques to understand the underlying causes of inter-expert variability. Methods: The experiments are carried out on a dataset consisting of 34 retinal images, each with diagnoses provided by 22 independent experts. Feature selection techniques are applied to discover the most important features considered by a given expert. Those features selected by each expert are then compared to the features selected by other experts by applying similarity measures. Finally, an automated diagnosis system is built in order to check if this approach can be helpful in solving the problem of understanding high inter-rater variability. Results: The experimental results reveal that some features are mostly selected by the feature selection methods regardless the considered expert. Moreover, for pairs of experts with high percentage agreement among them, the feature selection algorithms also select similar features. By using the relevant selected features, the classification performance of the automatic system was improved or maintained. Conclusions: The proposed methodology provides a handy framework to identify important features for experts and check whether the selected features reflect the pairwise agreements/disagreements. These findings may lead to improved diagnostic accuracy and standardization among clinicians, and pave the way for the application of this methodology to other problems which present inter-expert variability.

Original languageEnglish (US)
Pages (from-to)1-15
Number of pages15
JournalComputer Methods and Programs in Biomedicine
Volume122
Issue number1
DOIs
StatePublished - Oct 1 2015

Fingerprint

Retinopathy of Prematurity
Learning systems
Feature extraction
Low Birth Weight Infant
Blindness
Standardization
Decision making
Machine Learning
Experiments

Keywords

  • Classification
  • Clinical decision-making
  • Feature selection
  • Inter-expert variability
  • Machine learning
  • Retinopathy of prematurity

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Health Informatics

Cite this

Bolón-Canedo, V., Ataer-Cansizoglu, E., Erdogmus, D., Kalpathy-Cramer, J., Fontenla-Romero, O., Alonso-Betanzos, A., & Chiang, M. (2015). Dealing with inter-expert variability in retinopathy of prematurity: A machine learning approach. Computer Methods and Programs in Biomedicine, 122(1), 1-15. https://doi.org/10.1016/j.cmpb.2015.06.004

Dealing with inter-expert variability in retinopathy of prematurity : A machine learning approach. / Bolón-Canedo, V.; Ataer-Cansizoglu, E.; Erdogmus, D.; Kalpathy-Cramer, Jayashree; Fontenla-Romero, O.; Alonso-Betanzos, A.; Chiang, Michael.

In: Computer Methods and Programs in Biomedicine, Vol. 122, No. 1, 01.10.2015, p. 1-15.

Research output: Contribution to journalArticle

Bolón-Canedo, V, Ataer-Cansizoglu, E, Erdogmus, D, Kalpathy-Cramer, J, Fontenla-Romero, O, Alonso-Betanzos, A & Chiang, M 2015, 'Dealing with inter-expert variability in retinopathy of prematurity: A machine learning approach', Computer Methods and Programs in Biomedicine, vol. 122, no. 1, pp. 1-15. https://doi.org/10.1016/j.cmpb.2015.06.004
Bolón-Canedo V, Ataer-Cansizoglu E, Erdogmus D, Kalpathy-Cramer J, Fontenla-Romero O, Alonso-Betanzos A et al. Dealing with inter-expert variability in retinopathy of prematurity: A machine learning approach. Computer Methods and Programs in Biomedicine. 2015 Oct 1;122(1):1-15. https://doi.org/10.1016/j.cmpb.2015.06.004
Bolón-Canedo, V. ; Ataer-Cansizoglu, E. ; Erdogmus, D. ; Kalpathy-Cramer, Jayashree ; Fontenla-Romero, O. ; Alonso-Betanzos, A. ; Chiang, Michael. / Dealing with inter-expert variability in retinopathy of prematurity : A machine learning approach. In: Computer Methods and Programs in Biomedicine. 2015 ; Vol. 122, No. 1. pp. 1-15.
@article{95a4664fce23474dbbf1b02d0d7bd0cd,
title = "Dealing with inter-expert variability in retinopathy of prematurity: A machine learning approach",
abstract = "Background and objective: Understanding the causes of disagreement among experts in clinical decision making has been a challenge for decades. In particular, a high amount of variability exists in diagnosis of retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and a major cause of childhood blindness. A possible cause of variability, that has been mostly neglected in the literature, is related to discrepancies in the sets of important features considered by different experts. In this paper we propose a methodology which makes use of machine learning techniques to understand the underlying causes of inter-expert variability. Methods: The experiments are carried out on a dataset consisting of 34 retinal images, each with diagnoses provided by 22 independent experts. Feature selection techniques are applied to discover the most important features considered by a given expert. Those features selected by each expert are then compared to the features selected by other experts by applying similarity measures. Finally, an automated diagnosis system is built in order to check if this approach can be helpful in solving the problem of understanding high inter-rater variability. Results: The experimental results reveal that some features are mostly selected by the feature selection methods regardless the considered expert. Moreover, for pairs of experts with high percentage agreement among them, the feature selection algorithms also select similar features. By using the relevant selected features, the classification performance of the automatic system was improved or maintained. Conclusions: The proposed methodology provides a handy framework to identify important features for experts and check whether the selected features reflect the pairwise agreements/disagreements. These findings may lead to improved diagnostic accuracy and standardization among clinicians, and pave the way for the application of this methodology to other problems which present inter-expert variability.",
keywords = "Classification, Clinical decision-making, Feature selection, Inter-expert variability, Machine learning, Retinopathy of prematurity",
author = "V. Bol{\'o}n-Canedo and E. Ataer-Cansizoglu and D. Erdogmus and Jayashree Kalpathy-Cramer and O. Fontenla-Romero and A. Alonso-Betanzos and Michael Chiang",
year = "2015",
month = "10",
day = "1",
doi = "10.1016/j.cmpb.2015.06.004",
language = "English (US)",
volume = "122",
pages = "1--15",
journal = "Computer Methods and Programs in Biomedicine",
issn = "0169-2607",
publisher = "Elsevier Ireland Ltd",
number = "1",

}

TY - JOUR

T1 - Dealing with inter-expert variability in retinopathy of prematurity

T2 - A machine learning approach

AU - Bolón-Canedo, V.

AU - Ataer-Cansizoglu, E.

AU - Erdogmus, D.

AU - Kalpathy-Cramer, Jayashree

AU - Fontenla-Romero, O.

AU - Alonso-Betanzos, A.

AU - Chiang, Michael

PY - 2015/10/1

Y1 - 2015/10/1

N2 - Background and objective: Understanding the causes of disagreement among experts in clinical decision making has been a challenge for decades. In particular, a high amount of variability exists in diagnosis of retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and a major cause of childhood blindness. A possible cause of variability, that has been mostly neglected in the literature, is related to discrepancies in the sets of important features considered by different experts. In this paper we propose a methodology which makes use of machine learning techniques to understand the underlying causes of inter-expert variability. Methods: The experiments are carried out on a dataset consisting of 34 retinal images, each with diagnoses provided by 22 independent experts. Feature selection techniques are applied to discover the most important features considered by a given expert. Those features selected by each expert are then compared to the features selected by other experts by applying similarity measures. Finally, an automated diagnosis system is built in order to check if this approach can be helpful in solving the problem of understanding high inter-rater variability. Results: The experimental results reveal that some features are mostly selected by the feature selection methods regardless the considered expert. Moreover, for pairs of experts with high percentage agreement among them, the feature selection algorithms also select similar features. By using the relevant selected features, the classification performance of the automatic system was improved or maintained. Conclusions: The proposed methodology provides a handy framework to identify important features for experts and check whether the selected features reflect the pairwise agreements/disagreements. These findings may lead to improved diagnostic accuracy and standardization among clinicians, and pave the way for the application of this methodology to other problems which present inter-expert variability.

AB - Background and objective: Understanding the causes of disagreement among experts in clinical decision making has been a challenge for decades. In particular, a high amount of variability exists in diagnosis of retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and a major cause of childhood blindness. A possible cause of variability, that has been mostly neglected in the literature, is related to discrepancies in the sets of important features considered by different experts. In this paper we propose a methodology which makes use of machine learning techniques to understand the underlying causes of inter-expert variability. Methods: The experiments are carried out on a dataset consisting of 34 retinal images, each with diagnoses provided by 22 independent experts. Feature selection techniques are applied to discover the most important features considered by a given expert. Those features selected by each expert are then compared to the features selected by other experts by applying similarity measures. Finally, an automated diagnosis system is built in order to check if this approach can be helpful in solving the problem of understanding high inter-rater variability. Results: The experimental results reveal that some features are mostly selected by the feature selection methods regardless the considered expert. Moreover, for pairs of experts with high percentage agreement among them, the feature selection algorithms also select similar features. By using the relevant selected features, the classification performance of the automatic system was improved or maintained. Conclusions: The proposed methodology provides a handy framework to identify important features for experts and check whether the selected features reflect the pairwise agreements/disagreements. These findings may lead to improved diagnostic accuracy and standardization among clinicians, and pave the way for the application of this methodology to other problems which present inter-expert variability.

KW - Classification

KW - Clinical decision-making

KW - Feature selection

KW - Inter-expert variability

KW - Machine learning

KW - Retinopathy of prematurity

UR - http://www.scopus.com/inward/record.url?scp=84939574276&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84939574276&partnerID=8YFLogxK

U2 - 10.1016/j.cmpb.2015.06.004

DO - 10.1016/j.cmpb.2015.06.004

M3 - Article

C2 - 26120072

AN - SCOPUS:84939574276

VL - 122

SP - 1

EP - 15

JO - Computer Methods and Programs in Biomedicine

JF - Computer Methods and Programs in Biomedicine

SN - 0169-2607

IS - 1

ER -