Variability in reader selection of subtle cases for inclusion in a reduced size ROC study

Jill L. King, Howard E. Rockette, Melinda J M D Staiger, Jane Weissman, Howard Eisen, Chris M. Johns, David Gur

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Sample size in ROC studies may be significantly reduced by including only difficult cases in the image set, but variability across studies may be a possible obstacle in the development of such a methodology due to case selection. To assess this situation, 300 cases used in a previous large ROC study, which included nine observers, were independently classified as subtle or typical by two experienced readers. Data from the previous study were reanalyzed using data sets consisting only subtle or typical images as designated by each classifier. Results showed a marked decrease in observer performance from the original study's results when only the subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases, 3 modes and 2 diseases for typical cases), the Spearman rho correlation coefficient between the performance indices computed for each reader for the subsets classified as subtle and typical was high and significant (P <0.05). The results obtained in this preliminary study are encouraging and point out areas that warrant further investigation.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
EditorsHarold L. Kundel
Pages74-78
Number of pages5
Volume2436
StatePublished - 1995
Externally publishedYes
EventMedical Imaging 1995: Image Perception - San Diego, CA, USA
Duration: Mar 1 1995Mar 1 1995

Other

OtherMedical Imaging 1995: Image Perception
CitySan Diego, CA, USA
Period3/1/953/1/95

Fingerprint

readers
classifiers
Classifiers
inclusions
Imaging techniques
correlation coefficients
set theory
methodology

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Condensed Matter Physics

Cite this

King, J. L., Rockette, H. E., Staiger, M. J. M. D., Weissman, J., Eisen, H., Johns, C. M., & Gur, D. (1995). Variability in reader selection of subtle cases for inclusion in a reduced size ROC study. In H. L. Kundel (Ed.), Proceedings of SPIE - The International Society for Optical Engineering (Vol. 2436, pp. 74-78)

Variability in reader selection of subtle cases for inclusion in a reduced size ROC study. / King, Jill L.; Rockette, Howard E.; Staiger, Melinda J M D; Weissman, Jane; Eisen, Howard; Johns, Chris M.; Gur, David.

Proceedings of SPIE - The International Society for Optical Engineering. ed. / Harold L. Kundel. Vol. 2436 1995. p. 74-78.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

King, JL, Rockette, HE, Staiger, MJMD, Weissman, J, Eisen, H, Johns, CM & Gur, D 1995, Variability in reader selection of subtle cases for inclusion in a reduced size ROC study. in HL Kundel (ed.), Proceedings of SPIE - The International Society for Optical Engineering. vol. 2436, pp. 74-78, Medical Imaging 1995: Image Perception, San Diego, CA, USA, 3/1/95.
King JL, Rockette HE, Staiger MJMD, Weissman J, Eisen H, Johns CM et al. Variability in reader selection of subtle cases for inclusion in a reduced size ROC study. In Kundel HL, editor, Proceedings of SPIE - The International Society for Optical Engineering. Vol. 2436. 1995. p. 74-78
King, Jill L. ; Rockette, Howard E. ; Staiger, Melinda J M D ; Weissman, Jane ; Eisen, Howard ; Johns, Chris M. ; Gur, David. / Variability in reader selection of subtle cases for inclusion in a reduced size ROC study. Proceedings of SPIE - The International Society for Optical Engineering. editor / Harold L. Kundel. Vol. 2436 1995. pp. 74-78
@inproceedings{a4e10406caa448e8a2a227ee2b39cc7d,
title = "Variability in reader selection of subtle cases for inclusion in a reduced size ROC study",
abstract = "Sample size in ROC studies may be significantly reduced by including only difficult cases in the image set, but variability across studies may be a possible obstacle in the development of such a methodology due to case selection. To assess this situation, 300 cases used in a previous large ROC study, which included nine observers, were independently classified as subtle or typical by two experienced readers. Data from the previous study were reanalyzed using data sets consisting only subtle or typical images as designated by each classifier. Results showed a marked decrease in observer performance from the original study's results when only the subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases, 3 modes and 2 diseases for typical cases), the Spearman rho correlation coefficient between the performance indices computed for each reader for the subsets classified as subtle and typical was high and significant (P <0.05). The results obtained in this preliminary study are encouraging and point out areas that warrant further investigation.",
author = "King, {Jill L.} and Rockette, {Howard E.} and Staiger, {Melinda J M D} and Jane Weissman and Howard Eisen and Johns, {Chris M.} and David Gur",
year = "1995",
language = "English (US)",
isbn = "081941784X",
volume = "2436",
pages = "74--78",
editor = "Kundel, {Harold L.}",
booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",

}

TY - GEN

T1 - Variability in reader selection of subtle cases for inclusion in a reduced size ROC study

AU - King, Jill L.

AU - Rockette, Howard E.

AU - Staiger, Melinda J M D

AU - Weissman, Jane

AU - Eisen, Howard

AU - Johns, Chris M.

AU - Gur, David

PY - 1995

Y1 - 1995

N2 - Sample size in ROC studies may be significantly reduced by including only difficult cases in the image set, but variability across studies may be a possible obstacle in the development of such a methodology due to case selection. To assess this situation, 300 cases used in a previous large ROC study, which included nine observers, were independently classified as subtle or typical by two experienced readers. Data from the previous study were reanalyzed using data sets consisting only subtle or typical images as designated by each classifier. Results showed a marked decrease in observer performance from the original study's results when only the subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases, 3 modes and 2 diseases for typical cases), the Spearman rho correlation coefficient between the performance indices computed for each reader for the subsets classified as subtle and typical was high and significant (P <0.05). The results obtained in this preliminary study are encouraging and point out areas that warrant further investigation.

AB - Sample size in ROC studies may be significantly reduced by including only difficult cases in the image set, but variability across studies may be a possible obstacle in the development of such a methodology due to case selection. To assess this situation, 300 cases used in a previous large ROC study, which included nine observers, were independently classified as subtle or typical by two experienced readers. Data from the previous study were reanalyzed using data sets consisting only subtle or typical images as designated by each classifier. Results showed a marked decrease in observer performance from the original study's results when only the subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases, 3 modes and 2 diseases for typical cases), the Spearman rho correlation coefficient between the performance indices computed for each reader for the subsets classified as subtle and typical was high and significant (P <0.05). The results obtained in this preliminary study are encouraging and point out areas that warrant further investigation.

UR - http://www.scopus.com/inward/record.url?scp=0029462539&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029462539&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0029462539

SN - 081941784X

SN - 9780819417848

VL - 2436

SP - 74

EP - 78

BT - Proceedings of SPIE - The International Society for Optical Engineering

A2 - Kundel, Harold L.

ER -