Potholes and molehills

Bias in the diagnostic performance of diffusion-tensor imaging in concussion

Richard Watts, Alex Thomas, Christopher G. Filippi, Joshua Nickerson, Kalev Freeman

Research output: Contribution to journalArticle

22 Citations (Scopus)

Abstract

Purpose: To investigate the extent of bias in a clinical study involving "pothole analysis" of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias. Materials and Methods: Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test. Results: Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 ± 34.3 vs 50.6 ± 28.9, P < .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 ± 34.3; control group, 93.4 ± 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30% in the control group. Conclusion: The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.

Original languageEnglish (US)
Pages (from-to)217-223
Number of pages7
JournalRadiology
Volume272
Issue number1
DOIs
StatePublished - Jan 1 2014
Externally publishedYes

Fingerprint

Brain Concussion
Diffusion Tensor Imaging
Control Groups
Research Ethics Committees
Anisotropy
Nonparametric Statistics
Neuroimaging
Population
Observational Studies
Hospital Emergency Service
Cohort Studies
Theoretical Models

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

Potholes and molehills : Bias in the diagnostic performance of diffusion-tensor imaging in concussion. / Watts, Richard; Thomas, Alex; Filippi, Christopher G.; Nickerson, Joshua; Freeman, Kalev.

In: Radiology, Vol. 272, No. 1, 01.01.2014, p. 217-223.

Research output: Contribution to journalArticle

Watts, Richard ; Thomas, Alex ; Filippi, Christopher G. ; Nickerson, Joshua ; Freeman, Kalev. / Potholes and molehills : Bias in the diagnostic performance of diffusion-tensor imaging in concussion. In: Radiology. 2014 ; Vol. 272, No. 1. pp. 217-223.
@article{a0e04741aa3a472da140f6022794e366,
title = "Potholes and molehills: Bias in the diagnostic performance of diffusion-tensor imaging in concussion",
abstract = "Purpose: To investigate the extent of bias in a clinical study involving {"}pothole analysis{"} of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias. Materials and Methods: Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test. Results: Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 ± 34.3 vs 50.6 ± 28.9, P < .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 ± 34.3; control group, 93.4 ± 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30{\%} in the control group. Conclusion: The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.",
author = "Richard Watts and Alex Thomas and Filippi, {Christopher G.} and Joshua Nickerson and Kalev Freeman",
year = "2014",
month = "1",
day = "1",
doi = "10.1148/radiol.14131856",
language = "English (US)",
volume = "272",
pages = "217--223",
journal = "Radiology",
issn = "0033-8419",
publisher = "Radiological Society of North America Inc.",
number = "1",

}

TY - JOUR

T1 - Potholes and molehills

T2 - Bias in the diagnostic performance of diffusion-tensor imaging in concussion

AU - Watts, Richard

AU - Thomas, Alex

AU - Filippi, Christopher G.

AU - Nickerson, Joshua

AU - Freeman, Kalev

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Purpose: To investigate the extent of bias in a clinical study involving "pothole analysis" of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias. Materials and Methods: Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test. Results: Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 ± 34.3 vs 50.6 ± 28.9, P < .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 ± 34.3; control group, 93.4 ± 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30% in the control group. Conclusion: The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.

AB - Purpose: To investigate the extent of bias in a clinical study involving "pothole analysis" of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias. Materials and Methods: Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test. Results: Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 ± 34.3 vs 50.6 ± 28.9, P < .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 ± 34.3; control group, 93.4 ± 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30% in the control group. Conclusion: The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.

UR - http://www.scopus.com/inward/record.url?scp=84903733815&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84903733815&partnerID=8YFLogxK

U2 - 10.1148/radiol.14131856

DO - 10.1148/radiol.14131856

M3 - Article

VL - 272

SP - 217

EP - 223

JO - Radiology

JF - Radiology

SN - 0033-8419

IS - 1

ER -