Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry

The Results of an International Challenge

Marthony Robins, Jayashree Kalpathy-Cramer, Nancy A. Obuchowski, Andrew Buckler, Maria Athelogou, Rudresh Jarecha, Nicholas Petrick, Aria Pezeshk, Berkman Sahiner, Ehsan Samei

Research output: Contribution to journalArticle

Abstract

Rationale and Objectives: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. Materials and Methods: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). Results: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were −1.8 ± 1.4%, −2.5 ± 1.1%, −3 ± 1%, −1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%–16% and 7%–18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. Conclusion: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4%). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies.

Original languageEnglish (US)
Pages (from-to)e161-e173
JournalAcademic radiology
Volume26
Issue number7
DOIs
StatePublished - Jul 1 2019
Externally publishedYes

Fingerprint

Thorax
Biomarkers
Compliance
Imaging Phantoms
Tomography
Databases
Proxy
Datasets
Confidence Intervals
Therapeutics
Neoplasms

Keywords

  • CT
  • Hybrid dataset
  • Lung cancer
  • Quantitative imaging
  • Segmentation
  • Volumetry

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

Robins, M., Kalpathy-Cramer, J., Obuchowski, N. A., Buckler, A., Athelogou, M., Jarecha, R., ... Samei, E. (2019). Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry: The Results of an International Challenge. Academic radiology, 26(7), e161-e173. https://doi.org/10.1016/j.acra.2018.07.022

Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry : The Results of an International Challenge. / Robins, Marthony; Kalpathy-Cramer, Jayashree; Obuchowski, Nancy A.; Buckler, Andrew; Athelogou, Maria; Jarecha, Rudresh; Petrick, Nicholas; Pezeshk, Aria; Sahiner, Berkman; Samei, Ehsan.

In: Academic radiology, Vol. 26, No. 7, 01.07.2019, p. e161-e173.

Research output: Contribution to journalArticle

Robins, M, Kalpathy-Cramer, J, Obuchowski, NA, Buckler, A, Athelogou, M, Jarecha, R, Petrick, N, Pezeshk, A, Sahiner, B & Samei, E 2019, 'Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry: The Results of an International Challenge', Academic radiology, vol. 26, no. 7, pp. e161-e173. https://doi.org/10.1016/j.acra.2018.07.022
Robins, Marthony ; Kalpathy-Cramer, Jayashree ; Obuchowski, Nancy A. ; Buckler, Andrew ; Athelogou, Maria ; Jarecha, Rudresh ; Petrick, Nicholas ; Pezeshk, Aria ; Sahiner, Berkman ; Samei, Ehsan. / Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry : The Results of an International Challenge. In: Academic radiology. 2019 ; Vol. 26, No. 7. pp. e161-e173.
@article{c1cef20e30a0432f8d16ac99b02f579f,
title = "Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry: The Results of an International Challenge",
abstract = "Rationale and Objectives: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. Materials and Methods: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). Results: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were −1.8 ± 1.4{\%}, −2.5 ± 1.1{\%}, −3 ± 1{\%}, −1.8 ± 1.5{\%} (±95{\%} confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5{\%} equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1{\%}–16{\%} and 7{\%}–18{\%} differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. Conclusion: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4{\%}). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies.",
keywords = "CT, Hybrid dataset, Lung cancer, Quantitative imaging, Segmentation, Volumetry",
author = "Marthony Robins and Jayashree Kalpathy-Cramer and Obuchowski, {Nancy A.} and Andrew Buckler and Maria Athelogou and Rudresh Jarecha and Nicholas Petrick and Aria Pezeshk and Berkman Sahiner and Ehsan Samei",
year = "2019",
month = "7",
day = "1",
doi = "10.1016/j.acra.2018.07.022",
language = "English (US)",
volume = "26",
pages = "e161--e173",
journal = "Academic Radiology",
issn = "1076-6332",
publisher = "Elsevier USA",
number = "7",

}

TY - JOUR

T1 - Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry

T2 - The Results of an International Challenge

AU - Robins, Marthony

AU - Kalpathy-Cramer, Jayashree

AU - Obuchowski, Nancy A.

AU - Buckler, Andrew

AU - Athelogou, Maria

AU - Jarecha, Rudresh

AU - Petrick, Nicholas

AU - Pezeshk, Aria

AU - Sahiner, Berkman

AU - Samei, Ehsan

PY - 2019/7/1

Y1 - 2019/7/1

N2 - Rationale and Objectives: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. Materials and Methods: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). Results: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were −1.8 ± 1.4%, −2.5 ± 1.1%, −3 ± 1%, −1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%–16% and 7%–18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. Conclusion: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4%). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies.

AB - Rationale and Objectives: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. Materials and Methods: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). Results: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were −1.8 ± 1.4%, −2.5 ± 1.1%, −3 ± 1%, −1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%–16% and 7%–18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. Conclusion: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4%). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies.

KW - CT

KW - Hybrid dataset

KW - Lung cancer

KW - Quantitative imaging

KW - Segmentation

KW - Volumetry

UR - http://www.scopus.com/inward/record.url?scp=85066462518&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066462518&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2018.07.022

DO - 10.1016/j.acra.2018.07.022

M3 - Article

VL - 26

SP - e161-e173

JO - Academic Radiology

JF - Academic Radiology

SN - 1076-6332

IS - 7

ER -