Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211

Mathieu Hatt, John A. Lee, Charles R. Schmidtlein, Issam El Naqa, Curtis Caldwell, Elisabetta De Bernardi, Wei Lu, Shiva Das, Xavier Geets, Vincent Gregoire, Robert Jeraj, Michael P. MacManus, Osama R. Mawlawi, Ursula Nestle, Andrei Pugachev, Heiko Schöder, Tony Shepherd, Emiliano Spezi, Dimitris Visvikis, Habib ZaidiAssen S. Kirov

Research output: Contribution to journalReview article

55 Citations (Scopus)

Abstract

Purpose: The purpose of this educational report is to provide an overview of the present stateof-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application. Approach: A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed. Findings: A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol. Conclusions: Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.

Original languageEnglish (US)
Pages (from-to)e1-e42
JournalMedical Physics
Volume44
Issue number6
DOIs
StatePublished - Jun 1 2017
Externally publishedYes

Fingerprint

Benchmarking
Computer-Assisted Image Processing
Signal-To-Noise Ratio
Art
Uncertainty
Neoplasms
Physicians
Datasets

Keywords

  • PET segmentation
  • PET/CT
  • Treatment assessment
  • Treatment planning

ASJC Scopus subject areas

  • Biophysics
  • Radiology Nuclear Medicine and imaging

Cite this

Hatt, M., Lee, J. A., Schmidtlein, C. R., El Naqa, I., Caldwell, C., De Bernardi, E., ... Kirov, A. S. (2017). Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211. Medical Physics, 44(6), e1-e42. https://doi.org/10.1002/mp.12124

Classification and evaluation strategies of auto-segmentation approaches for PET : Report of AAPM task group No. 211. / Hatt, Mathieu; Lee, John A.; Schmidtlein, Charles R.; El Naqa, Issam; Caldwell, Curtis; De Bernardi, Elisabetta; Lu, Wei; Das, Shiva; Geets, Xavier; Gregoire, Vincent; Jeraj, Robert; MacManus, Michael P.; Mawlawi, Osama R.; Nestle, Ursula; Pugachev, Andrei; Schöder, Heiko; Shepherd, Tony; Spezi, Emiliano; Visvikis, Dimitris; Zaidi, Habib; Kirov, Assen S.

In: Medical Physics, Vol. 44, No. 6, 01.06.2017, p. e1-e42.

Research output: Contribution to journalReview article

Hatt, M, Lee, JA, Schmidtlein, CR, El Naqa, I, Caldwell, C, De Bernardi, E, Lu, W, Das, S, Geets, X, Gregoire, V, Jeraj, R, MacManus, MP, Mawlawi, OR, Nestle, U, Pugachev, A, Schöder, H, Shepherd, T, Spezi, E, Visvikis, D, Zaidi, H & Kirov, AS 2017, 'Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211', Medical Physics, vol. 44, no. 6, pp. e1-e42. https://doi.org/10.1002/mp.12124
Hatt, Mathieu ; Lee, John A. ; Schmidtlein, Charles R. ; El Naqa, Issam ; Caldwell, Curtis ; De Bernardi, Elisabetta ; Lu, Wei ; Das, Shiva ; Geets, Xavier ; Gregoire, Vincent ; Jeraj, Robert ; MacManus, Michael P. ; Mawlawi, Osama R. ; Nestle, Ursula ; Pugachev, Andrei ; Schöder, Heiko ; Shepherd, Tony ; Spezi, Emiliano ; Visvikis, Dimitris ; Zaidi, Habib ; Kirov, Assen S. / Classification and evaluation strategies of auto-segmentation approaches for PET : Report of AAPM task group No. 211. In: Medical Physics. 2017 ; Vol. 44, No. 6. pp. e1-e42.
@article{b1b8fb4ab54d45a19820b2377e48b84a,
title = "Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211",
abstract = "Purpose: The purpose of this educational report is to provide an overview of the present stateof-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application. Approach: A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed. Findings: A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol. Conclusions: Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.",
keywords = "PET segmentation, PET/CT, Treatment assessment, Treatment planning",
author = "Mathieu Hatt and Lee, {John A.} and Schmidtlein, {Charles R.} and {El Naqa}, Issam and Curtis Caldwell and {De Bernardi}, Elisabetta and Wei Lu and Shiva Das and Xavier Geets and Vincent Gregoire and Robert Jeraj and MacManus, {Michael P.} and Mawlawi, {Osama R.} and Ursula Nestle and Andrei Pugachev and Heiko Sch{\"o}der and Tony Shepherd and Emiliano Spezi and Dimitris Visvikis and Habib Zaidi and Kirov, {Assen S.}",
year = "2017",
month = "6",
day = "1",
doi = "10.1002/mp.12124",
language = "English (US)",
volume = "44",
pages = "e1--e42",
journal = "Medical Physics",
issn = "0094-2405",
publisher = "AAPM - American Association of Physicists in Medicine",
number = "6",

}

TY - JOUR

T1 - Classification and evaluation strategies of auto-segmentation approaches for PET

T2 - Report of AAPM task group No. 211

AU - Hatt, Mathieu

AU - Lee, John A.

AU - Schmidtlein, Charles R.

AU - El Naqa, Issam

AU - Caldwell, Curtis

AU - De Bernardi, Elisabetta

AU - Lu, Wei

AU - Das, Shiva

AU - Geets, Xavier

AU - Gregoire, Vincent

AU - Jeraj, Robert

AU - MacManus, Michael P.

AU - Mawlawi, Osama R.

AU - Nestle, Ursula

AU - Pugachev, Andrei

AU - Schöder, Heiko

AU - Shepherd, Tony

AU - Spezi, Emiliano

AU - Visvikis, Dimitris

AU - Zaidi, Habib

AU - Kirov, Assen S.

PY - 2017/6/1

Y1 - 2017/6/1

N2 - Purpose: The purpose of this educational report is to provide an overview of the present stateof-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application. Approach: A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed. Findings: A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol. Conclusions: Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.

AB - Purpose: The purpose of this educational report is to provide an overview of the present stateof-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application. Approach: A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed. Findings: A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol. Conclusions: Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.

KW - PET segmentation

KW - PET/CT

KW - Treatment assessment

KW - Treatment planning

UR - http://www.scopus.com/inward/record.url?scp=85024498717&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85024498717&partnerID=8YFLogxK

U2 - 10.1002/mp.12124

DO - 10.1002/mp.12124

M3 - Review article

C2 - 28120467

AN - SCOPUS:85024498717

VL - 44

SP - e1-e42

JO - Medical Physics

JF - Medical Physics

SN - 0094-2405

IS - 6

ER -