Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation

Musaddiq Awan, Jayashree Kalpathy-Cramer, G. Brandon Gunn, Beth M. Beadle, Adam S. Garden, Jack Phan, Emma Holliday, William E. Jones, Elizabeth Maani, Abhilasha Patel, Jehee Choi, Virginia Clyburn, Bundhit Tantiwongkosi, David I. Rosenthal, Clifton D. Fuller

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Purpose: A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software-based feedback intervention to aid in contouring of OARs in the head and neck. Methods and Materials: The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 OARs on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 OARs on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 OARs on both scans. A simultaneous truth and performance level estimation (STAPLE) composite of the expert contours was used as a gold-standard set for analysis of OAR contouring. Results: Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice similarity coefficients were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean dice similarity coefficients across all structures increased between phase 1 and phase 2 for each resident observer, demonstrating a statistically significant improvement in overall OAR-contouring ability (P <.01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (P ≤ .05) including all otic structures and suprahyoid lymph node levels of the head and neck. Conclusions: Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.

Original languageEnglish (US)
Pages (from-to)186-193
Number of pages8
JournalPractical Radiation Oncology
Volume3
Issue number3
DOIs
StatePublished - Jul 2013

Fingerprint

Organs at Risk
Atlases
Neck
Software
Lymph Nodes
Head
Tomography
Gold
Ear
Teaching

ASJC Scopus subject areas

  • Oncology
  • Radiology Nuclear Medicine and imaging

Cite this

Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck : Quantitative assessment of conformance to expert delineation. / Awan, Musaddiq; Kalpathy-Cramer, Jayashree; Gunn, G. Brandon; Beadle, Beth M.; Garden, Adam S.; Phan, Jack; Holliday, Emma; Jones, William E.; Maani, Elizabeth; Patel, Abhilasha; Choi, Jehee; Clyburn, Virginia; Tantiwongkosi, Bundhit; Rosenthal, David I.; Fuller, Clifton D.

In: Practical Radiation Oncology, Vol. 3, No. 3, 07.2013, p. 186-193.

Research output: Contribution to journalArticle

Awan, M, Kalpathy-Cramer, J, Gunn, GB, Beadle, BM, Garden, AS, Phan, J, Holliday, E, Jones, WE, Maani, E, Patel, A, Choi, J, Clyburn, V, Tantiwongkosi, B, Rosenthal, DI & Fuller, CD 2013, 'Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation', Practical Radiation Oncology, vol. 3, no. 3, pp. 186-193. https://doi.org/10.1016/j.prro.2012.11.002
Awan, Musaddiq ; Kalpathy-Cramer, Jayashree ; Gunn, G. Brandon ; Beadle, Beth M. ; Garden, Adam S. ; Phan, Jack ; Holliday, Emma ; Jones, William E. ; Maani, Elizabeth ; Patel, Abhilasha ; Choi, Jehee ; Clyburn, Virginia ; Tantiwongkosi, Bundhit ; Rosenthal, David I. ; Fuller, Clifton D. / Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck : Quantitative assessment of conformance to expert delineation. In: Practical Radiation Oncology. 2013 ; Vol. 3, No. 3. pp. 186-193.
@article{2452bee39291428184029416ad3e4d21,
title = "Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation",
abstract = "Purpose: A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software-based feedback intervention to aid in contouring of OARs in the head and neck. Methods and Materials: The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 OARs on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 OARs on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 OARs on both scans. A simultaneous truth and performance level estimation (STAPLE) composite of the expert contours was used as a gold-standard set for analysis of OAR contouring. Results: Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice similarity coefficients were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean dice similarity coefficients across all structures increased between phase 1 and phase 2 for each resident observer, demonstrating a statistically significant improvement in overall OAR-contouring ability (P <.01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (P ≤ .05) including all otic structures and suprahyoid lymph node levels of the head and neck. Conclusions: Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.",
author = "Musaddiq Awan and Jayashree Kalpathy-Cramer and Gunn, {G. Brandon} and Beadle, {Beth M.} and Garden, {Adam S.} and Jack Phan and Emma Holliday and Jones, {William E.} and Elizabeth Maani and Abhilasha Patel and Jehee Choi and Virginia Clyburn and Bundhit Tantiwongkosi and Rosenthal, {David I.} and Fuller, {Clifton D.}",
year = "2013",
month = "7",
doi = "10.1016/j.prro.2012.11.002",
language = "English (US)",
volume = "3",
pages = "186--193",
journal = "Practical Radiation Oncology",
issn = "1879-8500",
publisher = "Elsevier BV",
number = "3",

}

TY - JOUR

T1 - Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck

T2 - Quantitative assessment of conformance to expert delineation

AU - Awan, Musaddiq

AU - Kalpathy-Cramer, Jayashree

AU - Gunn, G. Brandon

AU - Beadle, Beth M.

AU - Garden, Adam S.

AU - Phan, Jack

AU - Holliday, Emma

AU - Jones, William E.

AU - Maani, Elizabeth

AU - Patel, Abhilasha

AU - Choi, Jehee

AU - Clyburn, Virginia

AU - Tantiwongkosi, Bundhit

AU - Rosenthal, David I.

AU - Fuller, Clifton D.

PY - 2013/7

Y1 - 2013/7

N2 - Purpose: A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software-based feedback intervention to aid in contouring of OARs in the head and neck. Methods and Materials: The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 OARs on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 OARs on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 OARs on both scans. A simultaneous truth and performance level estimation (STAPLE) composite of the expert contours was used as a gold-standard set for analysis of OAR contouring. Results: Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice similarity coefficients were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean dice similarity coefficients across all structures increased between phase 1 and phase 2 for each resident observer, demonstrating a statistically significant improvement in overall OAR-contouring ability (P <.01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (P ≤ .05) including all otic structures and suprahyoid lymph node levels of the head and neck. Conclusions: Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.

AB - Purpose: A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software-based feedback intervention to aid in contouring of OARs in the head and neck. Methods and Materials: The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 OARs on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 OARs on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 OARs on both scans. A simultaneous truth and performance level estimation (STAPLE) composite of the expert contours was used as a gold-standard set for analysis of OAR contouring. Results: Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice similarity coefficients were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean dice similarity coefficients across all structures increased between phase 1 and phase 2 for each resident observer, demonstrating a statistically significant improvement in overall OAR-contouring ability (P <.01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (P ≤ .05) including all otic structures and suprahyoid lymph node levels of the head and neck. Conclusions: Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.

UR - http://www.scopus.com/inward/record.url?scp=84879888975&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84879888975&partnerID=8YFLogxK

U2 - 10.1016/j.prro.2012.11.002

DO - 10.1016/j.prro.2012.11.002

M3 - Article

AN - SCOPUS:84879888975

VL - 3

SP - 186

EP - 193

JO - Practical Radiation Oncology

JF - Practical Radiation Oncology

SN - 1879-8500

IS - 3

ER -