Development of a diagnostic test set to assess agreement in breast pathology: Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)

Natalia V. Oster, Patricia (Patty) Carney, Kimberly H. Allison, Donald L. Weaver, Lisa M. Reisch, Gary Longton, Tracy Onega, Margaret Pepe, Berta M. Geller, Heidi Nelson, Tyler R. Ross, N. Anna Tosteson, Joann G. Elmore

Research output: Contribution to journalArticle

29 Citations (Scopus)

Abstract

Background: Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.Methods: Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.Conclusions: We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

Original languageEnglish (US)
Article number3
JournalBMC Women's Health
Volume13
Issue number1
DOIs
StatePublished - Feb 5 2013

Fingerprint

Routine Diagnostic Tests
Breast
Guidelines
Pathology
Carcinoma, Intraductal, Noninfiltrating
Biopsy
National Cancer Institute (U.S.)
Reproducibility of Results
Breast Neoplasms
Carcinoma
Research
Population
Pathologists

Keywords

  • Agreement studies
  • Breast
  • Diagnostic techniques
  • Pathology
  • Reliability of results
  • Reporting guidelines

ASJC Scopus subject areas

  • Medicine(all)
  • Obstetrics and Gynecology
  • Reproductive Medicine

Cite this

Development of a diagnostic test set to assess agreement in breast pathology : Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). / Oster, Natalia V.; Carney, Patricia (Patty); Allison, Kimberly H.; Weaver, Donald L.; Reisch, Lisa M.; Longton, Gary; Onega, Tracy; Pepe, Margaret; Geller, Berta M.; Nelson, Heidi; Ross, Tyler R.; Tosteson, N. Anna; Elmore, Joann G.

In: BMC Women's Health, Vol. 13, No. 1, 3, 05.02.2013.

Research output: Contribution to journalArticle

Oster, Natalia V. ; Carney, Patricia (Patty) ; Allison, Kimberly H. ; Weaver, Donald L. ; Reisch, Lisa M. ; Longton, Gary ; Onega, Tracy ; Pepe, Margaret ; Geller, Berta M. ; Nelson, Heidi ; Ross, Tyler R. ; Tosteson, N. Anna ; Elmore, Joann G. / Development of a diagnostic test set to assess agreement in breast pathology : Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). In: BMC Women's Health. 2013 ; Vol. 13, No. 1.
@article{b7e1de731dfc4cc19490ec1fe090e56b,
title = "Development of a diagnostic test set to assess agreement in breast pathology: Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)",
abstract = "Background: Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.Methods: Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.Conclusions: We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.",
keywords = "Agreement studies, Breast, Diagnostic techniques, Pathology, Reliability of results, Reporting guidelines",
author = "Oster, {Natalia V.} and Carney, {Patricia (Patty)} and Allison, {Kimberly H.} and Weaver, {Donald L.} and Reisch, {Lisa M.} and Gary Longton and Tracy Onega and Margaret Pepe and Geller, {Berta M.} and Heidi Nelson and Ross, {Tyler R.} and Tosteson, {N. Anna} and Elmore, {Joann G.}",
year = "2013",
month = "2",
day = "5",
doi = "10.1186/1472-6874-13-3",
language = "English (US)",
volume = "13",
journal = "BMC Women's Health",
issn = "1472-6874",
publisher = "BioMed Central",
number = "1",

}

TY - JOUR

T1 - Development of a diagnostic test set to assess agreement in breast pathology

T2 - Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)

AU - Oster, Natalia V.

AU - Carney, Patricia (Patty)

AU - Allison, Kimberly H.

AU - Weaver, Donald L.

AU - Reisch, Lisa M.

AU - Longton, Gary

AU - Onega, Tracy

AU - Pepe, Margaret

AU - Geller, Berta M.

AU - Nelson, Heidi

AU - Ross, Tyler R.

AU - Tosteson, N. Anna

AU - Elmore, Joann G.

PY - 2013/2/5

Y1 - 2013/2/5

N2 - Background: Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.Methods: Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.Conclusions: We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

AB - Background: Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.Methods: Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.Conclusions: We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

KW - Agreement studies

KW - Breast

KW - Diagnostic techniques

KW - Pathology

KW - Reliability of results

KW - Reporting guidelines

UR - http://www.scopus.com/inward/record.url?scp=84873260151&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84873260151&partnerID=8YFLogxK

U2 - 10.1186/1472-6874-13-3

DO - 10.1186/1472-6874-13-3

M3 - Article

C2 - 23379630

AN - SCOPUS:84873260151

VL - 13

JO - BMC Women's Health

JF - BMC Women's Health

SN - 1472-6874

IS - 1

M1 - 3

ER -