Reliability of Surgeon-specific Reporting of Complications after Colectomy

Terry Shih, Adam I. Cole, Paul M. Al-Attar, Apurba Chakrabarti, Hussein A. Fardous, Peter F. Helvie, Michael T. Kemp, Chris Lee, Eytan Shtull-Leber, Darrell A. Campbell, Michael J. Englesbe

Research output: Contribution to journalArticle

14 Citations (Scopus)

Abstract

Objective: We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy. Background: Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality. Methods: We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment. Results: A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%-25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold. Conclusions: The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

Original languageEnglish (US)
Pages (from-to)920-925
Number of pages6
JournalAnnals of surgery
Volume261
Issue number5
DOIs
StatePublished - May 1 2015
Externally publishedYes

Fingerprint

Colectomy
Surgeons
Noise
Logistic Models

Keywords

  • colectomy
  • complications
  • health services research
  • outcomes
  • surgeon-specific reporting

ASJC Scopus subject areas

  • Surgery

Cite this

Shih, T., Cole, A. I., Al-Attar, P. M., Chakrabarti, A., Fardous, H. A., Helvie, P. F., ... Englesbe, M. J. (2015). Reliability of Surgeon-specific Reporting of Complications after Colectomy. Annals of surgery, 261(5), 920-925. https://doi.org/10.1097/SLA.0000000000001032

Reliability of Surgeon-specific Reporting of Complications after Colectomy. / Shih, Terry; Cole, Adam I.; Al-Attar, Paul M.; Chakrabarti, Apurba; Fardous, Hussein A.; Helvie, Peter F.; Kemp, Michael T.; Lee, Chris; Shtull-Leber, Eytan; Campbell, Darrell A.; Englesbe, Michael J.

In: Annals of surgery, Vol. 261, No. 5, 01.05.2015, p. 920-925.

Research output: Contribution to journalArticle

Shih, T, Cole, AI, Al-Attar, PM, Chakrabarti, A, Fardous, HA, Helvie, PF, Kemp, MT, Lee, C, Shtull-Leber, E, Campbell, DA & Englesbe, MJ 2015, 'Reliability of Surgeon-specific Reporting of Complications after Colectomy', Annals of surgery, vol. 261, no. 5, pp. 920-925. https://doi.org/10.1097/SLA.0000000000001032
Shih T, Cole AI, Al-Attar PM, Chakrabarti A, Fardous HA, Helvie PF et al. Reliability of Surgeon-specific Reporting of Complications after Colectomy. Annals of surgery. 2015 May 1;261(5):920-925. https://doi.org/10.1097/SLA.0000000000001032
Shih, Terry ; Cole, Adam I. ; Al-Attar, Paul M. ; Chakrabarti, Apurba ; Fardous, Hussein A. ; Helvie, Peter F. ; Kemp, Michael T. ; Lee, Chris ; Shtull-Leber, Eytan ; Campbell, Darrell A. ; Englesbe, Michael J. / Reliability of Surgeon-specific Reporting of Complications after Colectomy. In: Annals of surgery. 2015 ; Vol. 261, No. 5. pp. 920-925.
@article{5d38d31edc9a48f0b42e15a8e1cc1150,
title = "Reliability of Surgeon-specific Reporting of Complications after Colectomy",
abstract = "Objective: We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy. Background: Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality. Methods: We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment. Results: A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5{\%}. Approximately 86{\%} of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14{\%} represented true signal. Risk-adjusted complication rates varied from 0{\%} to 55.1{\%} across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4{\%}-25.6{\%}). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold. Conclusions: The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.",
keywords = "colectomy, complications, health services research, outcomes, surgeon-specific reporting",
author = "Terry Shih and Cole, {Adam I.} and Al-Attar, {Paul M.} and Apurba Chakrabarti and Fardous, {Hussein A.} and Helvie, {Peter F.} and Kemp, {Michael T.} and Chris Lee and Eytan Shtull-Leber and Campbell, {Darrell A.} and Englesbe, {Michael J.}",
year = "2015",
month = "5",
day = "1",
doi = "10.1097/SLA.0000000000001032",
language = "English (US)",
volume = "261",
pages = "920--925",
journal = "Annals of Surgery",
issn = "0003-4932",
publisher = "Lippincott Williams and Wilkins",
number = "5",

}

TY - JOUR

T1 - Reliability of Surgeon-specific Reporting of Complications after Colectomy

AU - Shih, Terry

AU - Cole, Adam I.

AU - Al-Attar, Paul M.

AU - Chakrabarti, Apurba

AU - Fardous, Hussein A.

AU - Helvie, Peter F.

AU - Kemp, Michael T.

AU - Lee, Chris

AU - Shtull-Leber, Eytan

AU - Campbell, Darrell A.

AU - Englesbe, Michael J.

PY - 2015/5/1

Y1 - 2015/5/1

N2 - Objective: We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy. Background: Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality. Methods: We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment. Results: A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%-25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold. Conclusions: The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

AB - Objective: We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy. Background: Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality. Methods: We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment. Results: A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%-25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold. Conclusions: The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

KW - colectomy

KW - complications

KW - health services research

KW - outcomes

KW - surgeon-specific reporting

UR - http://www.scopus.com/inward/record.url?scp=84943515243&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84943515243&partnerID=8YFLogxK

U2 - 10.1097/SLA.0000000000001032

DO - 10.1097/SLA.0000000000001032

M3 - Article

VL - 261

SP - 920

EP - 925

JO - Annals of Surgery

JF - Annals of Surgery

SN - 0003-4932

IS - 5

ER -