Do risk calculators accurately predict surgical site occurrences?

Ventral Hernia Outcome Collaborative

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

Introduction: Current risk assessment models for surgical site occurrence (SSO) and surgical site infection (SSI) after open ventral hernia repair (VHR) have limited external validation. Our aim was to determine (1) whether existing models stratify patients into groups by risk and (2) which model best predicts the rate of SSO and SSI. Methods: Patients who underwent open VHR and were followed for at least 1 mo were included. Using two data sets - a retrospective multicenter database (Ventral Hernia Outcomes Collaborative) and a single-center prospective database (Prospective) - each patient was assigned a predicted risk with each of the following models: Ventral Hernia Risk Score (VHRS), Ventral Hernia Working Group (VHWG), Centers for Disease Control and Prevention Wound Class, and Hernia Wound Risk Assessment Tool (HW-RAT). Patients in the Prospective database were also assigned a predicted risk from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Areas under the receiver operating characteristic curve (area under the curve [AUC]) were compared to assess the predictive accuracy of the models for SSO and SSI. Pearson's chi-square was used to determine which models were able to risk-stratify patients into groups with significantly differing rates of actual SSO and SSI. Results: The Ventral Hernia Outcomes Collaborative database (n = 795) had an overall SSO and SSI rate of 23% and 17%, respectively. The AUCs were low for SSO (0.56, 0.54, 0.52, and 0.60) and SSI (0.55, 0.53, 0.50, and 0.58). The VHRS (P = 0.01) and HW-RAT (P <0.01) significantly stratified patients into tiers for SSO, whereas the VHWG (P <0.05) and HW-RAT (P <0.05) stratified for SSI. In the Prospective database (n = 88), 14% and 8% developed an SSO and SSI, respectively. The AUCs were low for SSO (0.63, 0.54, 0.50, 0.57, and 0.69) and modest for SSI (0.81, 0.64, 0.55, 0.62, and 0.73). The ACS-NSQIP (P <0.01) stratified for SSO, whereas the VHRS (P <0.01) and ACS-NSQIP (P <0.05) stratified for SSI. In both databases, VHRS, VHWG, and Centers for Disease Control and Prevention overestimated risk of SSO and SSI, whereas HW-RAT and ACS-NSQIP underestimated risk for all groups. Conclusions: All five existing predictive models have limited ability to risk-stratify patients and accurately assess risk of SSO. However, both the VHRS and ACS-NSQIP demonstrate modest success in identifying patients at risk for SSI. Continued model refinement is needed to improve the two highest performing models (VHRS and ACS-NSQIP) along with investigation to determine whether modifications to perioperative management based on risk stratification can improve outcomes.

Original languageEnglish (US)
Pages (from-to)56-63
Number of pages8
JournalJournal of Surgical Research
Volume203
Issue number1
DOIs
StatePublished - Jun 1 2016

Fingerprint

Ventral Hernia
Surgical Wound Infection
Quality Improvement
Databases
Hernia
Wounds and Injuries
Area Under Curve
Anatomic Models
Herniorrhaphy
Centers for Disease Control and Prevention (U.S.)
ROC Curve

Keywords

  • NSQIP
  • Risk calculator
  • Surgical risk
  • Surgical site infection
  • Surgical site occurrence
  • Ventral hernia

ASJC Scopus subject areas

  • Surgery

Cite this

Do risk calculators accurately predict surgical site occurrences? / Ventral Hernia Outcome Collaborative.

In: Journal of Surgical Research, Vol. 203, No. 1, 01.06.2016, p. 56-63.

Research output: Contribution to journalArticle

Ventral Hernia Outcome Collaborative. / Do risk calculators accurately predict surgical site occurrences?. In: Journal of Surgical Research. 2016 ; Vol. 203, No. 1. pp. 56-63.
@article{1f1ccb7b70f34b5ab23e94ab556cb1fb,
title = "Do risk calculators accurately predict surgical site occurrences?",
abstract = "Introduction: Current risk assessment models for surgical site occurrence (SSO) and surgical site infection (SSI) after open ventral hernia repair (VHR) have limited external validation. Our aim was to determine (1) whether existing models stratify patients into groups by risk and (2) which model best predicts the rate of SSO and SSI. Methods: Patients who underwent open VHR and were followed for at least 1 mo were included. Using two data sets - a retrospective multicenter database (Ventral Hernia Outcomes Collaborative) and a single-center prospective database (Prospective) - each patient was assigned a predicted risk with each of the following models: Ventral Hernia Risk Score (VHRS), Ventral Hernia Working Group (VHWG), Centers for Disease Control and Prevention Wound Class, and Hernia Wound Risk Assessment Tool (HW-RAT). Patients in the Prospective database were also assigned a predicted risk from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Areas under the receiver operating characteristic curve (area under the curve [AUC]) were compared to assess the predictive accuracy of the models for SSO and SSI. Pearson's chi-square was used to determine which models were able to risk-stratify patients into groups with significantly differing rates of actual SSO and SSI. Results: The Ventral Hernia Outcomes Collaborative database (n = 795) had an overall SSO and SSI rate of 23{\%} and 17{\%}, respectively. The AUCs were low for SSO (0.56, 0.54, 0.52, and 0.60) and SSI (0.55, 0.53, 0.50, and 0.58). The VHRS (P = 0.01) and HW-RAT (P <0.01) significantly stratified patients into tiers for SSO, whereas the VHWG (P <0.05) and HW-RAT (P <0.05) stratified for SSI. In the Prospective database (n = 88), 14{\%} and 8{\%} developed an SSO and SSI, respectively. The AUCs were low for SSO (0.63, 0.54, 0.50, 0.57, and 0.69) and modest for SSI (0.81, 0.64, 0.55, 0.62, and 0.73). The ACS-NSQIP (P <0.01) stratified for SSO, whereas the VHRS (P <0.01) and ACS-NSQIP (P <0.05) stratified for SSI. In both databases, VHRS, VHWG, and Centers for Disease Control and Prevention overestimated risk of SSO and SSI, whereas HW-RAT and ACS-NSQIP underestimated risk for all groups. Conclusions: All five existing predictive models have limited ability to risk-stratify patients and accurately assess risk of SSO. However, both the VHRS and ACS-NSQIP demonstrate modest success in identifying patients at risk for SSI. Continued model refinement is needed to improve the two highest performing models (VHRS and ACS-NSQIP) along with investigation to determine whether modifications to perioperative management based on risk stratification can improve outcomes.",
keywords = "NSQIP, Risk calculator, Surgical risk, Surgical site infection, Surgical site occurrence, Ventral hernia",
author = "{Ventral Hernia Outcome Collaborative} and Mitchell, {Thomas O.} and Holihan, {Julie L.} and Askenasy, {Erik P.} and Greenberg, {Jacob A.} and Keith, {Jerrod N.} and Robert Martindale and Roth, {John Scott} and Liang, {Mike K.}",
year = "2016",
month = "6",
day = "1",
doi = "10.1016/j.jss.2016.03.040",
language = "English (US)",
volume = "203",
pages = "56--63",
journal = "Journal of Surgical Research",
issn = "0022-4804",
publisher = "Academic Press Inc.",
number = "1",

}

TY - JOUR

T1 - Do risk calculators accurately predict surgical site occurrences?

AU - Ventral Hernia Outcome Collaborative

AU - Mitchell, Thomas O.

AU - Holihan, Julie L.

AU - Askenasy, Erik P.

AU - Greenberg, Jacob A.

AU - Keith, Jerrod N.

AU - Martindale, Robert

AU - Roth, John Scott

AU - Liang, Mike K.

PY - 2016/6/1

Y1 - 2016/6/1

N2 - Introduction: Current risk assessment models for surgical site occurrence (SSO) and surgical site infection (SSI) after open ventral hernia repair (VHR) have limited external validation. Our aim was to determine (1) whether existing models stratify patients into groups by risk and (2) which model best predicts the rate of SSO and SSI. Methods: Patients who underwent open VHR and were followed for at least 1 mo were included. Using two data sets - a retrospective multicenter database (Ventral Hernia Outcomes Collaborative) and a single-center prospective database (Prospective) - each patient was assigned a predicted risk with each of the following models: Ventral Hernia Risk Score (VHRS), Ventral Hernia Working Group (VHWG), Centers for Disease Control and Prevention Wound Class, and Hernia Wound Risk Assessment Tool (HW-RAT). Patients in the Prospective database were also assigned a predicted risk from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Areas under the receiver operating characteristic curve (area under the curve [AUC]) were compared to assess the predictive accuracy of the models for SSO and SSI. Pearson's chi-square was used to determine which models were able to risk-stratify patients into groups with significantly differing rates of actual SSO and SSI. Results: The Ventral Hernia Outcomes Collaborative database (n = 795) had an overall SSO and SSI rate of 23% and 17%, respectively. The AUCs were low for SSO (0.56, 0.54, 0.52, and 0.60) and SSI (0.55, 0.53, 0.50, and 0.58). The VHRS (P = 0.01) and HW-RAT (P <0.01) significantly stratified patients into tiers for SSO, whereas the VHWG (P <0.05) and HW-RAT (P <0.05) stratified for SSI. In the Prospective database (n = 88), 14% and 8% developed an SSO and SSI, respectively. The AUCs were low for SSO (0.63, 0.54, 0.50, 0.57, and 0.69) and modest for SSI (0.81, 0.64, 0.55, 0.62, and 0.73). The ACS-NSQIP (P <0.01) stratified for SSO, whereas the VHRS (P <0.01) and ACS-NSQIP (P <0.05) stratified for SSI. In both databases, VHRS, VHWG, and Centers for Disease Control and Prevention overestimated risk of SSO and SSI, whereas HW-RAT and ACS-NSQIP underestimated risk for all groups. Conclusions: All five existing predictive models have limited ability to risk-stratify patients and accurately assess risk of SSO. However, both the VHRS and ACS-NSQIP demonstrate modest success in identifying patients at risk for SSI. Continued model refinement is needed to improve the two highest performing models (VHRS and ACS-NSQIP) along with investigation to determine whether modifications to perioperative management based on risk stratification can improve outcomes.

AB - Introduction: Current risk assessment models for surgical site occurrence (SSO) and surgical site infection (SSI) after open ventral hernia repair (VHR) have limited external validation. Our aim was to determine (1) whether existing models stratify patients into groups by risk and (2) which model best predicts the rate of SSO and SSI. Methods: Patients who underwent open VHR and were followed for at least 1 mo were included. Using two data sets - a retrospective multicenter database (Ventral Hernia Outcomes Collaborative) and a single-center prospective database (Prospective) - each patient was assigned a predicted risk with each of the following models: Ventral Hernia Risk Score (VHRS), Ventral Hernia Working Group (VHWG), Centers for Disease Control and Prevention Wound Class, and Hernia Wound Risk Assessment Tool (HW-RAT). Patients in the Prospective database were also assigned a predicted risk from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Areas under the receiver operating characteristic curve (area under the curve [AUC]) were compared to assess the predictive accuracy of the models for SSO and SSI. Pearson's chi-square was used to determine which models were able to risk-stratify patients into groups with significantly differing rates of actual SSO and SSI. Results: The Ventral Hernia Outcomes Collaborative database (n = 795) had an overall SSO and SSI rate of 23% and 17%, respectively. The AUCs were low for SSO (0.56, 0.54, 0.52, and 0.60) and SSI (0.55, 0.53, 0.50, and 0.58). The VHRS (P = 0.01) and HW-RAT (P <0.01) significantly stratified patients into tiers for SSO, whereas the VHWG (P <0.05) and HW-RAT (P <0.05) stratified for SSI. In the Prospective database (n = 88), 14% and 8% developed an SSO and SSI, respectively. The AUCs were low for SSO (0.63, 0.54, 0.50, 0.57, and 0.69) and modest for SSI (0.81, 0.64, 0.55, 0.62, and 0.73). The ACS-NSQIP (P <0.01) stratified for SSO, whereas the VHRS (P <0.01) and ACS-NSQIP (P <0.05) stratified for SSI. In both databases, VHRS, VHWG, and Centers for Disease Control and Prevention overestimated risk of SSO and SSI, whereas HW-RAT and ACS-NSQIP underestimated risk for all groups. Conclusions: All five existing predictive models have limited ability to risk-stratify patients and accurately assess risk of SSO. However, both the VHRS and ACS-NSQIP demonstrate modest success in identifying patients at risk for SSI. Continued model refinement is needed to improve the two highest performing models (VHRS and ACS-NSQIP) along with investigation to determine whether modifications to perioperative management based on risk stratification can improve outcomes.

KW - NSQIP

KW - Risk calculator

KW - Surgical risk

KW - Surgical site infection

KW - Surgical site occurrence

KW - Ventral hernia

UR - http://www.scopus.com/inward/record.url?scp=84965125514&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84965125514&partnerID=8YFLogxK

U2 - 10.1016/j.jss.2016.03.040

DO - 10.1016/j.jss.2016.03.040

M3 - Article

VL - 203

SP - 56

EP - 63

JO - Journal of Surgical Research

JF - Journal of Surgical Research

SN - 0022-4804

IS - 1

ER -