Analysis of model development strategies: predicting ventral hernia recurrence

Julie L. Holihan, Linda T. Li, Erik P. Askenasy, Jacob A. Greenberg, Jerrod N. Keith, Robert Martindale, J. Scott Roth, Mike K. Liang

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background There have been many attempts to identify variables associated with ventral hernia recurrence; however, it is unclear which statistical modeling approach results in models with greatest internal and external validity. We aim to assess the predictive accuracy of models developed using five common variable selection strategies to determine variables associated with hernia recurrence. Methods Two multicenter ventral hernia databases were used. Database 1 was randomly split into “development” and “internal validation” cohorts. Database 2 was designated “external validation”. The dependent variable for model development was hernia recurrence. Five variable selection strategies were used: (1) “clinical”—variables considered clinically relevant, (2) “selective stepwise”—all variables with a P value <0.20 were assessed in a step-backward model, (3) “liberal stepwise”—all variables were included and step-backward regression was performed, (4) “restrictive internal resampling,” and (5) “liberal internal resampling.” Variables were included with P < 0.05 for the Restrictive model and P < 0.10 for the Liberal model. A time-to-event analysis using Cox regression was performed using these strategies. The predictive accuracy of the developed models was tested on the internal and external validation cohorts using Harrell's C-statistic where C > 0.70 was considered “reasonable”. Results The recurrence rate was 32.9% (n = 173/526; median/range follow-up, 20/1-58 mo) for the development cohort, 36.0% (n = 95/264, median/range follow-up 20/1-61 mo) for the internal validation cohort, and 12.7% (n = 155/1224, median/range follow-up 9/1-50 mo) for the external validation cohort. Internal validation demonstrated reasonable predictive accuracy (C-statistics = 0.772, 0.760, 0.767, 0.757, 0.763), while on external validation, predictive accuracy dipped precipitously (C-statistic = 0.561, 0.557, 0.562, 0.553, 0.560). Conclusions Predictive accuracy was equally adequate on internal validation among models; however, on external validation, all five models failed to demonstrate utility. Future studies should report multiple variable selection techniques and demonstrate predictive accuracy on external data sets for model validation.

Original languageEnglish (US)
Pages (from-to)159-167
Number of pages9
JournalJournal of Surgical Research
Volume206
Issue number1
DOIs
StatePublished - Nov 1 2016

Fingerprint

Ventral Hernia
Recurrence
Databases
Hernia

Keywords

  • Bootstrapping
  • Hernia recurrence
  • Multivariate
  • Predictive model
  • Regression

ASJC Scopus subject areas

  • Surgery
  • Medicine(all)

Cite this

Holihan, J. L., Li, L. T., Askenasy, E. P., Greenberg, J. A., Keith, J. N., Martindale, R., ... Liang, M. K. (2016). Analysis of model development strategies: predicting ventral hernia recurrence. Journal of Surgical Research, 206(1), 159-167. https://doi.org/10.1016/j.jss.2016.07.042

Analysis of model development strategies : predicting ventral hernia recurrence. / Holihan, Julie L.; Li, Linda T.; Askenasy, Erik P.; Greenberg, Jacob A.; Keith, Jerrod N.; Martindale, Robert; Roth, J. Scott; Liang, Mike K.

In: Journal of Surgical Research, Vol. 206, No. 1, 01.11.2016, p. 159-167.

Research output: Contribution to journalArticle

Holihan, JL, Li, LT, Askenasy, EP, Greenberg, JA, Keith, JN, Martindale, R, Roth, JS & Liang, MK 2016, 'Analysis of model development strategies: predicting ventral hernia recurrence', Journal of Surgical Research, vol. 206, no. 1, pp. 159-167. https://doi.org/10.1016/j.jss.2016.07.042
Holihan, Julie L. ; Li, Linda T. ; Askenasy, Erik P. ; Greenberg, Jacob A. ; Keith, Jerrod N. ; Martindale, Robert ; Roth, J. Scott ; Liang, Mike K. / Analysis of model development strategies : predicting ventral hernia recurrence. In: Journal of Surgical Research. 2016 ; Vol. 206, No. 1. pp. 159-167.
@article{02985f8ddb5642ac89624bd4d3108dc7,
title = "Analysis of model development strategies: predicting ventral hernia recurrence",
abstract = "Background There have been many attempts to identify variables associated with ventral hernia recurrence; however, it is unclear which statistical modeling approach results in models with greatest internal and external validity. We aim to assess the predictive accuracy of models developed using five common variable selection strategies to determine variables associated with hernia recurrence. Methods Two multicenter ventral hernia databases were used. Database 1 was randomly split into “development” and “internal validation” cohorts. Database 2 was designated “external validation”. The dependent variable for model development was hernia recurrence. Five variable selection strategies were used: (1) “clinical”—variables considered clinically relevant, (2) “selective stepwise”—all variables with a P value <0.20 were assessed in a step-backward model, (3) “liberal stepwise”—all variables were included and step-backward regression was performed, (4) “restrictive internal resampling,” and (5) “liberal internal resampling.” Variables were included with P < 0.05 for the Restrictive model and P < 0.10 for the Liberal model. A time-to-event analysis using Cox regression was performed using these strategies. The predictive accuracy of the developed models was tested on the internal and external validation cohorts using Harrell's C-statistic where C > 0.70 was considered “reasonable”. Results The recurrence rate was 32.9{\%} (n = 173/526; median/range follow-up, 20/1-58 mo) for the development cohort, 36.0{\%} (n = 95/264, median/range follow-up 20/1-61 mo) for the internal validation cohort, and 12.7{\%} (n = 155/1224, median/range follow-up 9/1-50 mo) for the external validation cohort. Internal validation demonstrated reasonable predictive accuracy (C-statistics = 0.772, 0.760, 0.767, 0.757, 0.763), while on external validation, predictive accuracy dipped precipitously (C-statistic = 0.561, 0.557, 0.562, 0.553, 0.560). Conclusions Predictive accuracy was equally adequate on internal validation among models; however, on external validation, all five models failed to demonstrate utility. Future studies should report multiple variable selection techniques and demonstrate predictive accuracy on external data sets for model validation.",
keywords = "Bootstrapping, Hernia recurrence, Multivariate, Predictive model, Regression",
author = "Holihan, {Julie L.} and Li, {Linda T.} and Askenasy, {Erik P.} and Greenberg, {Jacob A.} and Keith, {Jerrod N.} and Robert Martindale and Roth, {J. Scott} and Liang, {Mike K.}",
year = "2016",
month = "11",
day = "1",
doi = "10.1016/j.jss.2016.07.042",
language = "English (US)",
volume = "206",
pages = "159--167",
journal = "Journal of Surgical Research",
issn = "0022-4804",
publisher = "Academic Press Inc.",
number = "1",

}

TY - JOUR

T1 - Analysis of model development strategies

T2 - predicting ventral hernia recurrence

AU - Holihan, Julie L.

AU - Li, Linda T.

AU - Askenasy, Erik P.

AU - Greenberg, Jacob A.

AU - Keith, Jerrod N.

AU - Martindale, Robert

AU - Roth, J. Scott

AU - Liang, Mike K.

PY - 2016/11/1

Y1 - 2016/11/1

N2 - Background There have been many attempts to identify variables associated with ventral hernia recurrence; however, it is unclear which statistical modeling approach results in models with greatest internal and external validity. We aim to assess the predictive accuracy of models developed using five common variable selection strategies to determine variables associated with hernia recurrence. Methods Two multicenter ventral hernia databases were used. Database 1 was randomly split into “development” and “internal validation” cohorts. Database 2 was designated “external validation”. The dependent variable for model development was hernia recurrence. Five variable selection strategies were used: (1) “clinical”—variables considered clinically relevant, (2) “selective stepwise”—all variables with a P value <0.20 were assessed in a step-backward model, (3) “liberal stepwise”—all variables were included and step-backward regression was performed, (4) “restrictive internal resampling,” and (5) “liberal internal resampling.” Variables were included with P < 0.05 for the Restrictive model and P < 0.10 for the Liberal model. A time-to-event analysis using Cox regression was performed using these strategies. The predictive accuracy of the developed models was tested on the internal and external validation cohorts using Harrell's C-statistic where C > 0.70 was considered “reasonable”. Results The recurrence rate was 32.9% (n = 173/526; median/range follow-up, 20/1-58 mo) for the development cohort, 36.0% (n = 95/264, median/range follow-up 20/1-61 mo) for the internal validation cohort, and 12.7% (n = 155/1224, median/range follow-up 9/1-50 mo) for the external validation cohort. Internal validation demonstrated reasonable predictive accuracy (C-statistics = 0.772, 0.760, 0.767, 0.757, 0.763), while on external validation, predictive accuracy dipped precipitously (C-statistic = 0.561, 0.557, 0.562, 0.553, 0.560). Conclusions Predictive accuracy was equally adequate on internal validation among models; however, on external validation, all five models failed to demonstrate utility. Future studies should report multiple variable selection techniques and demonstrate predictive accuracy on external data sets for model validation.

AB - Background There have been many attempts to identify variables associated with ventral hernia recurrence; however, it is unclear which statistical modeling approach results in models with greatest internal and external validity. We aim to assess the predictive accuracy of models developed using five common variable selection strategies to determine variables associated with hernia recurrence. Methods Two multicenter ventral hernia databases were used. Database 1 was randomly split into “development” and “internal validation” cohorts. Database 2 was designated “external validation”. The dependent variable for model development was hernia recurrence. Five variable selection strategies were used: (1) “clinical”—variables considered clinically relevant, (2) “selective stepwise”—all variables with a P value <0.20 were assessed in a step-backward model, (3) “liberal stepwise”—all variables were included and step-backward regression was performed, (4) “restrictive internal resampling,” and (5) “liberal internal resampling.” Variables were included with P < 0.05 for the Restrictive model and P < 0.10 for the Liberal model. A time-to-event analysis using Cox regression was performed using these strategies. The predictive accuracy of the developed models was tested on the internal and external validation cohorts using Harrell's C-statistic where C > 0.70 was considered “reasonable”. Results The recurrence rate was 32.9% (n = 173/526; median/range follow-up, 20/1-58 mo) for the development cohort, 36.0% (n = 95/264, median/range follow-up 20/1-61 mo) for the internal validation cohort, and 12.7% (n = 155/1224, median/range follow-up 9/1-50 mo) for the external validation cohort. Internal validation demonstrated reasonable predictive accuracy (C-statistics = 0.772, 0.760, 0.767, 0.757, 0.763), while on external validation, predictive accuracy dipped precipitously (C-statistic = 0.561, 0.557, 0.562, 0.553, 0.560). Conclusions Predictive accuracy was equally adequate on internal validation among models; however, on external validation, all five models failed to demonstrate utility. Future studies should report multiple variable selection techniques and demonstrate predictive accuracy on external data sets for model validation.

KW - Bootstrapping

KW - Hernia recurrence

KW - Multivariate

KW - Predictive model

KW - Regression

UR - http://www.scopus.com/inward/record.url?scp=84983449231&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84983449231&partnerID=8YFLogxK

U2 - 10.1016/j.jss.2016.07.042

DO - 10.1016/j.jss.2016.07.042

M3 - Article

C2 - 27916356

AN - SCOPUS:84983449231

VL - 206

SP - 159

EP - 167

JO - Journal of Surgical Research

JF - Journal of Surgical Research

SN - 0022-4804

IS - 1

ER -