Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting: Validity and reliability

Gretchen M. Lentz, Lynn S. Mandel, David Lee, Carolyn Gardella, Jennifer Melville, Barbara A. Goff

Research output: Contribution to journalArticle

102 Citations (Scopus)

Abstract

OBJECTIVE: Resident surgical skills are acquired mainly through observing and later performing procedures in the operating room. Evaluation of surgical skills has traditionally been done through subjective faculty evaluation, a technique that has poor reliability and unknown validity. Our goal was to develop specific surgical tasks, both laparoscopic and open abdominal, that could be objectively and reliably evaluated in a bench laboratory setting. STUDY DESIGN: The prospective development of a reliable and valid resident surgical skills test in a bench laboratory setting was our goal. A written test of surgical knowledge and 12 skills tests were administered to 36 residents. Laparoscopic bench tasks were simulated with the use of a box and camera with a video display. Six laparoscopic tasks were assessed, including placing pegs on a board, running the bowel simulation, and other tasks that involve hand-eye coordination and manual dexterity. Open abdominal skills simulated incision closure, suturing a vaginal cuff, knot tying, and using a tie on a passer. Residents were timed at each given station and were given a rating score by 2 examiners. RESULTS: Knowledge scores showed a significant improvement by residency level. Assessment of construct validity (the ability to discriminate among residency levels) demonstrated significant differences on the rating of overall performance and individual tasks by level (determined by 1-way analysis of variance). Interrater reliability (agreement between 2 raters) with the use of intraclass correlation was 0.79 for the total score. The cost to administer the bench laboratory test was less than $50 and required 30 hours of faculty time. CONCLUSION: The results of this study suggest that surgical bench laboratory tasks can assess residents' surgical skills with good reliability and validity on most tasks. Our previous study, which used an animal laboratory, was expensive, and the bench laboratory model may provide an alternative means to assess surgical skills.

Original languageEnglish (US)
Pages (from-to)1462-1470
Number of pages9
JournalAmerican Journal of Obstetrics and Gynecology
Volume184
Issue number7
DOIs
StatePublished - 2001
Externally publishedYes

Fingerprint

Reproducibility of Results
Obstetrics
Internship and Residency
Aptitude
Laboratory Animals
Task Performance and Analysis
Operating Rooms
Running
Analysis of Variance
Hand
Costs and Cost Analysis

Keywords

  • Inanimate laboratory
  • Resident education
  • Surgical skills

ASJC Scopus subject areas

  • Medicine(all)
  • Obstetrics and Gynecology

Cite this

Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting : Validity and reliability. / Lentz, Gretchen M.; Mandel, Lynn S.; Lee, David; Gardella, Carolyn; Melville, Jennifer; Goff, Barbara A.

In: American Journal of Obstetrics and Gynecology, Vol. 184, No. 7, 2001, p. 1462-1470.

Research output: Contribution to journalArticle

Lentz, Gretchen M. ; Mandel, Lynn S. ; Lee, David ; Gardella, Carolyn ; Melville, Jennifer ; Goff, Barbara A. / Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting : Validity and reliability. In: American Journal of Obstetrics and Gynecology. 2001 ; Vol. 184, No. 7. pp. 1462-1470.
@article{5e45e0baaeb6447eab91de64bfb3a6e2,
title = "Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting: Validity and reliability",
abstract = "OBJECTIVE: Resident surgical skills are acquired mainly through observing and later performing procedures in the operating room. Evaluation of surgical skills has traditionally been done through subjective faculty evaluation, a technique that has poor reliability and unknown validity. Our goal was to develop specific surgical tasks, both laparoscopic and open abdominal, that could be objectively and reliably evaluated in a bench laboratory setting. STUDY DESIGN: The prospective development of a reliable and valid resident surgical skills test in a bench laboratory setting was our goal. A written test of surgical knowledge and 12 skills tests were administered to 36 residents. Laparoscopic bench tasks were simulated with the use of a box and camera with a video display. Six laparoscopic tasks were assessed, including placing pegs on a board, running the bowel simulation, and other tasks that involve hand-eye coordination and manual dexterity. Open abdominal skills simulated incision closure, suturing a vaginal cuff, knot tying, and using a tie on a passer. Residents were timed at each given station and were given a rating score by 2 examiners. RESULTS: Knowledge scores showed a significant improvement by residency level. Assessment of construct validity (the ability to discriminate among residency levels) demonstrated significant differences on the rating of overall performance and individual tasks by level (determined by 1-way analysis of variance). Interrater reliability (agreement between 2 raters) with the use of intraclass correlation was 0.79 for the total score. The cost to administer the bench laboratory test was less than $50 and required 30 hours of faculty time. CONCLUSION: The results of this study suggest that surgical bench laboratory tasks can assess residents' surgical skills with good reliability and validity on most tasks. Our previous study, which used an animal laboratory, was expensive, and the bench laboratory model may provide an alternative means to assess surgical skills.",
keywords = "Inanimate laboratory, Resident education, Surgical skills",
author = "Lentz, {Gretchen M.} and Mandel, {Lynn S.} and David Lee and Carolyn Gardella and Jennifer Melville and Goff, {Barbara A.}",
year = "2001",
doi = "10.1067/mob.2001.114850",
language = "English (US)",
volume = "184",
pages = "1462--1470",
journal = "American Journal of Obstetrics and Gynecology",
issn = "0002-9378",
publisher = "Mosby Inc.",
number = "7",

}

TY - JOUR

T1 - Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting

T2 - Validity and reliability

AU - Lentz, Gretchen M.

AU - Mandel, Lynn S.

AU - Lee, David

AU - Gardella, Carolyn

AU - Melville, Jennifer

AU - Goff, Barbara A.

PY - 2001

Y1 - 2001

N2 - OBJECTIVE: Resident surgical skills are acquired mainly through observing and later performing procedures in the operating room. Evaluation of surgical skills has traditionally been done through subjective faculty evaluation, a technique that has poor reliability and unknown validity. Our goal was to develop specific surgical tasks, both laparoscopic and open abdominal, that could be objectively and reliably evaluated in a bench laboratory setting. STUDY DESIGN: The prospective development of a reliable and valid resident surgical skills test in a bench laboratory setting was our goal. A written test of surgical knowledge and 12 skills tests were administered to 36 residents. Laparoscopic bench tasks were simulated with the use of a box and camera with a video display. Six laparoscopic tasks were assessed, including placing pegs on a board, running the bowel simulation, and other tasks that involve hand-eye coordination and manual dexterity. Open abdominal skills simulated incision closure, suturing a vaginal cuff, knot tying, and using a tie on a passer. Residents were timed at each given station and were given a rating score by 2 examiners. RESULTS: Knowledge scores showed a significant improvement by residency level. Assessment of construct validity (the ability to discriminate among residency levels) demonstrated significant differences on the rating of overall performance and individual tasks by level (determined by 1-way analysis of variance). Interrater reliability (agreement between 2 raters) with the use of intraclass correlation was 0.79 for the total score. The cost to administer the bench laboratory test was less than $50 and required 30 hours of faculty time. CONCLUSION: The results of this study suggest that surgical bench laboratory tasks can assess residents' surgical skills with good reliability and validity on most tasks. Our previous study, which used an animal laboratory, was expensive, and the bench laboratory model may provide an alternative means to assess surgical skills.

AB - OBJECTIVE: Resident surgical skills are acquired mainly through observing and later performing procedures in the operating room. Evaluation of surgical skills has traditionally been done through subjective faculty evaluation, a technique that has poor reliability and unknown validity. Our goal was to develop specific surgical tasks, both laparoscopic and open abdominal, that could be objectively and reliably evaluated in a bench laboratory setting. STUDY DESIGN: The prospective development of a reliable and valid resident surgical skills test in a bench laboratory setting was our goal. A written test of surgical knowledge and 12 skills tests were administered to 36 residents. Laparoscopic bench tasks were simulated with the use of a box and camera with a video display. Six laparoscopic tasks were assessed, including placing pegs on a board, running the bowel simulation, and other tasks that involve hand-eye coordination and manual dexterity. Open abdominal skills simulated incision closure, suturing a vaginal cuff, knot tying, and using a tie on a passer. Residents were timed at each given station and were given a rating score by 2 examiners. RESULTS: Knowledge scores showed a significant improvement by residency level. Assessment of construct validity (the ability to discriminate among residency levels) demonstrated significant differences on the rating of overall performance and individual tasks by level (determined by 1-way analysis of variance). Interrater reliability (agreement between 2 raters) with the use of intraclass correlation was 0.79 for the total score. The cost to administer the bench laboratory test was less than $50 and required 30 hours of faculty time. CONCLUSION: The results of this study suggest that surgical bench laboratory tasks can assess residents' surgical skills with good reliability and validity on most tasks. Our previous study, which used an animal laboratory, was expensive, and the bench laboratory model may provide an alternative means to assess surgical skills.

KW - Inanimate laboratory

KW - Resident education

KW - Surgical skills

UR - http://www.scopus.com/inward/record.url?scp=0034972666&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0034972666&partnerID=8YFLogxK

U2 - 10.1067/mob.2001.114850

DO - 10.1067/mob.2001.114850

M3 - Article

C2 - 11408869

AN - SCOPUS:0034972666

VL - 184

SP - 1462

EP - 1470

JO - American Journal of Obstetrics and Gynecology

JF - American Journal of Obstetrics and Gynecology

SN - 0002-9378

IS - 7

ER -