Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills

Glenn Woodworth, Patricia (Patty) Carney, Joshua M. Cohen, Sandy L. Kopp, Lindsey E. Vokach-Brodsky, Jean Louis E Horn, Andres Missair, Shawn E. Banks, Nathan Dieckmann, Robert B. Maniker

Research output: Contribution to journalArticle

10 Citations (Scopus)

Abstract

Background Interpretation of ultrasound images and knowledge of anatomy are essential skills for ultrasound-guided peripheral nerve blocks. Competency-based educational models promoted by the Accreditation Council for Graduate Medical Education require the development of assessment tools for the achievement of different competency milestones to demonstrate the longitudinal development of skills that occur during training. Methods A rigorous study guided by psychometric principles was undertaken to identify and validate the domains and items in an assessment of ultrasound interpretation skills for regional anesthesia. A survey of residents, academic faculty, and community anesthesiologists, as well as video recordings of experts teaching ultrasound-guided peripheral nerve blocks, was used to develop short video clips with accompanying multiple choice-style questions. Four rounds of pilot testing produced a 50-question assessment that was subsequently administered online to residents, fellows, and faculty from multiple institutions. Results Test results from 90 participants were analyzed with Item Response Theory model fitting indicating that a 47-item subset of the test fits the model well (P = 0.11). There was a significant linear relation between expected and predicted item difficulty (P <0.001). Overall test scores increased linearly with higher levels of formal anesthesia training, regional anesthesia training, number of ultrasound-guided blocks performed per year, and a self-rating of regional anesthesia skill (all P <0.001). Conclusions This study provides evidence for the reliability, content validity, and construct validity of a 47-item multiple choice-style online test of ultrasound interpretation skills for regional anesthesia, which can be used as an assessment of competency milestone achievement in anesthesiology training.

Original languageEnglish (US)
Pages (from-to)306-314
Number of pages9
JournalRegional Anesthesia and Pain Medicine
Volume40
Issue number4
DOIs
StatePublished - Jul 11 2015

Fingerprint

Conduction Anesthesia
Nerve Block
Peripheral Nerves
Educational Models
Graduate Medical Education
Video Recording
Anesthesiology
Accreditation
Surgical Instruments
Psychometrics
Reproducibility of Results
Anatomy
Teaching
Anesthesia

ASJC Scopus subject areas

  • Anesthesiology and Pain Medicine

Cite this

Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills. / Woodworth, Glenn; Carney, Patricia (Patty); Cohen, Joshua M.; Kopp, Sandy L.; Vokach-Brodsky, Lindsey E.; Horn, Jean Louis E; Missair, Andres; Banks, Shawn E.; Dieckmann, Nathan; Maniker, Robert B.

In: Regional Anesthesia and Pain Medicine, Vol. 40, No. 4, 11.07.2015, p. 306-314.

Research output: Contribution to journalArticle

Woodworth, Glenn ; Carney, Patricia (Patty) ; Cohen, Joshua M. ; Kopp, Sandy L. ; Vokach-Brodsky, Lindsey E. ; Horn, Jean Louis E ; Missair, Andres ; Banks, Shawn E. ; Dieckmann, Nathan ; Maniker, Robert B. / Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills. In: Regional Anesthesia and Pain Medicine. 2015 ; Vol. 40, No. 4. pp. 306-314.
@article{45c07f39d59241b0aa31be63b3dea165,
title = "Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills",
abstract = "Background Interpretation of ultrasound images and knowledge of anatomy are essential skills for ultrasound-guided peripheral nerve blocks. Competency-based educational models promoted by the Accreditation Council for Graduate Medical Education require the development of assessment tools for the achievement of different competency milestones to demonstrate the longitudinal development of skills that occur during training. Methods A rigorous study guided by psychometric principles was undertaken to identify and validate the domains and items in an assessment of ultrasound interpretation skills for regional anesthesia. A survey of residents, academic faculty, and community anesthesiologists, as well as video recordings of experts teaching ultrasound-guided peripheral nerve blocks, was used to develop short video clips with accompanying multiple choice-style questions. Four rounds of pilot testing produced a 50-question assessment that was subsequently administered online to residents, fellows, and faculty from multiple institutions. Results Test results from 90 participants were analyzed with Item Response Theory model fitting indicating that a 47-item subset of the test fits the model well (P = 0.11). There was a significant linear relation between expected and predicted item difficulty (P <0.001). Overall test scores increased linearly with higher levels of formal anesthesia training, regional anesthesia training, number of ultrasound-guided blocks performed per year, and a self-rating of regional anesthesia skill (all P <0.001). Conclusions This study provides evidence for the reliability, content validity, and construct validity of a 47-item multiple choice-style online test of ultrasound interpretation skills for regional anesthesia, which can be used as an assessment of competency milestone achievement in anesthesiology training.",
author = "Glenn Woodworth and Carney, {Patricia (Patty)} and Cohen, {Joshua M.} and Kopp, {Sandy L.} and Vokach-Brodsky, {Lindsey E.} and Horn, {Jean Louis E} and Andres Missair and Banks, {Shawn E.} and Nathan Dieckmann and Maniker, {Robert B.}",
year = "2015",
month = "7",
day = "11",
doi = "10.1097/AAP.0000000000000236",
language = "English (US)",
volume = "40",
pages = "306--314",
journal = "Regional Anesthesia and Pain Medicine",
issn = "1098-7339",
publisher = "Lippincott Williams and Wilkins",
number = "4",

}

TY - JOUR

T1 - Development and Validation of an Assessment of Regional Anesthesia Ultrasound Interpretation Skills

AU - Woodworth, Glenn

AU - Carney, Patricia (Patty)

AU - Cohen, Joshua M.

AU - Kopp, Sandy L.

AU - Vokach-Brodsky, Lindsey E.

AU - Horn, Jean Louis E

AU - Missair, Andres

AU - Banks, Shawn E.

AU - Dieckmann, Nathan

AU - Maniker, Robert B.

PY - 2015/7/11

Y1 - 2015/7/11

N2 - Background Interpretation of ultrasound images and knowledge of anatomy are essential skills for ultrasound-guided peripheral nerve blocks. Competency-based educational models promoted by the Accreditation Council for Graduate Medical Education require the development of assessment tools for the achievement of different competency milestones to demonstrate the longitudinal development of skills that occur during training. Methods A rigorous study guided by psychometric principles was undertaken to identify and validate the domains and items in an assessment of ultrasound interpretation skills for regional anesthesia. A survey of residents, academic faculty, and community anesthesiologists, as well as video recordings of experts teaching ultrasound-guided peripheral nerve blocks, was used to develop short video clips with accompanying multiple choice-style questions. Four rounds of pilot testing produced a 50-question assessment that was subsequently administered online to residents, fellows, and faculty from multiple institutions. Results Test results from 90 participants were analyzed with Item Response Theory model fitting indicating that a 47-item subset of the test fits the model well (P = 0.11). There was a significant linear relation between expected and predicted item difficulty (P <0.001). Overall test scores increased linearly with higher levels of formal anesthesia training, regional anesthesia training, number of ultrasound-guided blocks performed per year, and a self-rating of regional anesthesia skill (all P <0.001). Conclusions This study provides evidence for the reliability, content validity, and construct validity of a 47-item multiple choice-style online test of ultrasound interpretation skills for regional anesthesia, which can be used as an assessment of competency milestone achievement in anesthesiology training.

AB - Background Interpretation of ultrasound images and knowledge of anatomy are essential skills for ultrasound-guided peripheral nerve blocks. Competency-based educational models promoted by the Accreditation Council for Graduate Medical Education require the development of assessment tools for the achievement of different competency milestones to demonstrate the longitudinal development of skills that occur during training. Methods A rigorous study guided by psychometric principles was undertaken to identify and validate the domains and items in an assessment of ultrasound interpretation skills for regional anesthesia. A survey of residents, academic faculty, and community anesthesiologists, as well as video recordings of experts teaching ultrasound-guided peripheral nerve blocks, was used to develop short video clips with accompanying multiple choice-style questions. Four rounds of pilot testing produced a 50-question assessment that was subsequently administered online to residents, fellows, and faculty from multiple institutions. Results Test results from 90 participants were analyzed with Item Response Theory model fitting indicating that a 47-item subset of the test fits the model well (P = 0.11). There was a significant linear relation between expected and predicted item difficulty (P <0.001). Overall test scores increased linearly with higher levels of formal anesthesia training, regional anesthesia training, number of ultrasound-guided blocks performed per year, and a self-rating of regional anesthesia skill (all P <0.001). Conclusions This study provides evidence for the reliability, content validity, and construct validity of a 47-item multiple choice-style online test of ultrasound interpretation skills for regional anesthesia, which can be used as an assessment of competency milestone achievement in anesthesiology training.

UR - http://www.scopus.com/inward/record.url?scp=84953860083&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84953860083&partnerID=8YFLogxK

U2 - 10.1097/AAP.0000000000000236

DO - 10.1097/AAP.0000000000000236

M3 - Article

C2 - 26017720

AN - SCOPUS:84953860083

VL - 40

SP - 306

EP - 314

JO - Regional Anesthesia and Pain Medicine

JF - Regional Anesthesia and Pain Medicine

SN - 1098-7339

IS - 4

ER -