Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones

A 2-Year Validation

Kenneth Gundle, Dayne T. Mickelson, Arien Cherones, Jason Black, Doug P. Hanel

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Objective To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents. Setting Single academic medical center, including a trauma center and pediatrics tertiary hospital. Participants Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators. Methods Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time. Results Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51% (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01). Conclusions Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.

Original languageEnglish (US)
Pages (from-to)1116-1123
Number of pages8
JournalJournal of Surgical Education
Volume74
Issue number6
DOIs
StatePublished - Nov 1 2017

Fingerprint

varespladib methyl
patient care
surgery
Orthopedics
Patient Care
resident
evaluation
Diagnostic Self Evaluation
trainee
Pediatric Hospitals
Accreditation
Trauma Centers
Internship and Residency
Nonparametric Statistics
Information Systems
Tertiary Care Centers
self-study
accreditation
trauma

Keywords

  • competency
  • feedback
  • next accreditation system
  • Patient Care
  • Practice-Based Learning and Improvement
  • validation
  • web-based

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones : A 2-Year Validation. / Gundle, Kenneth; Mickelson, Dayne T.; Cherones, Arien; Black, Jason; Hanel, Doug P.

In: Journal of Surgical Education, Vol. 74, No. 6, 01.11.2017, p. 1116-1123.

Research output: Contribution to journalArticle

Gundle, Kenneth ; Mickelson, Dayne T. ; Cherones, Arien ; Black, Jason ; Hanel, Doug P. / Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones : A 2-Year Validation. In: Journal of Surgical Education. 2017 ; Vol. 74, No. 6. pp. 1116-1123.
@article{0948e8a5bb6349e5a77ad317ad6af478,
title = "Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones: A 2-Year Validation",
abstract = "Objective To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents. Setting Single academic medical center, including a trauma center and pediatrics tertiary hospital. Participants Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators. Methods Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time. Results Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51{\%} (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01). Conclusions Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.",
keywords = "competency, feedback, next accreditation system, Patient Care, Practice-Based Learning and Improvement, validation, web-based",
author = "Kenneth Gundle and Mickelson, {Dayne T.} and Arien Cherones and Jason Black and Hanel, {Doug P.}",
year = "2017",
month = "11",
day = "1",
doi = "10.1016/j.jsurg.2017.05.001",
language = "English (US)",
volume = "74",
pages = "1116--1123",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "6",

}

TY - JOUR

T1 - Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones

T2 - A 2-Year Validation

AU - Gundle, Kenneth

AU - Mickelson, Dayne T.

AU - Cherones, Arien

AU - Black, Jason

AU - Hanel, Doug P.

PY - 2017/11/1

Y1 - 2017/11/1

N2 - Objective To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents. Setting Single academic medical center, including a trauma center and pediatrics tertiary hospital. Participants Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators. Methods Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time. Results Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51% (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01). Conclusions Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.

AB - Objective To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents. Setting Single academic medical center, including a trauma center and pediatrics tertiary hospital. Participants Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators. Methods Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time. Results Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51% (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01). Conclusions Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.

KW - competency

KW - feedback

KW - next accreditation system

KW - Patient Care

KW - Practice-Based Learning and Improvement

KW - validation

KW - web-based

UR - http://www.scopus.com/inward/record.url?scp=85019369542&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85019369542&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2017.05.001

DO - 10.1016/j.jsurg.2017.05.001

M3 - Article

VL - 74

SP - 1116

EP - 1123

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 6

ER -