Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings

Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group

Research output: Contribution to journalArticle

Abstract

Purpose To perform a derivation study to determine in which subcompetencies marginal/unsatisfactory pediatric residents had the greatest deficits compared with their satisfactorily performing peers and which subcompetencies best discriminated between marginal/unsatisfactory and satisfactorily performing residents. Method Multi-institutional cohort study of all 21 milestones (rated on four or five levels) reported to the Accreditation Council for Graduate Medical Education, and global marginal/unsatisfactory versus satisfactory performance reported to the American Board of Pediatrics. Data were gathered in 2013-2014. For each level of training (postgraduate year [PGY] 1, 2, and 3), mean differences between milestone levels of residents with marginal/unsatisfactory and satisfactory performance adjusted for clustering by program and C-statistics (area under receiver operating characteristic curve) were calculated. A Bonferroni-corrected significance threshold of.0007963 was used to account for multiple comparisons. Results Milestone and overall performance evaluations for 1,704 pediatric residents in 41 programs were obtained. For PGY1s, two subcompetencies had almost a one-point difference in milestone levels between marginal/unsatisfactory and satisfactory trainees and outstanding discrimination (≥ 0.90): organize/prioritize (0.93; C-statistic: 0.91) and transfer of care (0.97; C-statistic: 0.90). The largest difference between marginal/unsatisfactory and satisfactory PGY2s was trustworthiness (0.78). The largest differences between marginal/unsatisfactory and satisfactory PGY3s were ethical behavior (1.17), incorporating feedback (1.03), and professionalization (0.96). For PGY2s and PGY3s, no subcompetencies had outstanding discrimination. Conclusions Marginal/unsatisfactory pediatric residents had different subcompetency gaps at different training levels. While PGY1s may have global deficits, senior residents may have different performance deficiencies requiring individualized counseling and targeted performance improvement plans.

Original languageEnglish (US)
Pages (from-to)119-129
Number of pages11
JournalAcademic Medicine
Volume93
Issue number1
DOIs
StatePublished - Jan 1 2018

Fingerprint

trainee
rating
resident
performance
statistics
deficit
discrimination
performance disorder
trustworthiness
professionalization
accreditation
counseling
recipient
graduate
evaluation
education

ASJC Scopus subject areas

  • Education

Cite this

Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group (2018). Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Academic Medicine, 93(1), 119-129. https://doi.org/10.1097/ACM.0000000000001775

Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. / Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group.

In: Academic Medicine, Vol. 93, No. 1, 01.01.2018, p. 119-129.

Research output: Contribution to journalArticle

Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group 2018, 'Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings', Academic Medicine, vol. 93, no. 1, pp. 119-129. https://doi.org/10.1097/ACM.0000000000001775
Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Academic Medicine. 2018 Jan 1;93(1):119-129. https://doi.org/10.1097/ACM.0000000000001775
Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. / Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. In: Academic Medicine. 2018 ; Vol. 93, No. 1. pp. 119-129.
@article{25144b7d610b4f6db018a6f6ea863aba,
title = "Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings",
abstract = "Purpose To perform a derivation study to determine in which subcompetencies marginal/unsatisfactory pediatric residents had the greatest deficits compared with their satisfactorily performing peers and which subcompetencies best discriminated between marginal/unsatisfactory and satisfactorily performing residents. Method Multi-institutional cohort study of all 21 milestones (rated on four or five levels) reported to the Accreditation Council for Graduate Medical Education, and global marginal/unsatisfactory versus satisfactory performance reported to the American Board of Pediatrics. Data were gathered in 2013-2014. For each level of training (postgraduate year [PGY] 1, 2, and 3), mean differences between milestone levels of residents with marginal/unsatisfactory and satisfactory performance adjusted for clustering by program and C-statistics (area under receiver operating characteristic curve) were calculated. A Bonferroni-corrected significance threshold of.0007963 was used to account for multiple comparisons. Results Milestone and overall performance evaluations for 1,704 pediatric residents in 41 programs were obtained. For PGY1s, two subcompetencies had almost a one-point difference in milestone levels between marginal/unsatisfactory and satisfactory trainees and outstanding discrimination (≥ 0.90): organize/prioritize (0.93; C-statistic: 0.91) and transfer of care (0.97; C-statistic: 0.90). The largest difference between marginal/unsatisfactory and satisfactory PGY2s was trustworthiness (0.78). The largest differences between marginal/unsatisfactory and satisfactory PGY3s were ethical behavior (1.17), incorporating feedback (1.03), and professionalization (0.96). For PGY2s and PGY3s, no subcompetencies had outstanding discrimination. Conclusions Marginal/unsatisfactory pediatric residents had different subcompetency gaps at different training levels. While PGY1s may have global deficits, senior residents may have different performance deficiencies requiring individualized counseling and targeted performance improvement plans.",
author = "{Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group} and Li, {Su Ting T.} and Tancredi, {Daniel J.} and Alan Schwartz and Ann Guillot and Ann Burke and Trimm, {R. Franklin} and Susan Guralnick and Mahan, {John D.} and Gifford, {Kimberly A.} and Dennis Basila and Katherine Dougherty and Robus, {Richard S.} and Christa Matrone and Schumacher, {Daniel J.} and Dewar, {Stephanie B.} and Dena Hofkosh and Rhett Lieberman and Poynter, {Sue E.} and Dawn Tuell and Huff, {Janara J.} and Marielisa Rincon and Sease, {Kerry K.} and Yeh, {Sylvia H.} and McKenna, {Michael P.} and Laura Price and Donnelly, {Kathleen M.} and Carter, {Meredith L.} and Rappaport, {David I.} and Wittler, {Robert R.} and Ariel Frey-Vogel and Scott-Vernaglia, {Shannon E.} and Kate Perkins and Savanna Carson and Kapklein, {Matthew J.} and Unti, {Sharon M.} and Cynthia Ferrell and Dunbar, {Alston E.} and Bhavaraju, {Vasudha L.} and Caputo, {Grace L.} and Kramer, {Jane E.} and Michelle Miner and Sharon Calaman and Mario Cruz and Spector, {Nancy D.} and Barone, {Stephen R.} and Tammy Camp and Elliott, {Sean P.} and Franke, {Hillary A.} and Heather Fagan and Nicola Orlov",
year = "2018",
month = "1",
day = "1",
doi = "10.1097/ACM.0000000000001775",
language = "English (US)",
volume = "93",
pages = "119--129",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "1",

}

TY - JOUR

T1 - Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings

AU - Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group

AU - Li, Su Ting T.

AU - Tancredi, Daniel J.

AU - Schwartz, Alan

AU - Guillot, Ann

AU - Burke, Ann

AU - Trimm, R. Franklin

AU - Guralnick, Susan

AU - Mahan, John D.

AU - Gifford, Kimberly A.

AU - Basila, Dennis

AU - Dougherty, Katherine

AU - Robus, Richard S.

AU - Matrone, Christa

AU - Schumacher, Daniel J.

AU - Dewar, Stephanie B.

AU - Hofkosh, Dena

AU - Lieberman, Rhett

AU - Poynter, Sue E.

AU - Tuell, Dawn

AU - Huff, Janara J.

AU - Rincon, Marielisa

AU - Sease, Kerry K.

AU - Yeh, Sylvia H.

AU - McKenna, Michael P.

AU - Price, Laura

AU - Donnelly, Kathleen M.

AU - Carter, Meredith L.

AU - Rappaport, David I.

AU - Wittler, Robert R.

AU - Frey-Vogel, Ariel

AU - Scott-Vernaglia, Shannon E.

AU - Perkins, Kate

AU - Carson, Savanna

AU - Kapklein, Matthew J.

AU - Unti, Sharon M.

AU - Ferrell, Cynthia

AU - Dunbar, Alston E.

AU - Bhavaraju, Vasudha L.

AU - Caputo, Grace L.

AU - Kramer, Jane E.

AU - Miner, Michelle

AU - Calaman, Sharon

AU - Cruz, Mario

AU - Spector, Nancy D.

AU - Barone, Stephen R.

AU - Camp, Tammy

AU - Elliott, Sean P.

AU - Franke, Hillary A.

AU - Fagan, Heather

AU - Orlov, Nicola

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Purpose To perform a derivation study to determine in which subcompetencies marginal/unsatisfactory pediatric residents had the greatest deficits compared with their satisfactorily performing peers and which subcompetencies best discriminated between marginal/unsatisfactory and satisfactorily performing residents. Method Multi-institutional cohort study of all 21 milestones (rated on four or five levels) reported to the Accreditation Council for Graduate Medical Education, and global marginal/unsatisfactory versus satisfactory performance reported to the American Board of Pediatrics. Data were gathered in 2013-2014. For each level of training (postgraduate year [PGY] 1, 2, and 3), mean differences between milestone levels of residents with marginal/unsatisfactory and satisfactory performance adjusted for clustering by program and C-statistics (area under receiver operating characteristic curve) were calculated. A Bonferroni-corrected significance threshold of.0007963 was used to account for multiple comparisons. Results Milestone and overall performance evaluations for 1,704 pediatric residents in 41 programs were obtained. For PGY1s, two subcompetencies had almost a one-point difference in milestone levels between marginal/unsatisfactory and satisfactory trainees and outstanding discrimination (≥ 0.90): organize/prioritize (0.93; C-statistic: 0.91) and transfer of care (0.97; C-statistic: 0.90). The largest difference between marginal/unsatisfactory and satisfactory PGY2s was trustworthiness (0.78). The largest differences between marginal/unsatisfactory and satisfactory PGY3s were ethical behavior (1.17), incorporating feedback (1.03), and professionalization (0.96). For PGY2s and PGY3s, no subcompetencies had outstanding discrimination. Conclusions Marginal/unsatisfactory pediatric residents had different subcompetency gaps at different training levels. While PGY1s may have global deficits, senior residents may have different performance deficiencies requiring individualized counseling and targeted performance improvement plans.

AB - Purpose To perform a derivation study to determine in which subcompetencies marginal/unsatisfactory pediatric residents had the greatest deficits compared with their satisfactorily performing peers and which subcompetencies best discriminated between marginal/unsatisfactory and satisfactorily performing residents. Method Multi-institutional cohort study of all 21 milestones (rated on four or five levels) reported to the Accreditation Council for Graduate Medical Education, and global marginal/unsatisfactory versus satisfactory performance reported to the American Board of Pediatrics. Data were gathered in 2013-2014. For each level of training (postgraduate year [PGY] 1, 2, and 3), mean differences between milestone levels of residents with marginal/unsatisfactory and satisfactory performance adjusted for clustering by program and C-statistics (area under receiver operating characteristic curve) were calculated. A Bonferroni-corrected significance threshold of.0007963 was used to account for multiple comparisons. Results Milestone and overall performance evaluations for 1,704 pediatric residents in 41 programs were obtained. For PGY1s, two subcompetencies had almost a one-point difference in milestone levels between marginal/unsatisfactory and satisfactory trainees and outstanding discrimination (≥ 0.90): organize/prioritize (0.93; C-statistic: 0.91) and transfer of care (0.97; C-statistic: 0.90). The largest difference between marginal/unsatisfactory and satisfactory PGY2s was trustworthiness (0.78). The largest differences between marginal/unsatisfactory and satisfactory PGY3s were ethical behavior (1.17), incorporating feedback (1.03), and professionalization (0.96). For PGY2s and PGY3s, no subcompetencies had outstanding discrimination. Conclusions Marginal/unsatisfactory pediatric residents had different subcompetency gaps at different training levels. While PGY1s may have global deficits, senior residents may have different performance deficiencies requiring individualized counseling and targeted performance improvement plans.

UR - http://www.scopus.com/inward/record.url?scp=85021084270&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021084270&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000001775

DO - 10.1097/ACM.0000000000001775

M3 - Article

VL - 93

SP - 119

EP - 129

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 1

ER -