An extended validity argument for assessing feedback culture

Steven Rougas, Brian Clyne, Anna T. Cianciolo, Teresa M. Chan, Jonathan Sherbino, Lalena Yarris

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.

Original languageEnglish (US)
Pages (from-to)355-358
Number of pages4
JournalTeaching and Learning in Medicine
Volume27
Issue number4
DOIs
StatePublished - 2015

Fingerprint

Organizational Culture
organizational culture
Hospital Emergency Service
Emergency Medicine
Internship and Residency
resident
Formative Feedback
Delphi Technique
Foster Home Care
Information Storage and Retrieval
medicine
Focus Groups
Sample Size
market
survey research
Organizations
director
flexibility
Health
Growth

Keywords

  • Feedback
  • Organizational culture
  • Validity

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

An extended validity argument for assessing feedback culture. / Rougas, Steven; Clyne, Brian; Cianciolo, Anna T.; Chan, Teresa M.; Sherbino, Jonathan; Yarris, Lalena.

In: Teaching and Learning in Medicine, Vol. 27, No. 4, 2015, p. 355-358.

Research output: Contribution to journalArticle

Rougas, Steven ; Clyne, Brian ; Cianciolo, Anna T. ; Chan, Teresa M. ; Sherbino, Jonathan ; Yarris, Lalena. / An extended validity argument for assessing feedback culture. In: Teaching and Learning in Medicine. 2015 ; Vol. 27, No. 4. pp. 355-358.
@article{1b1bacba48d947539a88677d6e4914e0,
title = "An extended validity argument for assessing feedback culture",
abstract = "Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55{\%}) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.",
keywords = "Feedback, Organizational culture, Validity",
author = "Steven Rougas and Brian Clyne and Cianciolo, {Anna T.} and Chan, {Teresa M.} and Jonathan Sherbino and Lalena Yarris",
year = "2015",
doi = "10.1080/10401334.2015.1077133",
language = "English (US)",
volume = "27",
pages = "355--358",
journal = "Teaching and Learning in Medicine",
issn = "1040-1334",
publisher = "Routledge",
number = "4",

}

TY - JOUR

T1 - An extended validity argument for assessing feedback culture

AU - Rougas, Steven

AU - Clyne, Brian

AU - Cianciolo, Anna T.

AU - Chan, Teresa M.

AU - Sherbino, Jonathan

AU - Yarris, Lalena

PY - 2015

Y1 - 2015

N2 - Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.

AB - Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.

KW - Feedback

KW - Organizational culture

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=84954516930&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84954516930&partnerID=8YFLogxK

U2 - 10.1080/10401334.2015.1077133

DO - 10.1080/10401334.2015.1077133

M3 - Article

VL - 27

SP - 355

EP - 358

JO - Teaching and Learning in Medicine

JF - Teaching and Learning in Medicine

SN - 1040-1334

IS - 4

ER -