Training patients to review scientific reports for the Patient-Centered Outcomes Research Institute: An observational study

Objective The peer review of completed Patient-Centered Outcomes Research Institute (PCORI) funded research includes reviews from patient reviewers (patients, caregivers, and patient advocates). Very little is known about how best to support these reviewers in writing helpful comments from a patient-centred perspective. This study aimed to evaluate the effect of a new training in peer review for patient reviewers. Design Observational study. Setting Online. Participants Adults registered in the PCORI Reviewer Database as a patient stakeholder. Intervention A new online training in peer review. Main outcome measures Changes in reviewers' knowledge and skills; change in self-efficacy and attitudes, satisfaction with the training and perceived benefits and relevance of the training. Results Before-after training survey data were analysed for 37 (29.4% of 126) patient reviewers invited to participate in an online training as part of a quality improvement effort or as part of a PCORI peer review. The reviewers improved their answers to the knowledge questions (p<0.001, median number of answers improved 4 (95% CI 3 to 5), large effect size (ES) Cohen's w=0.94) after the training, particularly in the questions targeting the specifics of PCORI peer review. Reviewers improved their skills in recognising helpful review comments, but those without peer-review background improved proportionally more (p=0.008, median number of answers improved 2 (95% CI 1 to 3), medium ES w=0.60). The use of training modestly increased reviewers' confidence in completing a high-quality peer review (p=0.005, mean increase in 5-point Likert rating 0.51 (95% CI 0.17 to 0.86), small-to-medium ES Cliff's delta=0.32) and their excitement about providing a review slightly increased (p=0.019, mean increase in 5-point Likert rating 0.35 (95% CI 0.03 to 0.68), small ES delta=0.19). All reviewers were satisfied with the training and would recommend it to other reviewers. Conclusions Training improved knowledge, skills and self-efficacy and slightly increased enthusiasm for completing a PCORI peer review.