Methodological shortcomings predicted lower harm estimates in one of two sets of studies of clinical interventions

Objectives: High quality harms data are necessary to appropriately assess the balance between benefits and harms of interventions. Little is known, however, about whether perceived methodological shortcomings are associated with lower estimates of harms. Study Design and Setting: Studies reporting harms associated with carotid endarterectomy (CEA) and rofecoxib were identified using published systematic reviews. A standardized abstraction form, including eight predefined criteria for assessing the quality of harms reporting, was used to extract data. Univariate and multivariate analyses were performed to empirically evaluate the association between quality criteria and estimates of harms. Results: In 111 studies of CEA, meeting five of the eight-quality criteria was associated with significantly higher adverse event rates. A quality-rating instrument with four criteria predicted adverse events (5.7% in studies rated "adequate," compared to 3.9% in studies rated "inadequate" [P = 0.0003]). In multivariate analyses, the quality-rating assignment remained significant when controlling for other clinical and study-related variables. Different quality criteria, however, predicted estimates of risk for myocardial infarction in 16 trials of rofecoxib. Conclusion: The presence of methodological shortcomings can predict lower estimates of serious harms. Clinicians and researchers should consider methodological shortcomings when evaluating estimates of harms associated with clinical interventions.