Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals

Myung Sun Kim, Vinay Prasad

Research output: Contribution to journalArticle

Abstract

Importance: Response rates are a well-recognized outcome of clinical trials and provide an objective measure of drug activity. Objectives: To quantify the difference between objective response rate and visual representation of response in waterfall plots in recent articles in major medical journals and to assess the change in frequency over time with which waterfall plots are used. Design, Setting, and Participants: In a cross-sectional study, original articles of 6 top journals between July 2016 and June 2018 were manually reviewed to identify articles including a waterfall plot to describe a treatment effect of cancer therapy. Response rates visually represented in waterfall plots were compared with response rates reported as study outcomes. The number of original articles with a waterfall plot as a percentage of total original articles was evaluated, sampling articles from January, February, and March for the years 2004, 2008, 2012, 2016, and 2018. Main Outcomes and Measures: Difference between response rates depicted in waterfall plots and response rates reported as study outcomes. Results: One hundred twenty-six articles were selected for analysis. Of the 97 articles reporting investigator-assessed response rates, waterfall plots showed response rates a median (interquartile range) of 6.1% (1.8%-12.0%) higher than rates derived from investigator assessment. Forty-two articles reported response rates based on central assessment as an outcome, and waterfall plots showed response rates a median (interquartile range) of 12.0% (7.7%-18.5%) higher compared with centrally assessed response rates. The estimated percentage of original articles using waterfall plots increased from 0% in 2004 to 7% in 2018. Conclusions and Relevance: This study suggests that waterfall plots are becoming more common in oncology literature. Waterfall plots provide a visual overestimate of response rate of cancer therapies and should be evaluated with caution.

Original languageEnglish (US)
Pages (from-to)e193981
JournalJAMA network open
Volume2
Issue number5
DOIs
StatePublished - May 3 2019
Externally publishedYes

Fingerprint

Outcome Assessment (Health Care)
Neoplasms
Research Personnel
Cross-Sectional Studies
Clinical Trials
Therapeutics
Pharmaceutical Preparations

Cite this

Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals. / Kim, Myung Sun; Prasad, Vinay.

In: JAMA network open, Vol. 2, No. 5, 03.05.2019, p. e193981.

Research output: Contribution to journalArticle

@article{93e558aa124443ea89c14f5b23ceca37,
title = "Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals",
abstract = "Importance: Response rates are a well-recognized outcome of clinical trials and provide an objective measure of drug activity. Objectives: To quantify the difference between objective response rate and visual representation of response in waterfall plots in recent articles in major medical journals and to assess the change in frequency over time with which waterfall plots are used. Design, Setting, and Participants: In a cross-sectional study, original articles of 6 top journals between July 2016 and June 2018 were manually reviewed to identify articles including a waterfall plot to describe a treatment effect of cancer therapy. Response rates visually represented in waterfall plots were compared with response rates reported as study outcomes. The number of original articles with a waterfall plot as a percentage of total original articles was evaluated, sampling articles from January, February, and March for the years 2004, 2008, 2012, 2016, and 2018. Main Outcomes and Measures: Difference between response rates depicted in waterfall plots and response rates reported as study outcomes. Results: One hundred twenty-six articles were selected for analysis. Of the 97 articles reporting investigator-assessed response rates, waterfall plots showed response rates a median (interquartile range) of 6.1{\%} (1.8{\%}-12.0{\%}) higher than rates derived from investigator assessment. Forty-two articles reported response rates based on central assessment as an outcome, and waterfall plots showed response rates a median (interquartile range) of 12.0{\%} (7.7{\%}-18.5{\%}) higher compared with centrally assessed response rates. The estimated percentage of original articles using waterfall plots increased from 0{\%} in 2004 to 7{\%} in 2018. Conclusions and Relevance: This study suggests that waterfall plots are becoming more common in oncology literature. Waterfall plots provide a visual overestimate of response rate of cancer therapies and should be evaluated with caution.",
author = "Kim, {Myung Sun} and Vinay Prasad",
year = "2019",
month = "5",
day = "3",
doi = "10.1001/jamanetworkopen.2019.3981",
language = "English (US)",
volume = "2",
pages = "e193981",
journal = "JAMA network open",
issn = "2574-3805",
publisher = "American Medical Association",
number = "5",

}

TY - JOUR

T1 - Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals

AU - Kim, Myung Sun

AU - Prasad, Vinay

PY - 2019/5/3

Y1 - 2019/5/3

N2 - Importance: Response rates are a well-recognized outcome of clinical trials and provide an objective measure of drug activity. Objectives: To quantify the difference between objective response rate and visual representation of response in waterfall plots in recent articles in major medical journals and to assess the change in frequency over time with which waterfall plots are used. Design, Setting, and Participants: In a cross-sectional study, original articles of 6 top journals between July 2016 and June 2018 were manually reviewed to identify articles including a waterfall plot to describe a treatment effect of cancer therapy. Response rates visually represented in waterfall plots were compared with response rates reported as study outcomes. The number of original articles with a waterfall plot as a percentage of total original articles was evaluated, sampling articles from January, February, and March for the years 2004, 2008, 2012, 2016, and 2018. Main Outcomes and Measures: Difference between response rates depicted in waterfall plots and response rates reported as study outcomes. Results: One hundred twenty-six articles were selected for analysis. Of the 97 articles reporting investigator-assessed response rates, waterfall plots showed response rates a median (interquartile range) of 6.1% (1.8%-12.0%) higher than rates derived from investigator assessment. Forty-two articles reported response rates based on central assessment as an outcome, and waterfall plots showed response rates a median (interquartile range) of 12.0% (7.7%-18.5%) higher compared with centrally assessed response rates. The estimated percentage of original articles using waterfall plots increased from 0% in 2004 to 7% in 2018. Conclusions and Relevance: This study suggests that waterfall plots are becoming more common in oncology literature. Waterfall plots provide a visual overestimate of response rate of cancer therapies and should be evaluated with caution.

AB - Importance: Response rates are a well-recognized outcome of clinical trials and provide an objective measure of drug activity. Objectives: To quantify the difference between objective response rate and visual representation of response in waterfall plots in recent articles in major medical journals and to assess the change in frequency over time with which waterfall plots are used. Design, Setting, and Participants: In a cross-sectional study, original articles of 6 top journals between July 2016 and June 2018 were manually reviewed to identify articles including a waterfall plot to describe a treatment effect of cancer therapy. Response rates visually represented in waterfall plots were compared with response rates reported as study outcomes. The number of original articles with a waterfall plot as a percentage of total original articles was evaluated, sampling articles from January, February, and March for the years 2004, 2008, 2012, 2016, and 2018. Main Outcomes and Measures: Difference between response rates depicted in waterfall plots and response rates reported as study outcomes. Results: One hundred twenty-six articles were selected for analysis. Of the 97 articles reporting investigator-assessed response rates, waterfall plots showed response rates a median (interquartile range) of 6.1% (1.8%-12.0%) higher than rates derived from investigator assessment. Forty-two articles reported response rates based on central assessment as an outcome, and waterfall plots showed response rates a median (interquartile range) of 12.0% (7.7%-18.5%) higher compared with centrally assessed response rates. The estimated percentage of original articles using waterfall plots increased from 0% in 2004 to 7% in 2018. Conclusions and Relevance: This study suggests that waterfall plots are becoming more common in oncology literature. Waterfall plots provide a visual overestimate of response rate of cancer therapies and should be evaluated with caution.

UR - http://www.scopus.com/inward/record.url?scp=85066857547&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066857547&partnerID=8YFLogxK

U2 - 10.1001/jamanetworkopen.2019.3981

DO - 10.1001/jamanetworkopen.2019.3981

M3 - Article

VL - 2

SP - e193981

JO - JAMA network open

JF - JAMA network open

SN - 2574-3805

IS - 5

ER -