ASJC Scopus subject areas
Access to Document
Other files and links
Multiplicity in oncology randomised controlled trials : a threat to medical evidence? / Prasad, Vinay; Booth, Christopher M.In: The Lancet Oncology, Vol. 20, No. 12, 12.2019, p. 1638-1640.
Research output: Contribution to journal › Comment/debate › peer-review
TY - JOUR
T1 - Multiplicity in oncology randomised controlled trials
T2 - a threat to medical evidence?
AU - Prasad, Vinay
AU - Booth, Christopher M.
N1 - Funding Information: Vinay Prasad a email@example.com Christopher M Booth b c a Department of Medicine and Center for Health Care Ethics, Oregon Health & Science University, Portland, OR 97202, USA Department of Medicine and Center for Health Care Ethics Oregon Health & Science University Portland OR 97202 USA Department of Medicine and Center for Health Care Ethics, Oregon Health & Science University, Portland, OR 97202, USA b Department of Oncology, Queen's University, Kingston, ON, Canada Department of Oncology Queen's University Kingston ON Canada Department of Oncology, Queen's University, Kingston, ON, Canada c Division of Cancer Care and Epidemiology, Queen's Cancer Research Institute, Kingston, ON, Canada Division of Cancer Care and Epidemiology Queen's Cancer Research Institute Kingston ON Canada Division of Cancer Care and Epidemiology, Queen's Cancer Research Institute, Kingston, ON, Canada Randomised controlled trials offer several advantages over observational studies in establishing whether cancer treatments improve patient outcomes. First, randomised controlled trials minimise bias and confounding. Observational studies judge treatments that were chosen not at random but deliberately by physicians, and the decision to treat or not treat an individual might be affected by patient characteristics, a type of bias called confounding by indication. Randomised controlled trials prescribe therapy and avoid this limitation. Second, randomised controlled trials restrict multiple hypothesis testing. Randomised controlled trials typically address a specific research question with a prespecified primary outcome and a predefined statistical analysis plan. This approach reduces the ability of investigators to assess multiple outcomes, which in turn minimises the risk of a spurious (eg, false positive) finding. For these reasons, David Sackett placed randomised controlled trials at the apex of the hierarchy of evidence. 1 Yet in the past 10 years, one of these assumptions is increasingly being questioned. Randomised controlled trials have proliferated in number, often test similar compounds with similar molecular targets, and are often run in redundant and duplicative trial portfolios or agendas. For this reason, multiplicity has emerged as a new threat to the validity of conclusions drawn from randomised controlled trials. In this Comment, we explore this issue. Multiplicity, or the number of times an analysis can be run, has long been recognised as a threat to causal inference from observational studies. Patel and colleagues 2 found that if researchers are afforded many opportunities to analyse a dataset and allowed to choose the variables they can adjust for, often both positive and negative associations can be reached for a single exposure–outcome relationship. The concern is that many investigators might explore a similar question in parallel, and only the researchers who discover significant findings report results, leading to a literature that finds both positive and negative associations and one that rarely reports null findings. When researchers randomly picked ingredients from a cookbook and searched for observational studies linking those nutrients to cancer risk, they found this was the case. 3 In the past 10 years, there has been substantial growth in the number of randomised controlled trials that evaluate individual anticancer drugs, including those of the same class and mechanism of action. Consider the immune checkpoint inhibitor class of medications. Drugs targeting PD-1 or PD-L1 are being tested in more than 2000 clinical trials with an estimated enrollment of more than 500 000 participants. 4 Multiple randomised controlled trials are being done to evaluate the combination of many types of chemotherapy plus a checkpoint inhibitor for many types of tumours, different checkpoint inhibitors from different companies, or combinations of targeted and immunotherapy drugs; the probability that some results will emerge as favourable by chance alone is high. Multiplicity in randomised controlled trials might explain the results of clinical trials of bevacizumab. Bevacizumab, a monoclonal antibody against VEGF, has been tested in at least 50 randomised controlled trials for people with solid tumours. 5 The results of these studies have varied even within a single malignancy. In non-small-cell lung cancer, one randomised controlled trial reported improved outcomes when bevacizumab was added to carboplatin and paclitaxel, yet a similar trial in the same population with cisplatin and gemcitabine as the chemotherapy backbone did not. 6 The same is true in colorectal cancer; a randomised controlled trial evaluating the addition of bevacizumab to chemotherapy with irinotecan, fluorouracil, and leucovorin in front-line colorectal cancer, by Hurwitz and colleagues, 7 found an overall survival benefit, but a similar randomised controlled trial evaluating the addition of bevacizumab to the FOLFOX (fluorouracil, oxaliplatin, leucovorin) regimen, led by Saltz and colleagues, 8 did not find any benefit. When trials of bevacizumab have been reported as positive, they have generally used a primary outcome of progression-free survival, whereas few have shown improvement in overall survival. Even in the trials that are deemed positive, the effect size of bevacizumab has been modest. These findings raise the question of whether bevacizumab truly improves survival. The drug might increase radiographical tumour response, and might lengthen radiographical progression, but are the few instances of survival benefit merely the artifact of multiple testing? In a recent analysis, 9,10 we did a multiplicity adjustment to all of the published randomised controlled trials that have evaluated chemotherapy with or without bevacizumab, regardless of the type of malignancy being treated. Such adjustments are often done if multiple outcomes are analysed within a clinical study, but here we applied that same principle to multiple studies examining the same drug. When we applied a Bonferroni correction, a stringent adjustment for multiplicity, most of the trials that reported a statistically significant improvement in overall survival lost that significance and only one remained positive. For 30 trials reporting substantial improvements in progression-free survival, applying the multiplicity correction changed that conclusion in nine of them. Others might prefer to apply different false-discovery adjustments that are less stringent than the Bonferroni correction, which assumes that all data are independent. However, all false-discovery methods would lead to conclusions that are less optimistic than those reported for each trial individually. Philosophically, there is little difference between a single mega-randomised controlled trial comparing bevacizumab in combination with a chemotherapy backbone versus the same chemotherapy backbone in all solid tumours and subsequently doing a subgroup analysis by tumour type versus doing individual studies in all of these cohorts. However, based on the current framework for interpretation of oncology trials, we would always adjust for multiplicity in the former analysis or consider results to be hypothesis generating, or both, yet in the latter analysis, we often take the results as gospel. The risk of clinical practice being influenced by multiplicity is even greater for trials evaluating immune checkpoint inhibitors. Drugs targeting PD-1 and PD-L1 are generating high clinical interest. There are more than 20 such compounds in preclinical testing and they are being tested in various combinations. Many clinical trials are duplicative, running in clinical settings where PD-1 antibodies are thought to exert meaningful activity—for instance, non-small-cell lung cancer and melanoma. With so many trials testing similar hypotheses with similar drugs, the likelihood that any one trial will yield a significant result is increased by the large number of times that something has been tested. Here, we need to correct for the portfolio of trials not within a single pharmaceutical company but across all companies. Of course, robust results (eg, multiple positive trials in non-small-cell lung cancer and melanoma) represent real gains, but if only some immunotherapy trials are positive and others are not, particularly in the same clinical setting, then caution is warranted. Historically, oncologists did not have to worry about multiple hypothesis testing in clinical trials for the simple reason that there were small numbers of similar drugs developed simultaneously, and restricted research budgets with which to invest, thus the risk of multiple hypothesis testing was minimal. Accordingly, it was quite unlikely to have concurrent randomised controlled trials addressing the same clinical question. However, the ever-escalating cost of cancer drugs and their high-profit margins has incentivised the pharmaceutical industry to pursue the evaluation of similar drugs in duplicative, large portfolios of clinical studies. This approach has been further facilitated by the reluctance of the US Food and Drug Administration to compel manufacturers to test their drugs against existing compounds or to restrict the number of accelerated approvals. In the 20th century, randomised controlled trials leapt to the top of the hierarchy of evidence-based medicine because of their ability to minimise confounding and isolate the therapeutic effect of interventions. Studies were assessed individually on the basis of their conduct, transparency, and reporting. In the 21st century, large portfolios of clinical trials testing similar drugs for similar questions have raised the bar. For randomised controlled trials to remain atop the hierarchy, not only should each individual study be free from bias, but each individual result should be judged alongside the entire portfolio of trials. The rare so-called positive trial within a sea of negative studies is more likely a false positive than a true positive. The oncology community will need to be mindful of the risk of multiplicity leading to false conclusions from randomised controlled trials, especially for checkpoint inhibitors. To deliver high-quality cancer care in a sustainable health ecosystem, clinicians, investigators, and policy makers will need to identify therapies that offer benefits that are substantial in magnitude and not statistical artifacts. VP is supported by the Arnold Foundation. CMB is supported as the Canada Research Chair in Population Cancer Care. We gratefully acknowledge Elizabeth Eisenhauer and Ian Tannock for comments on an earlier draft of this manuscript. VP reports receiving royalties from his book Ending Medical Reversal and an advance for the forthcoming book Malignant: How Bad Policy and Bad Evidence Work Against People With Cancer . VP's work is funded by the Laura and John Arnold Foundation and VP has received honoraria for Grand Rounds/lectures from several universities, medical centres, and professional societies and payments for contributions to Medscape. VP also hosts the podcast Plenary Session, which has Patreon backers. CMB declares no competing interests.
PY - 2019/12
Y1 - 2019/12
UR - http://www.scopus.com/inward/record.url?scp=85075648044&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85075648044&partnerID=8YFLogxK
U2 - 10.1016/S1470-2045(19)30744-2
DO - 10.1016/S1470-2045(19)30744-2
M3 - Comment/debate
C2 - 31797779
AN - SCOPUS:85075648044
VL - 20
SP - 1638
EP - 1640
JO - The Lancet Oncology
JF - The Lancet Oncology
SN - 1470-2045
IS - 12