Analysis of Clinical Trials

Theme Co-ordinators: Neal Alexander, Baptiste Leurent, Clemence Leyrat, Amy Mulick, Stephen Nash, Linda Sharples

Please see here for slides and audio recordings of previous seminars relating to this theme.

Randomised controlled trials (RCTs) are one of the most important tools to estimate effects of medical interventions. However, there are a number of issues relating to the statistical analysis of RCTs over which there is much debate. We aim to consider and raise discussion of a number of these issues, and suggest possible statistical approaches for dealing with these in the analyses of RCTs. We briefly introduce here some of these issues.

Topics:

Cluster randomised trials
Covariate adjustment
Subgroup analysis
Missing data
Non-compliance
Sequential trials
Good practice in trials

Cluster randomised trials

In cluster randomised trials (CRTs), groups of participants, rather than the participants themselves, are randomised to intervention groups¹. This design is increasingly used to assess complex interventions, in particular community-level interventions. Many CRTs have been conducted at LSHTM to evaluate the effectiveness of health-care interventions, motivating a wide range of methodological research on their design^2,3, analysis^4,5 and reporting⁶ to take into account the specificities of such trials. One such challenge in CRTs arises because participants’ outcome measures are not independent and thus clustering must be accounted for in the analysis. For this reason, CRTs are also a main research area within the Design and Analysis for Dependent Data CSM theme.

Other issues arising from CRTs have been recently studied at the CSM, such as the risk of systematic baseline imbalance⁷, analysis in the presence of missing data⁸, and analysis of alternative cluster designs such as cluster cross-over trials⁹. Ongoing projects include research into spatial analysis of CRTs, the analysis strategy when only a small number of clusters are randomised, and the estimate of causal effects in CRTs with non-compliance.

Hayes RJ, Moulton LH. Cluster Randomised Trials. Taylor & Francis; 2009. 338 p.
Hayes RJ, Alexander ND, Bennett S, Cousens SN. Design and analysis issues in cluster-randomized trials of interventions against infectious diseases. Stat Methods Med Res. 2000 Apr;9(2):95–116.
Thomson A, Hayes R, Cousens S. Measures of between-cluster variability in cluster randomized trials with binary outcomes. Stat Med. 2009 May 30;28(12):1739–51.
Gomes M, Díaz-Ordaz K, Grieve R, Kenward MG. Multiple imputation methods for handling missing data in cost-effectiveness analyses that use data from hierarchical studies: an application to cluster randomized trials. Med Decis Mak Int J Soc Med Decis Mak. 2013 Nov;33(8):1051–63.
Alexander N, Emerson P. Analysis of incidence rates in cluster-randomized trials of interventions against recurrent infections, with an application to trachoma. Stat Med. 2005 Sep 15;24(17):2637–47.
Campbell MK, Piaggio G, Elbourne DR, Altman DG, for the CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012 Sep 4;345(sep04 1):e5661–e5661.
Leyrat C, Caille A, Foucher Y, Giraudeau B. Propensity score to detect baseline imbalance in cluster randomized trials: the role of the c-statistic. BMC Med Res Methodol. 2015;
DiazOrdaz K, Kenward MG, Gomes M, Grieve R. Multiple imputation methods for bivariate outcomes in cluster randomised trials. Stat Med. 2016 Sep 10;35(20):3482–96.
Morgan KE, Forbes AB, Keogh RH, Jairath V, Kahan BC. Choosing appropriate analysis methods for cluster randomised cross-over trials with a binary outcome. Stat Med. 2016 Sep 28;

Covariate adjustment

In RCTs, unadjusted analysis provides an unbiased estimate of the treatment effect. However, even though randomisation should insure baseline characteristics (covariates) are broadly balanced between groups, chance imbalances can occur, especially in smaller trials. Covariate adjustment for important predictors of outcome can be used to allow for such imbalances¹. In certain circumstances, adjustment using appropriate covariates can also be used to improve the power (or to reduce the required sample size) of an RCT irrespective of any baseline imbalance².

Within the CSM, work has been conducted to look at the impact of covariate adjustment on power in real settings³, and also to investigate alternative strategies to multivariable regression for covariate adjustment, in particular the use of propensity score weighting⁴. Furthermore, methodological research has been conducted to extend covariate adjustment methods to more challenging randomised trials such as cross-over⁵ or cluster randomised trials⁶, in which both chance and systematic imbalance can occur. Recent work also includes the development of recommendations for the implementation of covariate adjustment in practice⁷.

Altman DG. Adjustment for Covariate Imbalance. In: Biostatistics in Clinical Trials [Internet]. Chichester, UK: John Wiley & Sons, Ltd; 2001 [cited 2016 Oct 14]. p. 122–7.
Hernández AV, Steyerberg EW, Habbema JDF. Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. J Clin Epidemiol. 2004 May;57(5):454–60.
Turner EL, Perel P, Clayton T, Edwards P, Hernández AV, Roberts I, et al. Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury. J Clin Epidemiol. 2012 May;65(5):474–81.
Williamson EJ, Forbes A, White IR. Variance reduction in randomised trials by inverse probability weighting using the propensity score. Stat Med. 2014 Feb 28;33(5):721–37.
Kenward MG, Roger JH. The use of baseline covariates in crossover studies. Biostatistics. 2010 Jan 1;11(1):1–17.
Gomes M, Grieve R, Nixon R, Ng ES-W, Carpenter J, Thompson SG. Methods for Covariate Adjustment in Cost-Effectiveness Analysis That Use Cluster Randomised Trials. Health Econ. 2012;21(9):1101–1118.
Pocock SJ, McMurray JJV, Collier TJ. Statistical Controversies in Reporting of Clinical Trials: Part 2 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol. 2015 Dec 15;66(23):2648–62.

Subgroup analysis

The analysis of RCTs by subgroups of individuals (e.g. according to age, gender or medical history) remains controversial and often misunderstood^1–3. While a well conducted subgroup analysis can be justified, deviation from recommended practices can easily result in dubious findings³. It is recognised^2,4 that such analyses should be limited to a few key baseline factors which are specified prior to any analyses being undertaken. In addition, an appropriate analysis should report effect estimates and confidence intervals within such subgroups, together with an overall interaction test, rather than separate p-values within each category of the subgroup. Irrespective of the findings, it is recommended to interpret them with caution until replicated. Sun et al. derived a useful list of criteria to assess the credibility of subgroup analysis findings⁵.

Alternatives methods which have been explored at the CSM include analysis based on risk score, potentially improving the power to explore subgroup effect, and allowing examination of the absolute net benefits in view of the patients’ baseline risk ^4,6,7. Bayesian approach to subgroup analysis is another area of interest, allowing to take into account the clinical plausibility of the subgroup effect into the analysis⁸.

Example of subgroup analysis and forest plot, based on the SIPIRIT IV clinical trial⁹.

Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000;355(9209):1064-1069.
Wang R, Lagakos SW, et al. Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials. N Engl J Med. 2007;357:2189-2194.
Wallach JD, Sullivan PG, et al. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Internal Medicine. 2017 Apr 1;177(4):554-60.
Pocock SJ, McMurray JJ V, Collier TJ. Statistical Controversies in Reporting of Clinical Trials Part 2 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol. 2015;66(23):2648-2662.
Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ. 2010;340(9209):c117.
Pocock SJ, Lubsen J. More on Subgroup Analyses in Clinical Trials. 2008;19(8):2076-2077.
Fox KAA, Poole-Wilson P, Clayton TC, et al. 5-year outcome of an interventional strategy in non-ST-elevation acute coronary syndrome: The British Heart Foundation RITA 3 randomised trial. Lancet. 2005;366(9489):914-920.
White IR, Pocock SJ, Wang D. Eliciting and using expert opinions about influence of patient characteristics on treatment effects: A Bayesian analysis of the CHARM trials. Stat Med. 2005;24(24):3805-3821.
Stone GW, Rizvi A, et al. Everolimus-eluting versus paclitaxel-eluting stents in coronary artery disease. N Engl J Med. 2010 May 6;362(18):1663-74.

Missing data

Missing data are a common issue in clinical trials¹ and can results in underpowered studies or biased findings. The issue of missing data is an active area of research in the CSM, with a dedicated theme, and website www.missingdata.org.uk.

Several guidelines have been published on missing data in the context of clinical trials^2–4 . The report from the National Research Council concludes that methods should be chosen according to the plausibility of their underlying assumptions, and discourage simple fixes such as last observations carried forward. Because assumptions cannot be verified with the data at hand, conducting sensitivity analyses under alternative assumptions is also recommended^2–4, however a review by Bell et al. found that this was rarely done in practice¹.

Topics of particular interest in the CSM includes the use of multiple imputation⁵, for example in cluster-randomised trials^6–8, and developing practical approaches for sensitivity analysis when data may be missing not at random^4,9.

Bell ML, Fiero M, Horton NJ, et al. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol. 2014;14(1):118.
Burzykowski T, Carpenter J, Coens C, et al. Missing data: Discussion points from the PSI missing data expert group. Pharm Stat. 2010;9(4):288-297.
Little RJ, D’Agostino R, Cohen ML, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012;367(14):1355-1360.
Carpenter JR, Kenward MG. Missing Data in Randomised Controlled Trials — a Practical Guide.; 2007. http://missingdata.lshtm.ac.uk/downloads/rm04_jh17_mk.pdf.
Carpenter JR, Kenward MG. Multiple Imputation and Its Application. John Wiley & Sons; 2013.
DiazOrdaz K, Kenward MG, Gomes M, Grieve R. Multiple imputation methods for bivariate outcomes in cluster randomised trials. Stat Med. 2016;(February). doi:10.1002/sim.6935.
Gomes M, Díaz-Ordaz K, Grieve R, Kenward M. Multiple imputation methods for handling missing data in cost-effectiveness analyses that use data from hierarchical studies: an application to cluster randomized trials. Med Decis Mak. 2013;33(8):1051-1063.
Caille A, Leyrat C, Giraudeau B. A comparison of imputation strategies in cluster randomized trials with missing binary outcomes. Stat Methods Med Res. April 2014
Carpenter JR, Roger JH, Kenward MG. Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible assumptions, and inference via multiple imputation. J Biopharm Stat. 2013;23(3):1352-1371.

Non compliance

In RCTs, intention-to-treat (ITT) analysis, in which patients are analysed with respect to the intervention they have been allocated to regardless of what they actually received, is considered the gold-standard¹. ITT analysis estimates the effectiveness, or, following Carpenter et al. terminology, the de facto estimand, ie. “what would be the effect seen in practice?”². It is however also of interest to estimate the effect of the intervention if the patients were to comply with it (the efficacy or de jure estimand). Although per protocol analysis (including only patients who complied with their allocated treatment) have been proposed for this, it is now well recognised that it does not maintain the randomisation and is therefore at risk of bias. Methods to estimate a Complier Average Causal Effect (CACE) have been proposed to estimate the causal effect of the intervention amongst participants who actually received it, such as instrumental variables³ or propensity scores⁴, but several questions require further investigation, including how to estimate the CACE when there is more than one active treatment, or when the required assumptions of the causal inference framework do not hold (see the CSM Causal Inference theme).

Ongoing research at the School looks at the use of multiple imputation of the compliance strata to estimate the CACE and also on how to tackle non-compliance in specific study designs such as in cluster randomised trials, in which the compliance decision can occur both at the individual and at the cluster level, or in non-inferiority trials, in which an underestimation of the treatment effect with ITT analysis can have important consequences in practice.

Newell DJ. Intention-to-treat analysis: implications for quantitative and qualitative research. Int J Epidemiol. 1992 Oct;21(5):837–41.
Carpenter JR, Roger JH, Kenward MG. Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible assumptions, and inference via multiple imputation. J Biopharm Stat. 2013;23(6):1352–71.
Angrist JD, Imbens GW, Rubin DB. Identification of Causal Effects Using Instrumental Variables. J Am Stat Assoc. 1996;91(434):444–55.
Porcher R, Leyrat C, Baron G, Giraudeau B, Boutron I. Performance of principal scores to estimate the marginal compliers causal effect of an intervention. Stat Med. 2015 Sep 17;

Sequential trials

Sequential designs were originally based on industrial quality control, to monitor whether a process resulted in an abnormally high proportion of defects¹. They focus on decision making rather than estimation. The simplest sequential trial designs involve assessing each person as their outcome becomes available, although they can be modified to analyse at regular intervals^2-4. These modified designs can be used in a similar way as those based on the alpha-spending group sequential approach⁵, although the two approaches are based on different philosophies.

One of the simpler sequential designs is the triangular test for a single proportion. This is illustrated in the following figure which is from a non-comparative trial of three treatments for visceral leishmaniasis in East Africa⁶.

The outcome of interest is the number of patients who recovered, which is captured on the vertical axis. The horizontal axis (V) is proportional to the number of people recruited to the trial so far. The trial continues while the findings fall in the triangular region, which is calculated based on the acceptable cure rate and error probabilities. Findings above the upper boundary (as in the case here for the three treatments at the third time-point), indicate a favourable conclusion, while region below the triangle indicate an unfavourable conclusion.

Aspects of sequential designs being researched by members of the CSM and their collaborators include analysis methods for time points subsequent to assessment of the primary endpoint, and development of asymmetrical stopping boundaries e.g. the imposition of a minimum sample size^7,8.

Wald, A. Sequential Analysis. (Wiley, 1947).
Whitehead, J. The Design and Analysis of Sequential Clinical Trials. 1st edn, (Ellis Horwood, 1983).
Bellissant, E., Benichou, J. & Chastang, C. Application of the triangular test to phase II cancer clinical trials. Stat Med 9, 907-917 (1990).
Ranque, S., Badiaga, S., Delmont, J. & Brouqui, P. Triangular test applied to the clinical trial of azithromycin against relapses in Plasmodium vivax infections. Malar J 1, 13 (2002).
Lan, K. K. G. & DeMets, D. L. Discrete sequential boundaries for clinical trials. Biometrika 70, 659-663 (1983).
Wasunna, M. et al. Efficacy and Safety of AmBisome in Combination with Sodium Stibogluconate or Miltefosine and Miltefosine Monotherapy for African Visceral Leishmaniasis: Phase II Randomized Trial. PLoS Negl Trop Dis 10, e0004880.
Omollo, R. et al. Safety and efficacy of miltefosine alone and in combination with sodium stibogluconate and liposomal amphotericin B for the treatment of primary visceral leishmaniasis in East Africa: study protocol for a randomized controlled trial. Trials 12, 166 (2011).
Allison, A. et al. Generalizing boundaries for triangular designs, and efficacy estimation at extended follow-ups. Trials 16, 522, doi:10.1186/s13063-015-1018-1 (2015).

Good Practice in Trials

Clinical trials need to comply with different regulatory and legal requirements depending on the jurisdiction(s) to which they are subject. In particular, they usually need to comply with Good Clinical Practice (GCP) as defined by the International Conference on Harmonization (www.ich.org).

In addition, some reporting guidelines have become de facto standards due to their adoption by medical journals. For clinical trials the most important is CONSORT¹, or CONsolidated Standards of Reporting Trials (www.consort-statement.org/consort-2010), whose use is encouraged by the International Committee of Medical Journal Editors.

As well as contributing to the development of CONSORT and other reporting guidelines, LSHTM researchers have written prominent journal articles on the principles and practice of clinical trials. These include guidance on trial design^2,3, writing and interpreting clinical trial reports⁴, interpreting results from trials which did, or did not, find an effect of the intervention on the primary endpoint^5,6, and current statistical controversies in reporting clinical trials⁷.

Schulz, K. F., Altman, D. G. & Moher, D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med 8, 18, doi:1741-7015-8-18 (2010).
Pocock, S. J., Clayton, T. C. & Stone, G. W. Challenging Issues in Clinical Trial Design: Part 4 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol 66, 2886-2898 (2015).
Pocock, S. J., Clayton, T. C. & Stone, G. W. Design of Major Randomized Trials: Part 3 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol 66, 2757-2766 (2015).
Pocock, S. J., McMurray, J. J. & Collier, T. J. Making Sense of Statistics in Clinical Trial Reports: Part 1 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol 66, 2536-2549 (2015).
Pocock, S. J. & Stone, G. W. The Primary Outcome Is Positive – Is That Good Enough? N Engl J Med 375, 971-979 (2016).
Pocock, S. J. & Stone, G. W. The Primary Outcome Fails – What Next? N Engl J Med 375, 861-870, doi:10.1056/NEJMra1510064 (2016).
Pocock, S. J., McMurray, J. J. & Collier, T. J. Statistical Controversies in Reporting of Clinical Trials: Part 2 of a 4-Part Series on Statistics for Clinical Trials. J Am Coll Cardiol 66, 2648-2662 (2015).

Centre for Statistical Methodology