Apparent declining efficacy in randomized trials: examples of the Thai RV144 HIV vaccine and South African CAPRISA 004 microbicide trials.

Recent HIV prevention trials have given hope that a suite of interventions which effectively reduce individuals' risk of HIV infection will soon be widely available [1–6]. In two studies, the RV144 and CAPRISA 004 trials, the relative risk of infection increased towards the null value of one over time. The RV144 and CAPRISA investigators interpreted these trends as evidence that the interventions’ effects declined over the study period and suggested that their respective findings may be explained by waning vaccine efficacy and decreasing adherence [2,7,8]. Here, we discuss these trends in the trials’ results and note that, in addition to the possible mechanisms cited by the investigators, their apparent waning efficacy may be explained in part by selection bias due to heterogeneity in infection risk, an explanation that has not been considered previously. This bias arises when study participants vary in their susceptibility to infection, e.g., because of differences in immune systems or exposure to infection. This can lead to increasing differences in the composition of the study population in each trial arm over time as those at highest risk become infected, and can occur despite comparability between arms at baseline. This issue is termed “frailty” in statistics and demography, where a large body of literature addresses the matter [9–14]. We also discuss several methods that can improve understanding of the effects of infectious disease interventions and risk factors by assessing the impact of frailty on results. Variation in frailty among study participants likely creates trends in the incidence of infection over the course of a study. To understand this phenomenon, consider a theoretical placebo-controlled randomized trial in which the risk of infection varies among participants. The highest risk individuals in such a trial are expected to become infected earlier, leaving a pool of lower risk individuals at later time points. If the intervention being tested is effective, the decline in the incidence of infection over time will be larger in the placebo arm because these individuals experience no direct protection from the intervention, and so those at high-risk will be quickly depleted, thereby lowering the infection rate over follow-up. However, high-risk individuals in the active arm may remain uninfected due to the protection conferred by the intervention, so the active arm’s infection rate will be less affected. Consequently, the time-specific rate ratio (RR) for treatment vs. placebo will increase over time from a value of less than one initially to a value that may exceed one later. This phenomenon has also been termed “survivor bias”, “survivor cohort effect”, “crossing of hazards” and “depletion of susceptibles”, and is observed in both chronic and infectious disease research [15–19]. Many randomized trials report a weighted average of the time-specific RR over the entire follow-up, but, because of the phenomenon described above, the average RR will become increasingly attenuated as follow-up time increases. This attenuation will occur even if all risk factors were balanced between study arms at baseline, as expected in a large randomized trial, and if the biological efficacy of the intervention is constant over time, which can lead investigators to reject an efficacious intervention. Because the results of any rate analysis can be affected by frailty, it can also cause investigators to overlook a risk factor for infection that is in fact harmful. To illustrate these issues we assessed whether the results of the RV144 and CAPRISA trials are consistent with an attenuation of the average RR due to frailty [1,2]. Though we focused on the widely discussed RR estimates, the RV144 team also presented Kaplan-Meier estimates [7]. Tables 1–2 show average incidence rates of infection and RRs calculated from baseline up to several time points in each trial [1,2]. Both studies show decreasing incidence rates in the placebo arms and generally increasing RRs. The incidence rate of infection in the intervention arm of the CAPRISA microbicide trial decreased slightly [2]. The incidence rate of infection in the vaccine arm of the RV144 trial increased from 0–12 months to 0–24 months; it then decreased slightly but remained below that of the placebo arm. Table 1 RV144 HIV vaccine trial modified intent-to-treat (mITT) data[1] Table 2 CAPRISA HIV microbicide trial intent-to-treat (ITT) data[2] To test for variation in the time-specific RRs, we fit a Cox model with a product (“interaction”) term between the intervention arm and time (on the logarithmic scale) on study [20]. A positive coefficient for this product term would indicate that the time-specific RR increases over time, in the context of prevention trials where the average RR is less than one, this implies that the average RR becomes attenuated over time. Figure 2C of the RV144 publication provides information on the number of infections and loss to follow-up in each arm for various time points [1]. Using these data we reconstructed an interval-censored dataset of the trial for analysis with a Cox model. The coefficient of the product term was 0.42, indicating that the time-specific RR increased over time, but the 95% confidence interval was wide (−0.09, 0.93). However, as such tests have limited power, failure to reject the null hypothesis cannot be taken as strong evidence that there was no attenuation over time [21]. Because any possible trend in estimates may not be linear, we conducted several other tests of the proportional hazards assumption (e.g., using quadratic forms of time), and each produced similar results. There was insufficient information to perform a similar test on the CAPRISA data. The attenuation of the average RR suggested by our analysis of the RV144 trial may have been due to frailty, waning efficacy, differential loss to follow-up, random variability, or any combination of the above. Furthermore, we note that frailty will have a large impact on results only when disease incidence is high in at least one subgroup. The RV144 investigators report that immune responses among the vaccinated waned during the first 6 months after vaccination; therefore vaccine efficacy may have similarly waned [7]. As waning efficacy and frailty are not mutually exclusive mechanisms of attenuation, both may have contributed to the increasing trend in the RR. The CAPRISA investigators report that some of the increase in the RR may be explained by decreasing adherence to microbicide use [2]. While waning vaccine efficacy and trends in adherence may help to explain the increase in RR observed over the course of both trials, neither mechanism can explain the decreasing incidence in the placebo arms. In contrast, frailty parsimoniously explains both observations. Because RR estimates are less affected by frailty during early follow-up, this may partly explain why studies that stop early because a considerable benefit is found can give larger estimates than those that continue. This is an alternative, and non-mutually exclusive, explanation to random variation [22] that has been insufficiently addressed in the literature on biases involved in stopping trials early [22–25]. Similarly, frailty has not been considered as a candidate explanation for null results in HIV prevention trials [26–30], or for variation in estimated effects across studies in meta-analyses [31,32]. Three approaches for dealing with frailty have been suggested. First, data on known risk factors for infection, including direct or proxy data on exposure to infection, could be collected and used in sensitivity analyses. This approach typically adjusts for baseline risk factors only, and uses standard methods, such as Cox models [9,10,32,33]. However, it may not be feasible to measure all risk factors for infection, and some risk factors may not even be known. When the goal is to estimate a biological measure of the direct effect of an intervention or risk factor, frailty effects due to heterogeneity in exposure to infection can be minimized by censoring individuals when they are no longer at risk of infection. Data on exposure to sexually transmitted infections can be obtained from serodiscordant couples studies, from partner report, or when participants report having no partners during a study period, and so could not have been exposed to HIV via sexual routes [4,34–38]. Survival curves, which use risks instead of rates, are not subject to the same frailty effects discussed above and can be estimated for observational studies as well as trials [15,39]. Second, some have suggested a type of crossover design to reduce the magnitude of attenuation in trials that examine reversible interventions through limiting the depletion of those most susceptible to infection by alternately giving both study arms access to the protective intervention [30]. Third, alternative analytical methods have been proposed, such as frailty models, which can account for heterogeneity by modeling the distribution of risk in the population based on parametric assumptions, and which have been used to analyze several vaccine trials [40–42]. Bayesian hierarchical models have been used to estimate the efficacy of a previous HIV vaccine candidate, and permit the incorporation of data on contact patterns and the prevalence of infection [43]. Frailty models have been widely used, particularly when the assumption of proportional hazards is considered unlikely to hold due to frailty effects [14]. In summary, accounting for frailty may improve our understanding of study results and facilitate decision-making on the use of interventions and avoidance of risk factors.