Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

Background The evaluation of web-based interventions (defined as an intervention that can be downloaded or accessed on the internet through a web browser) in randomized controlled trials (RCTs) has increased over the past two decades. Little is known about how participants’ use of the intervention is measured, reported, and analyzed in these studies. Objective This study aimed to review the evaluation of web-based interventions in RCTs, assessing study characteristics and the methods used to record, and adjust for, intervention usage. Methods A systematic review of the literature was undertaken to identify all published reports of RCTs that involved a web-based intervention. A random sample of 100 published trials was selected for detailed data extraction. Information on trial characteristics was extracted, including whether web usage data were recorded, and if so, the methods used to gather these data and whether these data were used to inform efficacy analyses. Results A PubMed search identified 812 trials of web-based interventions published up to the end of 2017 and demonstrated a growing trend over time. Of the 100 studies reviewed, 90 studies collected web usage data, but more than half (49/90, 54%) of these studies did not state the method used for recording web usage. Only four studies attempted to check on the reliability of their web usage data collection methods. A total of 39% (35/90) studies reported patterns or levels of web intervention use, of which 21% (19/90) studies adjusted for intervention use in their outcome analysis, but only two of these used appropriate statistical methods. Conclusions Trialists frequently report a measure of web-based intervention usage but do not always report the collection method or provide enough detail on their analysis of web usage. Appropriate statistical methods to account for intervention use are rarely used and are not well reported even in the very few trials in which they are used. The number of trialists who attempt to check on the reliability of their web usage collection methods is extremely low.