A systematic review of the quality of reporting of simulation studies about methods for the analysis of complex longitudinal patient-reported outcomes data

PurposeThis study describes the characteristics and quality of reporting for published computer simulation studies about statistical methods to analyze complex longitudinal (i.e., repeated measures) patient-reported outcomes (PROs); we included methods for longitudinal latent variable measurement and growth models and response shift.MethodsScopus, PsycINFO, PubMed, EMBASE, and Social Science Citation Index were searched for English-language studies published between 1999 and 2016 using selected keywords. Extracted information included characteristics of the study purpose/objectives, simulation design, software, execution, performance, and results. The quality of reporting was evaluated using published best-practice guidelines.SynthesisA total of 1470 articles were reviewed and 42 articles met the inclusion criteria. The majority of the included studies (73.8%) investigated an existing statistical method, primarily a latent variable model (95.2%). Most studies specified the population model, including variable distributions, mean parameters, and correlation/covariances. The number of time points and sample size(s) were reported by all studies, but justification for the selected values was rarely provided. The majority of the studies (52.4%) did not report on model non-convergence. Bias, accuracy, and model fit were commonly reported performance metrics. All studies reported results descriptively, and 26.2% also used an inferential method.ConclusionsWhile methodological research on statistical analyses of complex longitudinal PRO data is informed by computer simulation studies, current reporting practices of these studies have not been consistent with best-practice guidelines. Comprehensive reporting of simulation methods and results ensures that the strengths and limitations of the investigated statistical methods are thoroughly explored.

[1]  M. Valerio,et al.  Comparing two sampling methods to engage hard-to-reach communities in research priority setting , 2016, BMC Medical Research Methodology.

[2]  P. Fayers,et al.  Quality of Life: The Assessment, Analysis and Reporting of Patient-reported Outcomes , 2016 .

[3]  C. Schwartz,et al.  Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. , 1999, Social science & medicine.

[4]  A Skrondal,et al.  Design and Analysis of Monte Carlo Experiments: Attacking the Conventional Wisdom , 2000, Multivariate Behavioral Research.

[5]  C. Schwartz,et al.  Minimal evidence of response shift in the absence of a catalyst , 2014, Quality of Life Research.

[6]  J. Oud,et al.  Developments in statistical evaluation of clinical trials , 2014 .

[7]  Xitao Fan,et al.  Power of Latent Growth Modeling for Detecting Group Differences in Linear Growth Trajectory Parameters , 2003 .

[8]  V. Willson,et al.  Testing Measurement Invariance Across Groups in Longitudinal Data: Multigroup Second-Order Latent Growth Model , 2014 .

[9]  D. Andrade,et al.  Item response theory for longitudinal data: Item and population ability parameters estimation , 2006 .

[10]  Patrick Royston,et al.  The design of simulation studies in medical statistics , 2006, Statistics in medicine.

[11]  J. Bernhard,et al.  Quantitative assessment of changes in patients' constructs of quality of life: An application of multilevel models , 2004, Quality of Life Research.

[12]  Richard A. Feinberg,et al.  Conducting Simulation Studies in Psychometrics , 2016 .

[13]  Fan Jia,et al.  Planned missing designs to optimize the efficiency of latent growth parameter estimates , 2014 .

[14]  F. Guillemin,et al.  RespOnse Shift ALgorithm in Item response theory (ROSALI) for response shift detection with missing data in longitudinal patient-reported outcome studies , 2015, Quality of Life Research.

[15]  A. Morin,et al.  Statistical power of latent growth curve models to detect quadratic growth , 2014, Behavior research methods.

[16]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[17]  Michael Eid,et al.  Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses , 2015, Behavior research methods.

[18]  Kazuhiko Kuribayashi.  Design and Analysis of Clinical Trial Simulations , 2014 .

[19]  Sheng Luo,et al.  Robust Bayesian inference for multivariate longitudinal data by using normal/independent distributions , 2013, Statistics in medicine.

[20]  Jean-Paul Fox,et al.  Longitudinal measurement in health‐related surveys. A Bayesian joint growth model for multivariate ordinal responses , 2013, Statistics in medicine.

[21]  S. Iovleff,et al.  Three Stochastic Versions of the EM Algorithm for Estimating Longitudinal Rasch Model , 2003 .

[22]  Antoine Vanier,et al.  Overall performance of Oort’s procedure for response shift detection at item level: a pilot simulation study , 2015, Quality of Life Research.

[23]  V. Sébille,et al.  Assessment of score- and Rasch-based methods for group comparison of longitudinal patient-reported outcomes with intermittent missing data (informative and non-informative) , 2014, Quality of Life Research.

[24]  L. Lix,et al.  Montreal Accord on Patient-Reported Outcomes (PROs) use series-Paper 7: modern perspectives of measurement validation emphasize justification of inferences based on patient reported outcome scores. , 2017, Journal of clinical epidemiology.

[25]  Oi-Man Kwok,et al.  Using Modification Indexes to Detect Turning Points in Longitudinal Data: A Monte Carlo Study , 2010 .

[26]  A physiatrist's view of response shift , 2009 .

[27]  Kevin A Hallgren,et al.  Conducting Simulation Studies in the R Programming Environment. , 2013, Tutorials in quantitative methods for psychology.

[28]  David A Cole,et al.  Empirical and conceptual problems with longitudinal trait-state models: introducing a trait-state-occasion model. , 2005, Psychological methods.

[29]  Randall E. Schumacker,et al.  Random-Number Generator Validity in Simulation Studies: An Investigation of Normality , 1998 .

[30]  Carol Jagger,et al.  Assessing the validity of the Global Activity Limitation Indicator in fourteen European countries , 2015, BMC Medical Research Methodology.

[31]  I. Grama,et al.  Generalized Estimating Equations (GEE) for Mixed Logistic Models , 2003 .

[32]  Steffen Nestler.  How the 2SLS/IV estimator can handle equality constraints in structural equation models: a system-of-equations approach. , 2014, The British journal of mathematical and statistical psychology.

[33]  J. Twisk,et al.  Why item response theory should be used for longitudinal questionnaire data analysis in medical research , 2015, BMC Medical Research Methodology.

[34]  Mark Wilson,et al.  Formulating latent growth using an explanatory item response model approach. , 2012, Journal of applied measurement.

[35]  Edmund J. Crampin,et al.  Minimum Information About a Simulation Experiment (MIASE) , 2011, PLoS Comput. Biol..

[36]  L. Lix,et al.  Longitudinal Change in Response Processes: A Response Shift Perspective , 2017 .

[37]  M. Feddag,et al.  Power analysis on the time effect for the longitudinal Rasch model. , 2014, Journal of applied measurement.

[38]  S. Chacón-Moscoso,et al.  A Simulation Study of Threats to Validity in Quasi-Experimental Designs: Interrelationship between Design, Measurement, and Analysis , 2016, Front. Psychol..

[39]  Lisa M. Lix,et al.  Identifying reprioritization response shift in a stroke caregiver population: a comparison of missing data methods , 2015, Quality of Life Research.

[40]  N. Mayo,et al.  Identifying response shift statistically at the individual level , 2008, Quality of Life Research.

[41]  Anne Boomsma,et al.  Reporting Monte Carlo Studies in Structural Equation Modeling , 2013 .

[42]  S. Rabe-Hesketh,et al.  An autoregressive growth model for longitudinal item analysis , 2016, Psychometrika.

[43]  Jeffrey R. Harring,et al.  A Finite Mixture of Nonlinear Random Coefficient Models for Continuous Repeated Measures Data , 2016, Psychometrika.

[44]  Véronique Sébille,et al.  Rasch-family models are more valuable than score-based approaches for analysing longitudinal patient-reported outcomes with missing data , 2016, Statistical methods in medical research.

[45]  V. Sébille,et al.  Analysis of longitudinal Patient-Reported Outcomes with informative and non-informative dropout: Comparison of CTT and Rasch-based methods , 2011 .

[46]  Ginger Lockhart,et al.  First- Versus Second-Order Latent Growth Curve Models: Some Insights From Latent State-Trait Theory , 2013, Structural equation modeling : a multidisciplinary journal.

[47]  Kenneth A. Bollen,et al.  Monte Carlo Experiments: Design and Implementation , 2001 .

[48]  Klaas Sijtsma,et al.  Methodology Review: Evaluating Person Fit , 2001 .

[49]  S. Sterba.  Pattern Mixture Models for Quantifying Missing Data Uncertainty in Longitudinal Invariance Testing , 2017 .

[50]  L. Lix,et al.  Relative importance measures for reprioritization response shift , 2013, Quality of Life Research.

[51]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[52]  B. Falissard,et al.  Rasch modelling to deal with changes in the questionnaires used during long-term follow-up of cohort studies: a simulation study , 2016, BMC Medical Research Methodology.

[53]  V. Willson,et al.  Measurement Invariance Across Groups in Latent Growth Modeling , 2014 .

[54]  N. Logothetis,et al.  Is the frontal lobe involved in conscious perception? , 2014, Front. Psychol..

[55]  Dimiter Dobrev,et al.  Computer Simulation , 1966, J. Inf. Process. Cybern..

[56]  Andrea Marshall,et al.  Importance of protocols for simulation studies in clinical drug development , 2011, Statistical methods in medical research.

[57]  B. Zumbo,et al.  Minimal impact of response shift for SF-12 mental and physical health status in homeless and vulnerably housed individuals: an item-level multi-group analysis , 2017, Quality of Life Research.

[58]  Silvia Bacci,et al.  Longitudinal data: different approaches in the context of item-response theory models , 2012 .

[59]  V. Sébille,et al.  Comparison of CTT and Rasch‐based approaches for the analysis of longitudinal Patient Reported Outcomes , 2011, Statistics in medicine.

[60]  Su-Young Kim,et al.  Determining the Number of Latent Classes in Single- and Multiphase Growth Mixture Models , 2014, Structural equation modeling : a multidisciplinary journal.

[61]  Xitao Fan,et al.  Power of Latent Growth Modeling for Detecting Linear Growth: Number of Measurements and Comparison with Other Analytic Approaches. , 2005 .

[62]  L. Ring,et al.  Influence of response shift on evaluations of change in patient-reported outcomes , 2008, Expert review of pharmacoeconomics & outcomes research.

[63]  A Multilevel Higher Order Item Response Theory Model for Measuring Latent Growth in Longitudinal Data , 2015, Applied psychological measurement.

[64]  Stefan Höfer,et al.  Response shift masks the treatment impact on patient reported outcomes (PROs): the example of individual quality of life in edentulous patients , 2005, Health and quality of life outcomes.

[65]  D. Andrade,et al.  Item response theory for longitudinal data: population parameter estimation , 2005 .

[66]  R. Millsap.  Testing Measurement Invariance Using Item Response Theory in Longitudinal Data: An Introduction , 2010 .

[67]  Ryne Estabrook,et al.  Evaluating measurement of dynamic constructs: defining a measurement model of derivatives. , 2015, Psychological methods.