Toward dynamic phenotypes and the scalable measurement of human behavior

Precision psychiatry demands the rapid, efficient, and temporally dense collection of large scale and multi-omic data across diverse samples, for better diagnosis and treatment of dynamic clinical phenomena. To achieve this, we need approaches for measuring behavior that are readily scalable, both across participants and over time. Efforts to quantify behavior at scale are impeded by the fact that our methods for measuring human behavior are typically developed and validated for single time-point assessment, in highly controlled settings, and with relatively homogeneous samples. As a result, when taken to scale, these measures often suffer from poor reliability, generalizability, and participant engagement. In this review, we attempt to bridge the gap between gold standard behavioral measurements in the lab or clinic and the large-scale, high frequency assessments needed for precision psychiatry. To do this, we introduce and integrate two frameworks for the translation and validation of behavioral measurements. First, borrowing principles from computer science, we lay out an approach for iterative task development that can optimize behavioral measures based on psychometric, accessibility, and engagement criteria. Second, we advocate for a participatory research framework (e.g., citizen science) that can accelerate task development as well as make large-scale behavioral research more equitable and feasible. Finally, we suggest opportunities enabled by scalable behavioral research to move beyond single time-point assessment and toward dynamic models of behavior that more closely match clinical phenomena.

[1]  Participant Engagement in and Perspectives on a Web-Based Mindfulness Intervention for 9-1-1 Telecommunicators: Multimethod Study , 2019, Journal of medical Internet research.

[2]  Sarah J. Swierenga,et al.  What we know about dyslexia and Web accessibility: a research review , 2010, Universal Access in the Information Society.

[3]  Gail P Jarvik,et al.  Return of results: Ethical and legal distinctions between research and clinical care , 2014, American journal of medical genetics. Part C, Seminars in medical genetics.

[4]  T. Bourgeron,et al.  Both rare and common genetic variants contribute to autism in the Faroe Islands , 2019, npj Genomic Medicine.

[5]  Yeliz Yesilada,et al.  Web users with autism: eye tracking evidence for differences , 2018, Behav. Inf. Technol..

[6]  S. Rauch,et al.  Digital devices and continuous telemetry: opportunities for aligning psychiatry and neuroscience , 2018, Neuropsychopharmacology.

[7]  Krzysztof Z. Gajos,et al.  The Influence of Personality Traits and Cognitive Load on the Use of Adaptive User Interfaces , 2017, IUI.

[8]  Charles Weijer,et al.  Informing study participants of research results: an ethical imperative. , 2003, IRB.

[9]  M. Sliwinski Measurement-Burst Designs for Social Health Research , 2008 .

[10]  Stefano Tranquillini,et al.  Keep it simple: reward and task design in crowdsourcing , 2013, CHItaly '13.

[11]  M. Russell,et al.  Annual Research Review: Ecological momentary assessment studies in child psychology and psychiatry. , 2020, Journal of child psychology and psychiatry, and allied disciplines.

[12]  Annchen R. Knodt,et al.  The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences , 2017, Behavior Research Methods.

[13]  Robert M. Califf,et al.  Return of Research Results to Study Participants: Uncharted and Untested , 2018, JAMA.

[14]  J. Ioannidis,et al.  Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature , 2017, PLoS biology.

[15]  Stefan Palan,et al.  Prolific.ac—A subject pool for online experiments , 2017 .

[16]  Panayiotis Zaphiris,et al.  A systematic approach to the development of research-based web design guidelines for older people , 2007, Universal Access in the Information Society.

[17]  Anthony G. Greenwald,et al.  Psychology data from the Race Implicit Association Test on the Project Implicit Demo website , 2014 .

[18]  A. Yancey,et al.  Effective recruitment and retention of minority research participants. , 2006, Annual review of public health.

[19]  Diane Nelson Bryen,et al.  Web accessibility design recommendations for people with cognitive disabilities , 2008 .

[20]  D. Gerstorf,et al.  Time-structured and net intraindividual variability: tools for examining the development of dynamic characteristics and processes. , 2009, Psychology and aging.

[21]  Jakob Nielsen,et al.  Iterative user-interface design , 1993, Computer.

[22]  Joanna A. Christodoulou,et al.  Dispelling the Myth: Training in Education or Neuroscience Decreases but Does Not Eliminate Beliefs in Neuromyths , 2017, Front. Psychol..

[23]  A. Goldstein-Piekarski,et al.  Computational Psychiatry: New Perspectives on Mental Illnessedited by A. David Redish, Ph.D., and Joshua A. Gordon, M.D., Ph.D. Cambridge, Mass., MIT Press, 2016, 424 pp., $45.00 (hardcover). , 2017, The American journal of psychiatry.

[24]  R. McNally Attentional bias for threat: Crisis or opportunity? , 2019, Clinical psychology review.

[25]  Chris Van Pelt,et al.  Designing a scalable crowdsourcing platform , 2012, SIGMOD Conference.

[26]  E. Fox,et al.  Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements , 2019, Advances in Methods and Practices in Psychological Science.

[27]  Guillermo Sapiro,et al.  Computer vision analysis captures atypical attention in toddlers with autism , 2019, Autism : the international journal of research and practice.

[28]  S. Kosslyn,et al.  Genes, brain and cognition , 2001, Nature Neuroscience.

[29]  Gitte Lindgaard,et al.  An exploration of relations between visual appeal, trustworthiness and perceived usability of homepages , 2011, TCHI.

[30]  Jun Kong,et al.  Discriminating clinical phases of recovery from major depressive disorder using the dynamics of facial expression , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[31]  Steven J. Luck,et al.  Behavioral and ERP measures of attentional bias to threat in the dot-probe task: poor reliability and lack of correlation with anxiety , 2014, Front. Psychol..

[32]  A. Schwartz,et al.  Patient‐reported outcome measures for young people with developmental disabilities: incorporation of design features to reduce cognitive demands , 2018, Developmental medicine and child neurology.

[33]  L. Foschini,et al.  Accelerating Research With Technology: Rapid Recruitment for a Large-Scale Web-Based Sleep Study , 2019, JMIR research protocols.

[34]  Carol R. Horowitz,et al.  Community-Based Participatory Research From the Margin to the Mainstream: Are Researchers Prepared? , 2009, Circulation.

[35]  Constantine Stephanidis,et al.  Universal Design in HCI: A critical review of current research and practice , 2001 .

[36]  Katharina Reinecke,et al.  LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples , 2015, CSCW.

[37]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[38]  S. Shiffman,et al.  Ecological momentary assessment. , 2008, Annual review of clinical psychology.

[39]  H. Le,et al.  Recruiting Latino women in the U.S. and women in Mexico in postpartum depression prevention research , 2008, Archives of Women's Mental Health.

[40]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[41]  W. Dixon,et al.  Engagement and Participant Experiences With Consumer Smartwatches for Health Research: Longitudinal, Observational Feasibility Study , 2019, JMIR mHealth and uHealth.

[42]  Katharina Reinecke,et al.  Volunteer-Based Online Studies With Older Adults and People with Disabilities , 2018, ASSETS.

[43]  W. Dixon,et al.  Engagement and Participant Experiences With Consumer Smartwatches for Health Research: Longitudinal, Observational Feasibility Study , 2019, JMIR mHealth and uHealth.

[44]  C. Schmitt,et al.  The Ink4a/Arf locus operates as a regulator of the circadian clock modulating RAS activity , 2017, PLoS biology.

[45]  Peter C. M. Molenaar,et al.  The New Person-Specific Paradigm in Psychology , 2009 .

[46]  Stephen D. Shapiro,et al.  An Optimal Approach to Fault Tolerant Software Systems Design , 1978, IEEE Transactions on Software Engineering.

[47]  A. Kent,et al.  Population biobanks and returning individual research results: mission impossible or new directions? , 2011, Human Genetics.

[48]  P. Bower,et al.  Barriers to recruiting ethnic minorities to mental health research: a systematic review , 2014, International journal of methods in psychiatric research.

[49]  Daniel Ruderman,et al.  The emergence of dynamic phenotyping , 2017, Cell Biology and Toxicology.

[50]  Eoin A. King,et al.  Testing the accuracy of smartphones and sound level meter applications for measuring environmental noise , 2016 .

[51]  John Torous,et al.  Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements , 2018, Evidence Based Journals.

[52]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[53]  M. Sigman,et al.  Automated analysis of free speech predicts psychosis onset in high-risk youths , 2015, npj Schizophrenia.

[54]  R. Cohen The Neuropsychology of Attention , 1993, Critical Issues in Neuropsychology.

[55]  Ron Kohavi,et al.  Online Controlled Experiments and A/B Testing , 2017, Encyclopedia of Machine Learning and Data Mining.

[56]  J. Neuhaus,et al.  Recruitment and retention of healthy minority women into community-based longitudinal research. , 2001, Journal of women's health & gender-based medicine.

[57]  N. Ejiogu,et al.  Recruitment and Retention Strategies for Minority or Poor Clinical Research Participants: Lessons From the Healthy Aging in Neighborhoods of Diversity Across the Life Span Study , 2011, The Gerontologist.

[58]  Patrick G. Bissett,et al.  Large-scale analysis of test–retest reliabilities of self-regulation measures , 2019, Proceedings of the National Academy of Sciences.

[59]  William Revelle,et al.  Selected ICAR Data from the SAPA-Project: Development and Initial Validation of a Public-Domain Measure , 2016 .

[60]  T. McCoy,et al.  Improving Prediction of Suicide and Accidental Death After Discharge From General Hospitals With Natural Language Processing. , 2016, JAMA psychiatry.

[61]  Adam Gazzaley,et al.  Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense , 2016, BMJ Innovations.

[62]  S. Lea,et al.  The Effect of Longitudinal Burden on Survey Participation , 2022 .

[63]  S. Waechter,et al.  Measuring Attentional Bias to Threat: Reliability of Dot Probe and Eye Movement Indices , 2014, Cognitive Therapy and Research.

[64]  S. Rauch,et al.  Harnessing Smartphone-Based Digital Phenotyping to Enhance Behavioral and Mental Health , 2016, Neuropsychopharmacology.

[65]  Andrew Raij,et al.  Exploring micro-incentive strategies for participant compensation in high-burden studies , 2011, UbiComp '11.

[66]  Joachim Vandekerckhove,et al.  Metastudies for robust tests of theory , 2018, Proceedings of the National Academy of Sciences.

[67]  Andrea Cornwall,et al.  What is participatory research? , 1995, Social science & medicine.

[68]  Katharina Reinecke,et al.  A Case for Design Localization: Diversity of Website Aesthetics in 44 Countries , 2018, CHI.

[69]  L. Germine,et al.  Digital neuropsychology: Challenges and opportunities at the intersection of science and software , 2019, The Clinical neuropsychologist.

[70]  Steven H. Jones,et al.  Actigraphic assessment of circadian activity and sleep patterns in bipolar disorder. , 2005, Bipolar disorders.

[71]  J. Miranda,et al.  Introduction to the special section on recruiting and retaining minorities in psychotherapy research. , 1996, Journal of consulting and clinical psychology.

[72]  Joshua K. Hartshorne,et al.  When Does Cognitive Functioning Peak? The Asynchronous Rise and Fall of Different Cognitive Abilities Across the Life Span , 2015, Psychological science.

[73]  T. Rodebaugh,et al.  Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. , 2016, Journal of abnormal psychology.

[74]  D. Hawthorn,et al.  Interface design and engagement with older people , 2007, Behav. Inf. Technol..

[75]  Victor R. Basili,et al.  Iterative enhancement: A practical technique for software development , 1975, IEEE Transactions on Software Engineering.

[76]  Charis Eng,et al.  Largescale population genomics versus deep phenotyping: Brute force or elegant pragmatism towards precision medicine , 2019, npj Genomic Medicine.

[77]  K. Nakayama,et al.  Please Scroll down for Article Cognitive Neuropsychology Family Resemblance: Ten Family Members with Prosopagnosia and Within-class Object Agnosia , 2022 .

[78]  J. Smyth,et al.  Reliability and Validity of Ambulatory Cognitive Assessments , 2018, Assessment.

[79]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[80]  J. Smyth,et al.  Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. , 2010, British journal of health psychology.

[81]  David R. Williams,et al.  Social sources of racial disparities in health. , 2005, Health affairs.

[82]  Katharina Reinecke,et al.  Personalized Feedback Versus Money: The Effect on Reliability of Subjective Data in Online Experimental Platforms , 2017, CSCW Companion.

[83]  Katharina Reinecke,et al.  Citizen Science Opportunities in Volunteer-Based Online Experiments , 2017, CHI.

[84]  George Church,et al.  Ethical and Practical Guidelines for Reporting Genetic Research Results to Study Participants: Updated Guidelines from a National Heart, Lung, and Blood Institute Working Group , 2010, Circulation. Cardiovascular genetics.

[85]  Melissa L. Anderson,et al.  Deaf ACCESS: Adapting Consent Through Community Engagement and State-of-the-Art Simulation. , 2019, Journal of deaf studies and deaf education.

[86]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[87]  M. Sajatovic,et al.  Research with Severely Mentally Ill Latinas: Successful Recruitment and Retention Strategies , 2008, Journal of Immigrant and Minority Health.

[88]  P. Sankar,et al.  The Precision Medicine Initiative’s All of Us Research Program: an agenda for research on its ethical, legal, and social issues , 2016, Genetics in Medicine.

[89]  G. Cecchi,et al.  Prediction of psychosis across protocols and risk cohorts using automated language analysis , 2018, World psychiatry : official journal of the World Psychiatric Association.

[90]  Renaldo Bernard,et al.  Barriers and Facilitation Measures Related to People With Mental Disorders When Using the Web: A Systematic Review , 2016, Journal of medical Internet research.

[91]  Patrick C. Staples,et al.  Relapse prediction in schizophrenia through digital phenotyping: a pilot study , 2018, Neuropsychopharmacology.

[92]  Longitudinal survey development and design , 2007 .

[93]  Katharina Reinecke,et al.  Types of Motivation Affect Study Selection, Attention, and Dropouts in Online Experiments , 2017, Proc. ACM Hum. Comput. Interact..

[94]  William Revelle,et al.  The international cognitive ability resource: Development and initial validation of a public-domain measure , 2014 .

[95]  Heidi L Rehm,et al.  Return of genomic results to research participants: the floor, the ceiling, and the choices in between. , 2014, American journal of human genetics.

[96]  P. Areán,et al.  Recruitment and retention of older minorities in mental health services research. , 2003, The Gerontologist.

[97]  N. Wallerstein,et al.  Community-based participatory research for health : from process to outcomes , 2008 .

[98]  John Torous,et al.  User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity. , 2019, Psychiatric services.

[99]  Hongtu Chen,et al.  Engaging Asian Americans for Mental Health Research: Challenges and Solutions , 2005, Journal of Immigrant Health.

[100]  S. Kosslyn,et al.  Bridging psychology and biology. The analysis of individuals in groups. , 2002, The American psychologist.