Web-Based VR Experiments Powered by the Crowd

We build on the increasing availability of Virtual Reality (VR) devices and Web technologies to conduct behavioral experiments in VR using crowdsourcing techniques. A new recruiting and validation method allows us to create a panel of eligible experiment participants recruited from Amazon Mechanical Turk. Using this panel, we ran three different crowdsourced VR experiments, each reproducing one of three VR illusions: place illusion, embodiment illusion, and plausibility illusion. Our experience and worker feedback on these experiments show that conducting Web-based VR experiments using crowdsourcing is already feasible, though some challenges---including scale---remain. Such crowdsourced VR experiments on the Web have the potential to finally support replicable VR experiments with diverse populations at a low cost.

[1]  Jeremy N. Bailenson,et al.  Immersion at scale: Researcher's guide to ecologically valid mobile experiments , 2016, 2016 IEEE Virtual Reality (VR).

[2]  Michael S. Bernstein,et al.  Crowds in two seconds: enabling realtime crowd-powered interfaces , 2011, UIST.

[3]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[4]  R. Ulrich.  Natural Versus Urban Scenes , 1981 .

[5]  Niels Henze,et al.  "These are not my hands!": Effect of Gender on the Perception of Avatar Hands in Virtual Reality , 2017, CHI.

[6]  J. Dul,et al.  Knowledge workers' creativity and the role of the physical work environment , 2011 .

[7]  Crystal L. Hoyt,et al.  Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology , 2002 .

[8]  Ralph Schroeder,et al.  Possible Worlds: The Social Dynamic of Virtual Reality Technology , 1996 .

[9]  M. Zuckerman,et al.  Development of a situation-specific trait-state test for the prediction and measurement of affective responses. , 1977, Journal of consulting and clinical psychology.

[10]  R. Simons,et al.  Stress recovery during exposure to natural and urban environments , 1991 .

[11]  Brandon Van Der Heide,et al.  The Proteus Effect in Dyadic Communication , 2013, Commun. Res..

[12]  M. Nowak,et al.  Fairness versus reason in the ultimatum game. , 2000, Science.

[13]  Sarah Sitton,et al.  The Messy Desk Effect: How Tidiness Affects the Perception of Others , 1984 .

[14]  Ugur Güdükbay,et al.  Increasing the Sense of Presence in a Simulation Environment Using Image Generators Based on Visual Attention , 2010, PRESENCE: Teleoperators and Virtual Environments.

[15]  J. Bailenson,et al.  The Proteus Effect: The Effect of Transformed Self-Representation on Behavior , 2007 .

[16]  Siddharth Suri,et al.  Resilient cooperators stabilize long-run cooperation in the finitely repeated Prisoner's Dilemma , 2016, Nature Communications.

[17]  Soo Youn Oh,et al.  Let the Avatar Brighten Your Smile: Effects of Enhancing Facial Expressions in Virtual Environments , 2016, PloS one.

[18]  Reza Rawassizadeh,et al.  A persuasive approach for indoor environment tidiness , 2011 .

[19]  Jaron Lanier,et al.  Homuncular Flexibility in Virtual Reality , 2015, J. Comput. Mediat. Commun..

[20]  M. Krijn,et al.  Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia. , 2002, Behaviour research and therapy.

[21]  Katharina Reinecke,et al.  LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples , 2015, CSCW.

[22]  Walter S. Lasecki,et al.  LegionTools: A Toolkit + UI for Recruiting and Routing Crowds to Synchronous Real-Time Tasks , 2015, UIST.

[23]  Michael S. Bernstein,et al.  Expert crowdsourcing with flash teams , 2014, UIST.

[24]  Kangsoo Kim,et al.  The wobbly table: Increased social presence via subtle incidental movement of a real-virtual table , 2016, 2016 IEEE Virtual Reality (VR).

[25]  J. Bailenson,et al.  Transformed Representation 1 RUNNING HEAD: TRANSFORMED REPRESENTATION The Proteus Effect: Implications of Transformed Digital Self-Representation on Online and Offline Behavior , 2008 .

[26]  Sandra González-Bailón,et al.  Bit by bit: social research in the digital age , 2019, The Journal of Mathematical Sociology.

[27]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[28]  Raz Schwartz,et al.  All are welcome: Using VR ethnography to explore harassment behavior in immersive social virtual reality , 2017, 2017 IEEE Virtual Reality (VR).

[29]  Deltcho Valtchanov,et al.  Restorative Effects of Virtual Nature Settings , 2010, Cyberpsychology Behav. Soc. Netw..

[30]  S. Milgram,et al.  Note on the drawing power of crowds of different size. , 1969 .

[31]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[32]  Hunter G Hoffman,et al.  Virtual reality pain control during burn wound debridement of combat-related burn injuries using robot-like arm mounted VR goggles. , 2011, The Journal of trauma.

[33]  Peng Dai,et al.  POMDP-based control of workflows for crowdsourcing , 2013, Artif. Intell..

[34]  Krzysztof Z. Gajos,et al.  TurkServer: Enabling Synchronous and Longitudinal Online Experiments , 2012, HCOMP@AAAI.

[35]  Chris Dede,et al.  Many ways to walk a mile in another's moccasins: Type of social perspective taking and its effect on negotiation outcomes , 2015, Comput. Hum. Behav..

[36]  N. Bond,et al.  Museums as Restorative Environments , 2010 .

[37]  Lydia B. Chilton,et al.  Cascade: crowdsourcing taxonomy creation , 2013, CHI.

[38]  J. Lanier,et al.  Model of Illusions and Virtual Reality , 2017, Front. Psychol..

[39]  G. Evans,et al.  Restorative Effects of Natural Environment Experiences , 1991 .

[40]  D. Tingley,et al.  “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents , 2015 .

[41]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[42]  Duncan J. Watts,et al.  Cooperation and Contagion in Web-Based, Networked Public Goods Experiments , 2010, SECO.

[43]  Ryan Rahinel,et al.  Physical Order Produces Healthy Choices, Generosity, and Conventionality, Whereas Disorder Produces Creativity , 2013, Psychological science.

[44]  Burak ÇÇiflikli,et al.  Increasing the sense of presence in a simulation environment using image generators based on visual attention , 2010 .

[45]  J. Bailenson,et al.  Walk A Mile in Digital Shoes : The Impact of Embodied Perspective-Taking on The Reduction of Negative Stereotyping in Immersive Virtual Environments , 2006 .

[46]  D. Karmanov,et al.  Assessing the restorative potential of contemporary urban environment(s): Beyond the nature versus urban dichotomy , 2008 .

[47]  Michael S. Bernstein,et al.  Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision , 2016, CSCW.

[48]  Sun Joo Grace Ahn,et al.  The Effect of Embodied Experiences on Self-Other Merging, Attitude, and Helping Behavior , 2013 .

[49]  Jeremy N. Bailenson,et al.  The embodiment of sexualized virtual selves: The Proteus effect and experiences of self-objectification via avatars , 2013, Comput. Hum. Behav..

[50]  Mel Slater,et al.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[51]  Siddharth Suri,et al.  An Experimental Study of Team Size and Performance on a Complex Task , 2016, PloS one.

[52]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[53]  Mel Slater,et al.  First-Person Perspective Virtual Body Posture Influences Stress: A Virtual Reality Body Ownership Study , 2016, PloS one.

[54]  J. Bailenson,et al.  Virtual Superheroes: Using Superpowers in Virtual Reality to Encourage Prosocial Behavior , 2013, PloS one.

[55]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[56]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[57]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[58]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[59]  Enrico Rukzio,et al.  CarVR: Enabling In-Car Virtual Reality Entertainment , 2017, CHI.

[60]  Rob Miller,et al.  Real-time crowd control of existing interfaces , 2011, UIST.

[61]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.