Statistical Analysis and Design of Crowdsourcing Applications

STATISTICAL ANALYSIS AND DESIGN OF CROWDSOURCING APPLICATIONS

[1]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[2]  J. Friedman.  Stochastic gradient boosting , 2002 .

[3]  R. Larsen,et al.  The Satisfaction with Life Scale , 1985, Journal of personality assessment.

[4]  Adam Kapelner,et al.  Bayesian Additive Regression Trees With Parametric Models of Heteroskedasticity , 2014 .

[5]  Stanford E. Taylor.  Eye Movements in Reading: Facts and Fallacies , 1965 .

[6]  J. Angrist,et al.  Estimation of Limited Dependent Variable Models With Dummy Endogenous Regressors , 2001 .

[7]  C. Keyes,et al.  The structure of psychological well-being revisited. , 1995, Journal of personality and social psychology.

[8]  C. Assaid,et al.  The Theory of Response-Adaptive Randomization in Clinical Trials , 2007 .

[9]  Carlo Strapparava,et al.  Learning to identify emotions in text , 2008, SAC '08.

[10]  D. Rubin,et al.  MULTIPLE IMPUTATIONS IN SAMPLE SURVEYS-A PHENOMENOLOGICAL BAYESIAN APPROACH TO NONRESPONSE , 2002 .

[11]  R. Tibshirani,et al.  Bayesian Backfitting , 1998 .

[12]  Marion K Campbell,et al.  The method of minimization for allocation to clinical trials. a review. , 2002, Controlled clinical trials.

[13]  J. Rowe,et al.  Human aging: usual and successful. , 1987, Science.

[14]  S. Kruger.  Design Of Observational Studies , 2016 .

[15]  Susan Holmes,et al.  Quantitative, Architectural Analysis of Immune Cell Subsets in Tumor-Draining Lymph Nodes from Breast Cancer Patients and Healthy Lymph Nodes , 2010, PloS one.

[16]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[17]  Cindy K. Chung,et al.  The development and psychometric properties of LIWC2007 , 2007 .

[18]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[19]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[20]  Jeffrey H Silber,et al.  Optimal multivariate matching before randomization. , 2004, Biostatistics.

[21]  Moses Abramovitz.  Economic Growth and Its Discontents , 1973 .

[22]  Scott A. Golder,et al.  Diurnal and Seasonal Mood Vary with Work, Sleep, and Daylength Across Diverse Cultures , 2011 .

[23]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[24]  Peter Bühlmann,et al.  MissForest - non-parametric missing value imputation for mixed-type data , 2011, Bioinform..

[25]  Wendy McColskey,et al.  Measuring Student Engagement in Upper Elementary through High School: A Description of 21 Instruments. Summary. Issues & Answers. REL 2011-No. 098. , 2011 .

[26]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[27]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[28]  B. Efron.  Forcing a sequential experiment to be balanced , 1971 .

[29]  Adam D. I. Kramer.  An unobtrusive behavioral model of "gross national happiness" , 2010, CHI.

[30]  S. Raudenbush,et al.  Strategies for Improving Precision in Group-Randomized Experiments , 2007 .

[31]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[32]  Mike Wells,et al.  Structured Models for Fine-to-Coarse Sentiment Analysis , 2007, ACL.

[33]  Anne Elizabeth Preston,et al.  The Nonprofit Worker in a For-Profit World , 1989, Journal of Labor Economics.

[34]  Mark Stevenson,et al.  Introduction to the special issue on word sense disambiguation , 2004, Comput. Speech Lang..

[35]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[36]  Susan Holmes,et al.  An Interactive Java Statistical Image Segmentation System: GemIdent. , 2009, Journal of statistical software.

[37]  James K. Harter,et al.  INTERPERSONAL RELATIONS AND GROUP PROCESSES Wealth and Happiness Across the World : Material Prosperity Predicts Life Evaluation , Whereas Psychosocial Prosperity Predicts Positive Feeling , 2010 .

[38]  Jeffrey S. Simonoff,et al.  An Investigation of Missing Data Methods for Classification Trees , 2006, J. Mach. Learn. Res..

[39]  H. Friedman.  The Oxford Handbook of Health Psychology , 2013 .

[40]  Pushmeet Kohli,et al.  Personality and patterns of Facebook usage , 2012, WebSci '12.

[41]  Adam Kapelner,et al.  Prediction with missing data via Bayesian Additive Regression Trees , 2013, ArXiv.

[42]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[43]  Jerome H. Friedman.  Multivariate adaptive regression splines (with discussion) , 1991 .

[44]  E. M. Elderton.  THE LANARKSHIRE MILK EXPERIMENT , 1933 .

[45]  Nancy Ide,et al.  Introduction to the Special Issue on Word Sense Disambiguation: The State of the Art , 1998, Comput. Linguistics.

[46]  Hsin-Hsi Chen,et al.  Emotion Modeling from Writer/Reader Perspectives Using a Microblog Dataset , 2011 .

[47]  Edward I. George,et al.  Variable selection for BART: An application to gene regulation , 2013, 1310.4887.

[48]  James J. Appleton,et al.  Student engagement with school: Critical conceptual and methodological issues of the construct. , 2008 .

[49]  Shein-Chung Chow,et al.  Adaptive design methods in clinical trials – a review , 2008, Orphanet journal of rare diseases.

[50]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[51]  K. Mealey,et al.  Clinical pharmacology and therapeutics. , 2013, The Veterinary clinics of North America. Small animal practice.

[52]  Christiane Fellbaum,et al.  Analysis of a Hand-Tagging Task , 1997, Workshop On Tagging Text With Lexical Semantics: Why, What, And How?.

[53]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[54]  Martin E. P. Seligman,et al.  Doing the right thing: Measuring wellbeing for public policy , 2011 .

[55]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[56]  Sam K. Hui,et al.  Green-lighting Movie Scripts : Revenue Forecasting and Risk Management , 2010 .

[57]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[58]  清川 英男,et al.  CHALL, J. S. and DALE, E. (1995) Readability Revisited : The New Dale-Chall Readability Formula., Brookline Books , 1996 .

[59]  Adam Kapelner,et al.  bartMachine: A Powerful Tool for Machine Learning , 2013, ArXiv.

[60]  S. Pocock,et al.  Sequential treatment assignment with balancing for prognostic factors in the controlled clinical trial. , 1975, Biometrics.

[61]  Eli Biham,et al.  Advances in Cryptology — EUROCRYPT 2003 , 2003, Lecture Notes in Computer Science.

[62]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[63]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[64]  B. Hansen,et al.  Optimal Full Matching and Related Designs via Network Flows , 2006 .

[65]  Ronald Inglehart,et al.  Theory and Validity of Life Satisfaction Scales , 2013 .

[66]  Jun S. Liu,et al.  Extracting sequence features to predict protein–DNA interactions: a comparative study , 2008, Nucleic acids research.

[67]  KyungMann Kim.  Group Sequential Methods with Applications to Clinical Trials , 2001 .

[68]  M. Seligman.  Flourish: A Visionary New Understanding of Happiness and Well-being , 2011 .

[69]  Avalanche Forecasting: Using Bayesian Additive Regression Trees (BART) , 2014 .

[70]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[71]  C. B. Colby.  The weirdest people in the world , 1973 .

[72]  E. George.  Capture—recapture estimation via Gibbs sampling , 1992 .

[73]  Rada Mihalcea,et al.  Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation , 2010, Mturk@HLT-NAACL.

[74]  A. Bakker,et al.  The Measurement of Work Engagement With a Short Questionnaire , 2006 .

[75]  L. Eyde,et al.  Psychological testing and psychological assessment. A review of evidence and issues. , 2001, The American psychologist.

[76]  Deborah M. Gordon,et al.  The effect of individual variation on the structure and function of interaction networks in harvester ants , 2011, Journal of The Royal Society Interface.

[77]  C B Begg,et al.  The impact of treatment allocation procedures on nominal significance levels and bias. , 1987, Controlled clinical trials.

[78]  Klaus Krippendorff,et al.  Estimating the Reliability, Systematic Error and Random Error of Interval Data , 1970 .

[79]  Shalabh.  Design of Experiments: an Introduction based on Linear Models , 2012 .

[80]  Steven D. Levitt,et al.  What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World , 2007 .

[81]  R. McCrae,et al.  An introduction to the five-factor model and its applications. , 1992, Journal of personality.

[82]  Steven D. Levitt,et al.  FIELD EXPERIMENTS IN ECONOMICS : THE PAST , THE PRESENT , AND THE FUTURE , 2008 .

[83]  Damian McEntegart,et al.  Randomization by minimization for unbalanced treatment allocation , 2009, Statistics in medicine.

[84]  Jon Oberlander,et al.  What Are They Blogging About? Personality, Topic and Motivation in Blogs , 2009, ICWSM.

[85]  Nancy Cartwright,et al.  Are RCTs the Gold Standard? , 2007 .

[86]  E. Diener,et al.  Social relations, health behaviors, and health outcomes: a survey and synthesis. , 2013, Applied psychology. Health and well-being.

[87]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[88]  L. Pearlin,et al.  The structure of coping. , 1978, Journal of health and social behavior.

[89]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[90]  Shelley E. Taylor.  Social Support: A Review , 2011 .

[91]  Brent D. Rosso,et al.  On the meaning of work: A theoretical integration and review , 2010 .

[92]  David J. Hand,et al.  Good methods for coping with missing data in decision trees , 2008, Pattern Recognit. Lett..

[93]  Russ B. Altman,et al.  Missing value estimation methods for DNA microarrays , 2001, Bioinform..

[94]  H. Simon,et al.  A Behavioral Model of Rational Choice , 1955 .

[95]  Jonathon Read,et al.  Using Emoticons to Reduce Dependency in Machine Learning Techniques for Sentiment Classification , 2005, ACL.

[96]  G. Harrison,et al.  Field experiments , 1924, The Journal of Agricultural Science.

[97]  Y. Hayakawa,et al.  A BETA‐BINOMIAL MODEL FOR ESTIMATING THE SIZE OF A HETEROGENEOUS POPULATION , 2005 .

[98]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[99]  Martha Palmer,et al.  Number or Nuance: Which Factors Restrict Reliable Word Sense Annotation? , 2010, LREC.

[100]  Emil Pitkin,et al.  Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation , 2013, 1309.6392.

[101]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[102]  C B Begg,et al.  A treatment allocation procedure for sequential clinical trials. , 1980, Biometrics.

[103]  T. Kashdan,et al.  Understanding the search for meaning in life: personality, cognitive style, and the dynamic between seeking and experiencing meaning. , 2008, Journal of personality.

[104]  D R Taves,et al.  Minimization: A new method of assigning patients to treatment and control groups , 1974, Clinical pharmacology and therapeutics.

[105]  James R. Gattiker,et al.  Parallel Bayesian Additive Regression Trees , 2013, 1309.1906.

[106]  Damian McEntegart,et al.  The Pursuit of Balance Using Stratified and Dynamic Randomization Techniques: An Overview , 2003 .

[107]  A. Kapelner,et al.  An Interactive Statistical Image Segmentation and Visualization System , 2007, International Conference on Medical Information Visualisation - BioMedical Visualisation (MediVis 2007).

[108]  Gregory J. Park,et al.  Predicting Dark Triad Personality Traits from Twitter Usage and a Linguistic Analysis of Tweets , 2012, 2012 11th International Conference on Machine Learning and Applications.

[109]  M. Csíkszentmihályi.  Creativity: Flow and the Psychology of Discovery and Invention , 1996 .

[110]  Rada Mihalcea,et al.  Exploiting Agreement and Disagreement of Human Annotators for Word Sense Disambiguation , 2003 .

[111]  A. Atkinson.  Optimum biased coin designs for sequential clinical trials with prognostic factors , 1982 .

[112]  R. Little.  Pattern-Mixture Models for Multivariate Incomplete Data , 1993 .

[113]  Ananthanarayanan Parasuraman,et al.  Assessing response quality. A self‐disclosure approach to assessing response quality in mall intercept and telephone interviews , 1984 .

[114]  J. Cacioppo,et al.  The need for cognition. , 1982 .

[115]  Bo Pang,et al.  A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts , 2004, ACL.

[116]  David W. Aha,et al.  Instance‐based prediction of real‐valued attributes , 1989, Comput. Intell..

[117]  H Merabet,et al.  The design and analysis of sequential clinical trials , 2013 .

[118]  Martha Palmer,et al.  SemEval-2007 Task-17: English Lexical Sample, SRL and All Words , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[119]  Bhekisipho Twala,et al.  AN EMPIRICAL COMPARISON OF TECHNIQUES FOR HANDLING INCOMPLETE DATA USING DECISION TREES , 2009, Appl. Artif. Intell..

[120]  James W. Pennebaker,et al.  Linguistic Inquiry and Word Count (LIWC2007) , 2007 .

[121]  Tal Yarkoni.  Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers. , 2010, Journal of research in personality.

[122]  Rachel T. A. Croson,et al.  Gender Differences in Preferences , 2009 .

[123]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[124]  D. Rubin,et al.  Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies , 1978 .

[125]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[126]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[127]  Richard H. Thaler,et al.  Mental Accounting and Consumer Choice , 1985, Mark. Sci..

[128]  J. Stiglitz,et al.  Report by the commission on the measurement of economic performance and social progress , 2011 .

[129]  R. Simon,et al.  Restricted randomization designs in clinical trials. , 1979, Biometrics.

[130]  Roger Tourangeau,et al.  Taking the Audio Out of Audio-CASI , 2009 .

[131]  Roberto Navigli,et al.  Word sense disambiguation: A survey , 2009, CSUR.

[132]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[133]  James J. Appleton,et al.  Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument , 2006 .

[134]  D. Ariely,et al.  Man's search for meaning: The case of Legos , 2008 .

[135]  E. Lukacz,et al.  Effect of amitriptyline on symptoms in treatment naïve patients with interstitial cystitis/painful bladder syndrome. , 2010, The Journal of urology.

[136]  W. Rosenberger,et al.  The theory of response-adaptive randomization in clinical trials , 2006 .

[137]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[138]  E. Diener,et al.  Review of the Satisfaction with Life Scale , 1993 .

[139]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[140]  Cecilia Ovesdotter Alm,et al.  Emotions from Text: Machine Learning for Text-based Emotion Prediction , 2005, HLT.

[141]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[142]  K. Anders Ericsson,et al.  Attaining Excellence Through Deliberate Practice: Insights from the Study of Expert Performance , 2008 .

[143]  Hugo Liu,et al.  A Corpus-based Approach to Finding Happiness , 2006, AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.

[144]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[145]  Adam Kapelner,et al.  Matching on‐the‐fly: Sequential allocation with higher power and efficiency , 2014, Biometrics.

[146]  Kari Lock Morgan,et al.  Rerandomization to improve covariate balance in experiments , 2012, 1207.5625.

[147]  P. Rothwell,et al.  External validity of randomised controlled trials: “To whom do the results of this trial apply?” , 2005, The Lancet.

[148]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[149]  Richard E. Lucas,et al.  Predictors of Regional Well-Being: A County Level Analysis , 2011 .

[150]  Maxine Eskénazi,et al.  Clustering dictionary definitions using Amazon Mechanical Turk , 2010, Mturk@HLT-NAACL.

[151]  Marilyn A. Walker,et al.  Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text , 2007, J. Artif. Intell. Res..

[152]  Richard Layard,et al.  Rethinking public economics: the implications of rivalry and habit , 2003 .

[153]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[154]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[155]  Stan Szpakowicz,et al.  Identifying Expressions of Emotion in Text , 2007, TSD.

[156]  Christopher M. Danforth,et al.  Temporal Patterns of Happiness and Information in a Global Social Network: Hedonometrics and Twitter , 2011, PloS one.

[157]  Damaraju Raghavarao.  Use of Distance Function in Sequential Treatment Assignment for Prognostic Factors in the Controlled Clinical Trial , 1980 .

[158]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[159]  J. Hoffman,et al.  The role of visual attention in saccadic eye movements , 1995, Perception & psychophysics.

[160]  Dana Chandler,et al.  Preventing Satisficing in Online Surveys: A "Kapcha" to Ensure Higher Quality Data , 2010 .

[161]  M. Seligman,et al.  Pursuit of pleasure, engagement, and meaning: Relationships to subjective and objective measures of well-being , 2010 .

[162]  David E. Booth,et al.  Analysis of Incomplete Multivariate Data , 2000, Technometrics.

[163]  William F. Rosenberger,et al.  Handling Covariates in the Design of Clinical Trials. , 2008, 1102.3773.

[164]  C. F. Kao,et al.  The efficient assessment of need for cognition. , 1984, Journal of personality assessment.

[165]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[166]  David R. Anderson,et al.  Statistical inference from capture data on closed animal populations , 1980 .

[167]  B. Kindo,et al.  MBACT - Multiclass Bayesian Additive Classication Trees , 2013 .

[168]  Jon Oberlander,et al.  Identifying more bloggers: Towards large scale personality classification of personal weblogs , 2007, ICWSM.

[169]  John Langford,et al.  CAPTCHA: Using Hard AI Problems for Security , 2003, EUROCRYPT.

[170]  Ari Rappoport,et al.  Enhanced Sentiment Learning Using Twitter Hashtags and Smileys , 2010, COLING.

[171]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[172]  Robert B. Gramacy,et al.  Dynamic Trees for Learning and Design , 2009, 0912.1586.

[173]  P. McCullagh.  Estimating the Number of Unseen Species: How Many Words did Shakespeare Know? , 2008 .

[174]  J. Chall,et al.  Readability revisited : the new Dale-Chall readability formula , 1995 .

[175]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .

[176]  Christian P. Robert,et al.  The Bayesian choice : from decision-theoretic foundations to computational implementation , 2007 .

[177]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[178]  Nansook Park,et al.  Positive Psychology as the Evenhanded Positive Psychologist Views It. , 2003 .

[179]  Dean P. Foster,et al.  New Insights from Coarse Word Sense Disambiguation in the Crowd , 2012, COLING.

[180]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[181]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[182]  Ed Diener,et al.  Very Happy People , 2002, Psychological science.

[183]  P. Costa,et al.  Validation of the five-factor model of personality across instruments and observers. , 1987, Journal of personality and social psychology.

[184]  Jon Sprouse.  A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory , 2010, Behavior research methods.

[185]  Rohit Prasad,et al.  Automatic Detection of Psychological Distress Indicators and Severity Assessment from Online Forum Posts , 2012, COLING.

[186]  Jon A. Krosnick,et al.  Satisficing in surveys: Initial evidence , 1996 .

[187]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[188]  T. Therneau,et al.  An Introduction to Recursive Partitioning Using the RPART Routines , 2015 .

[189]  Barry Schwartz,et al.  Jobs, Careers, and Callings: People's Relations to Their Work , 1997 .

[190]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[191]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[192]  George Metakides,et al.  Web Science , 2008, ECAI.

[193]  Stephen Senn.  Consensus and Controversy in Pharmaceutical Statistics , 2000 .