Citizen Science: An Information Quality Research Frontier

The rapid proliferation of online content producing and sharing technologies resulted in an explosion of user-generated content (UGC), which now extends to scientific data. Citizen science, in which ordinary people contribute information for scientific research, epitomizes UGC. Citizen science projects are typically open to everyone, engage diverse audiences, and challenge ordinary people to produce data of highest quality to be usable in science. This also makes citizen science a very exciting area to study both traditional and innovative approaches to information quality management. With this paper we position citizen science as a leading information quality research frontier. We also show how citizen science opens a unique opportunity for the information systems community to contribute to a broad range of disciplines in natural and social sciences and humanities.

[1]  Matt Germonprez,et al.  The Potential for Citizen Science in Information Systems Research , 2017, Commun. Assoc. Inf. Syst..

[2]  Krzysztof Z. Gajos,et al.  Crowdsourcing as a Tool for Research: Implications of Uncertainty , 2017, CSCW.

[3]  Z. Popovic,et al.  Crystal structure of a monomeric retroviral protease solved by protein folding game players , 2011, Nature Structural &Molecular Biology.

[4]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[5]  Roger Blake,et al.  From Content to Context , 2017, ACM J. Data Inf. Qual..

[6]  Andrea Wiggins,et al.  Community-based Data Validation Practices in Citizen Science , 2016, CSCW.

[7]  J. Pearse,et al.  Monitoring Rocky Intertidal Shorelines: A Role for the Public in Resource Management , 2005 .

[8]  Donald P. Ballou,et al.  Modeling Completeness versus Consistency Tradeoffs in Information Decision Contexts , 2003, IEEE Trans. Knowl. Data Eng..

[9]  Diane M. Strong,et al.  Data quality in context , 1997, CACM.

[10]  A. W. Galloway,et al.  The Reliability of Citizen Science: A Case Study of Oregon White Oak Stand Surveys , 2006 .

[11]  Heiko Gewald,et al.  Determinants of Intention to Participate in Corporate BYOD-Programs – The Case of Digital Natives – , 2015 .

[12]  Ron Weber,et al.  Research Commentary: Information Systems and Conceptual Modeling - A Research Agenda , 2002, Inf. Syst. Res..

[13]  Gaganmeet Kaur Awal,et al.  Leveraging collective intelligence for behavioral prediction in signed social networks through evolutionary approach , 2017, Information Systems Frontiers.

[14]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[15]  Kevin Crowston,et al.  Surveying the citizen science landscape , 2014, First Monday.

[16]  Guoliang Li,et al.  Crowdsourced Data Management: A Survey , 2016, IEEE Transactions on Knowledge and Data Engineering.

[17]  Anna L. Cox,et al.  “I want to be a captain! I want to be a captain!”: gamification in the old weather citizen science project , 2013, Gamification.

[18]  Rick Bonney,et al.  The Theory and Practice of Citizen Science: Launching a New Journal , 2016 .

[19]  Stuart E. Madnick,et al.  Special Section: Assuring Information Quality , 2004, J. Manag. Inf. Syst..

[20]  Tom Heath,et al.  Linked Data: Evolving the Web into a Global Data Space , 2011, Linked Data.

[21]  David W. Macdonald,et al.  Validating mammal monitoring methods and assessing the performance of volunteers in wildlife conservation—“Sed quis custodiet ipsos custodies ?” , 2003 .

[22]  Michael F. Goodchild,et al.  Please Scroll down for Article International Journal of Digital Earth Crowdsourcing Geographic Information for Disaster Response: a Research Frontier Crowdsourcing Geographic Information for Disaster Response: a Research Frontier , 2022 .

[23]  Aditya G. Parameswaran,et al.  Challenges in Data Crowdsourcing , 2016, IEEE Transactions on Knowledge and Data Engineering.

[24]  Jeff Howe,et al.  Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business , 2008, Human Resource Management International Digest.

[25]  Ofer Arazy,et al.  On the measurability of information quality , 2011, J. Assoc. Inf. Sci. Technol..

[26]  Roman Lukyanenko,et al.  The IQ of the Crowd: Understanding and Improving Information Quality in Structured User-Generated Content , 2014, Inf. Syst. Res..

[27]  Jan Recker,et al.  Sensemaking and Sustainable Practicing: Functional Affordances of Information Systems in Green Transformations , 2013, MIS Q..

[28]  Alessandro F. Garcia,et al.  VazaDengue: An information system for preventing and combating mosquito-borne diseases with social networks , 2018, Inf. Syst..

[29]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[30]  Magdalena Balazinska.  Big data research , 2015, VLDB 2015.

[31]  Kevin Crowston,et al.  Discontinuities and continuities: a new way to understand virtual work , 2002, Inf. Technol. People.

[32]  Christine B. Williams,et al.  Web 2.0 and Politics: The 2008 U.S. Presidential Election and an E-Politics Research Agenda , 2010, MIS Q..

[33]  K. Peterman,et al.  Exploring Embedded Assessment to Document Scientific Inquiry Skills Within Citizen Science , 2017 .

[34]  Veda C. Storey,et al.  A Framework for Analysis of Data Quality Research , 1995, IEEE Trans. Knowl. Data Eng..

[35]  Donald P. Ballou,et al.  Designing Information Systems to Optimize the Accuracy-Timeliness Tradeoff , 1995, Inf. Syst. Res..

[36]  Y. Wiersma.  Birding 2.0: Citizen Science and Effective Monitoring in the Web 2.0 World , 2010 .

[37]  Andrew P. McAfee,et al.  Machine, Platform, Crowd: Harnessing Our Digital Future , 2017 .

[38]  Carlo Batini,et al.  From Data Quality to Big Data Quality , 2015, J. Database Manag..

[39]  Tova Milo,et al.  Managing General and Individual Knowledge in Crowd Mining Applications , 2015, CIDR.

[40]  Roman Lukyanenko,et al.  An information modeling approach to improve quality of user-generated content , 2014 .

[41]  Stuart E. Madnick,et al.  Overview and Framework for Data and Information Quality Research , 2009, JDIQ.

[42]  Kevin Crowston,et al.  Gaming for (Citizen) Science: Exploring Motivation and Data Quality in the Context of Crowdsourced Science through the Design and Evaluation of a Social-Computational System , 2011, 2011 IEEE Seventh International Conference on e-Science Workshops.

[43]  Ben R. Newell,et al.  The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers , 2015, Judgment and Decision Making.

[44]  E. Hand,et al.  Citizen science: People power , 2010, Nature.

[45]  Panagiotis G. Ipeirotis,et al.  Quizz: targeted crowdsourcing with a billion (potential) users , 2014, WWW.

[46]  Michel Bariche,et al.  Citizen science detects the undetected: the case of Abudefduf saxatilis from the Mediterranean Sea , 2013 .

[47]  Thomas Redman,et al.  Data quality for the information age , 1996 .

[48]  Martin W. Bauer,et al.  Public Knowledge of and Attitudes to Science: Alternative Measures That May End the “Science War” , 2000 .

[49]  Brian M Baker,et al.  Citizens unite for computational immunology! , 2015, Trends in immunology.

[50]  Stephen J. Andriole,et al.  Business impact of Web 2.0 technologies , 2010, Commun. ACM.

[51]  ButlerBrian,et al.  Beyond the organizational 'container' , 2014 .

[52]  Eric Paulos,et al.  Sensr: evaluating a flexible framework for authoring mobile data-collection tools for citizen science , 2013, CSCW.

[53]  Kevin Crowston,et al.  From Conservation to Crowdsourcing: A Typology of Citizen Science , 2011, 2011 44th Hawaii International Conference on System Sciences.

[54]  Jay F. Nunamaker,et al.  Special Issue: Information Systems Success , 2003, J. Manag. Inf. Syst..

[55]  David G. Delaney,et al.  Marine invasive species: validation of citizen science and implications for national monitoring networks , 2007, Biological Invasions.

[56]  A. Irwin.  Citizen Science: A Study of People, Expertise and Sustainable Development , 1995 .

[57]  Roman Lukyanenko,et al.  Integrating Scientific Research: Theory and Design of Discovering Similar Constructs , 2017 .

[58]  K. D. Joshi,et al.  The Duality of Empowerment and Marginalization in Microtask Crowdsourcing: Giving Voice to the Less Powerful Through Value Sensitive Design , 2016, MIS Q..

[59]  Daniel Schlagwein,et al.  Crowdsourcing for a better world: On the relation between IT affordances and donor motivations in charitable crowdfunding , 2016, Inf. Technol. People.

[60]  Yurong He Andrea Wiggins.  Implementing an Environmental Citizen Science Project: Strategies and Concerns from Educators’ Perspectives , 2017 .

[61]  Weiguo Fan,et al.  Project description and crowdfunding success: an exploratory study , 2018, Inf. Syst. Frontiers.

[62]  Cécile Paris,et al.  Automatic Moderation of Online Discussion Sites , 2011, Int. J. Electron. Commer..

[63]  Deborah L. Illman,et al.  Dimensions of Civic Science , 2001 .

[64]  Chrysanthos Dellarocas,et al.  Harnessing Crowds: Mapping the Genome of Collective Intelligence , 2009 .

[65]  Tomas J. Bird,et al.  Statistical solutions for error and bias in global citizen science datasets , 2014 .

[66]  Gerald C. Kane,et al.  What's Different about Social Media Networks? A Framework and Research Agenda , 2014, MIS Q..

[67]  Daniele Miorandi,et al.  A security-and quality-aware system architecture for Internet of Things , 2014, Information Systems Frontiers.

[68]  Erik Brynjolfsson,et al.  The second machine age: work, progress, and prosperity in a time of brilliant technologies, 1st Edition , 2014 .

[69]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[70]  M. Haklay.  Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation , 2013 .

[71]  Ailene K. Ettinger,et al.  The science of citizen science: Exploring barriers to use as a primary research tool , 2017 .

[72]  Mary Tate,et al.  Motivations for 21st century school children to bring their own device to school , 2017, Inf. Syst. Frontiers.

[73]  J. A. Glennona.  Crowdsourcing geographic information for disaster response: a research , 2010 .

[74]  Sudha Ram,et al.  Who does what: Collaboration patterns in the wikipedia and their impact on article quality , 2011, TMIS.

[75]  Boualem Benatallah,et al.  Quality Control in Crowdsourcing , 2018, ACM Comput. Surv..

[76]  Joseph S. Valacich,et al.  What Signals Are You Sending? How Website Quality Influences Perceptions of Product Quality and Purchase Intentions , 2011, MIS Q..

[77]  Patrick Barwise,et al.  The One Thing You Must Get Right When Building a Brand , 2010 .

[78]  Vishanth Weerakkody,et al.  Investigating success of an e-government initiative: Validation of an integrated IS success model , 2014, Information Systems Frontiers.

[79]  Na Li,et al.  The Intellectual Development of Human-Computer Interaction Research: A Critical Assessment of the MIS Literature (1990-2002) , 2005, J. Assoc. Inf. Syst..

[80]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Independent Variables , 2013, J. Manag. Inf. Syst..

[81]  David De Roure,et al.  Zooniverse: observing the world's largest citizen science platform , 2014, WWW.

[82]  Miguel Palacios,et al.  Crowdsourcing and organizational forms: Emerging trends and research implications , 2016 .

[83]  Jing Wang,et al.  Bonus, Disclosure, and Choice: What Motivates the Creation of High-Quality Paid Reviews? , 2012, ICIS.

[84]  Katherine Rowland.  Citizen science goes 'extreme' , 2012, Nature.

[85]  Kevin Crowston,et al.  Citizen science system assemblages: understanding the technologies that support crowdsourced science , 2012, iConference '12.

[86]  Carlo Batini,et al.  Data Quality: Concepts, Methodologies and Techniques , 2006, Data-Centric Systems and Applications.

[87]  Edward J. Garrity,et al.  Synthesizing User Centered and Designer Centered IS Development Approaches Using General Systems Theory , 2001, Inf. Syst. Frontiers.

[88]  K. Peterman,et al.  Embedded Assessment as an Essential Method for Understanding Public Engagement in Citizen Science , 2016 .

[89]  Mario Piattini,et al.  An Applicable Data Quality Model for Web Portal Data Consumers , 2008, World Wide Web.

[90]  J. Cohn.  Citizen Science: Can Volunteers Do Real Research? , 2008 .

[91]  Yogesh Kumar Dwivedi,et al.  Advances in Social Media Research: Past, Present and Future , 2017, Information Systems Frontiers.

[92]  N. Oliver,et al.  People power , 2014, Nature.

[93]  Eva J. Lewandowski,et al.  Influence of volunteer and project characteristics on data quality of biological surveys , 2015, Conservation biology : the journal of the Society for Conservation Biology.

[94]  Jian Tang,et al.  Categorizing consumer behavioral responses and artifact design features: The case of online advertising , 2014, Information Systems Frontiers.

[95]  Rise of the citizen scientist , 2015, Nature.

[96]  D. Hailey,et al.  Systematic review of evidence for the benefits of telemedicine , 2002, Journal of telemedicine and telecare.

[97]  Harris Wu,et al.  Quality of data standards: framework and illustration using XBRL taxonomy and instances , 2011, Electron. Mark..

[98]  Marijn Janssen,et al.  A Process Pattern Model for Tackling and Improving Big Data Quality , 2018, Inf. Syst. Frontiers.

[99]  Susanne Bødker,et al.  Creating Conditions for Participation: Conflicts and Resources in Systems Development , 1996, Hum. Comput. Interact..

[100]  Ailene K. Ettinger,et al.  Global change and local solutions: Tapping the unrealized potential of citizen science for biodiversity research , 2015 .

[101]  Patrick Weber,et al.  OpenStreetMap: User-Generated Street Maps , 2008, IEEE Pervasive Computing.

[102]  Thomas J. Stohlgren,et al.  Assessing citizen science data quality: an invasive species case study , 2011 .

[103]  R. Bonney,et al.  Next Steps for Citizen Science , 2014, Science.

[104]  Ahmed Imran,et al.  A Design Framework for Technology-Mediated Public Participatory System for the Environment , 2015, PACIS.

[105]  Izak Benbasat,et al.  An Exploratory Study of the Formation and Impact of Electronic Service Failures , 2016, MIS Q..

[106]  Anna Sidorova,et al.  Uncovering the Intellectual Core of the Information Systems Discipline , 2008, MIS Q..

[107]  Martin Connors,et al.  New science in plain sight: Citizen scientists lead to the discovery of optical structure in the upper atmosphere , 2018, Science Advances.

[108]  Roman Lukyanenko,et al.  Beyond Micro-Tasks: Research Opportunities in Observational Crowdsourcing , 2018, J. Database Manag..

[109]  Jennifer Jie Xu,et al.  Business Intelligence in Blogs: Understanding Consumer Interactions and Communities , 2012, MIS Q..

[110]  R. Sieber.  Public Participation Geographic Information Systems: A Literature Review and Framework , 2006 .

[111]  Sudha Ram,et al.  Who does what: Collaboration patterns in the wikipedia and their impact on data quality , 2009, International Conference on Wireless Information Technology and Systems.

[112]  E. Richard Hoebeke,et al.  Citizen scientist rediscovers rare nine-spotted lady beetle, Coccinella novemnotata, in eastern North America , 2007, Journal of Insect Conservation.

[113]  D. Meyer,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S6 References Evidence for a Collective Intelligence Factor in the Performance of Human Groups , 2022 .

[114]  Rudy Hirschheim,et al.  Crowdsourcing of information systems research , 2017, Eur. J. Inf. Syst..

[115]  Vladimir Zwass,et al.  Co-Creation: Toward a Taxonomy and an Integrated Research Perspective , 2010, Int. J. Electron. Commer..

[116]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[117]  BenatallahBoualem,et al.  Quality Control in Crowdsourcing , 2018 .

[118]  Richard Y. Wang,et al.  Toward quality data: An attribute-based approach , 2014, Decis. Support Syst..

[119]  Kalina Bontcheva,et al.  Crowdsourcing research opportunities: lessons from natural language processing , 2012, i-KNOW '12.

[120]  C. Aulbert,et al.  THE EINSTEIN@HOME GAMMA-RAY PULSAR SURVEY. I. SEARCH METHODS, SENSITIVITY, AND DISCOVERY OF NEW YOUNG GAMMA-RAY PULSARS , 2016, 1611.01015.

[121]  Jessica L. Cappadonna,et al.  Citizen Science Terminology Matters: Exploring Key Terms , 2017, Citizen Science: Theory and Practice.

[122]  John M. Jordan,et al.  Challenges to large-scale digital organization: the case of Uber , 2017, Journal of Organization Design.

[123]  Yolanda F. Wiersma,et al.  Aedes japonicus japonicus (Diptera: Culicidae) arrives at the most easterly point in North America , 2015, The Canadian Entomologist.

[124]  S. Lynn,et al.  Planet Hunters IX. KIC 8462852-where's the flux? , 2015, 1509.03622.

[125]  Detlef Schoder,et al.  Towards a Conceptualization of Data and Information Quality in Social Information Systems , 2017, Bus. Inf. Syst. Eng..

[126]  Elisa Bertino,et al.  Quality Control in Crowdsourcing Systems: Issues and Directions , 2013, IEEE Internet Computing.

[127]  Christopher Kullenberg,et al.  What Is Citizen Science? – A Scientometric Meta-Analysis , 2016, PloS one.

[128]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[129]  John Mylopoulos,et al.  Information Modeling in the Time of the Revolution , 1998, Inf. Syst..

[130]  E. Bonabeau.  Decisions 2.0: the power of collective intelligence , 2009 .

[131]  Oded Nov,et al.  Turbulent Stability of Emergent Roles: The Dualistic Nature of Self-Organizing Knowledge Coproduction , 2016, Inf. Syst. Res..

[132]  Stuart E. Madnick,et al.  Editorial for the Inaugural Issue of the ACM Journal of Data and Information Quality (JDIQ) , 2009, JDIQ.

[133]  Trisha Gura,et al.  Citizen science: Amateur experts , 2013, Nature.

[134]  Gregory M. P. O'Hare,et al.  Citizen OBservatory WEB (COBWEB): A Generic Infrastructure Platform to Facilitate the Collection of Citizen Science data for Environmental Monitoring , 2016, Int. J. Spatial Data Infrastructures Res..

[135]  C. Potter,et al.  Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions , 2014, Public understanding of science.

[136]  Lee Belbin,et al.  The Atlas of Living Australia‟s Spatial Portal , 2011 .

[137]  Andrew Burton-Jones,et al.  How Can We Develop Contextualized Theories of Effective Use? A Demonstration in the Context of Community-Care Electronic Health Records , 2017, Inf. Syst. Res..

[138]  Thit Juul Madsen,et al.  Design for Sharing , 2017 .

[139]  Kristine F. Stepenuck,et al.  Citizen science can improve conservation science, natural resource management, and environmental protection , 2017 .

[140]  R. Bonney,et al.  Scientific knowledge and attitude change: The impact of a citizen science project , 2005 .

[141]  Yong Tan,et al.  Social Networks and the Diffusion of User-Generated Content: Evidence from YouTube , 2012, Inf. Syst. Res..

[142]  Monica Chiarini Tremblay,et al.  Using Data Mining Techniques to Discover Bias Patterns in Missing Data , 2010, JDIQ.

[143]  Arturo Castellanos,et al.  Conceptual modeling research in information systems: What we now know and what we still do not know , 2017 .

[144]  T. Daugherty,et al.  Exploring Consumer Motivations for Creating User-Generated Content , 2008 .

[145]  Loren G. Terveen,et al.  Capturing quality: retaining provenance for curated volunteer monitoring data , 2014, CSCW.

[146]  Kevin Crowston,et al.  The future of citizen science: emerging technologies and shifting paradigms , 2012, Frontiers in Ecology and the Environment.

[147]  Emre Yetgin,et al.  Are Social Media Emancipatory or Hegemonic? Societal Effects of Mass Media Digitization in the Case of the SOPA Discourse , 2016, MIS Q..

[148]  Jennifer Preece,et al.  Accounting for Privacy in Citizen Science: Ethical Research in a Context of Openness , 2017, CSCW.

[149]  Pompeu Casanovas,et al.  Crowdsourcing roles, methods and tools for data-intensive disaster management , 2018, Inf. Syst. Frontiers.

[150]  Albert Bifet,et al.  Sentiment Knowledge Discovery in Twitter Streaming Data , 2010, Discovery Science.

[151]  Ian Parfitt,et al.  Citizen science in conservation biology : best practices in the geoweb era , 2013 .

[152]  Roman Lukyanenko,et al.  Emerging problems of data quality in citizen science , 2016, Conservation biology : the journal of the Society for Conservation Biology.

[153]  Carson C. Woo,et al.  The Role of Conceptual Modeling in Managing and Changing the Business , 2011, ER.

[154]  Arco J. van Strien,et al.  Opportunistic citizen science data of animal species produce reliable estimates of distribution trends if analysed with occupancy models , 2013 .

[155]  R. P. Eatough,et al.  Pulsar–black hole binaries: prospects for new gravity tests with future radio telescopes , 2014, 1409.3882.

[156]  Michael A. Nielsen,et al.  Reinventing Discovery: The New Era of Networked Science , 2011 .

[157]  Roman Lukyanenko,et al.  Information Quality Research Challenge: Adapting Information Quality Principles to User-Generated Content , 2015, JDIQ.

[158]  Katarina Elevant.  Why Share Weather? Motivational Model for "Share Weather" Online Communities and Three Empirical Studies , 2013, 2013 46th Hawaii International Conference on System Sciences.

[159]  Richard Y. Wang,et al.  A product perspective on total data quality management , 1998, CACM.

[160]  Coniferous softwood.  GENERAL TERMS , 2003 .

[161]  O. Ferrer-Roca,et al.  Aviation medicine: challenges for telemedicine , 2002, Journal of telemedicine and telecare.

[162]  Menno Schilthuizen,et al.  Three new minute leaf litter beetles discovered by citizen scientists in Maliau Basin, Malaysian Borneo (Coleoptera: Leiodidae, Chrysomelidae) , 2017, Biodiversity data journal.

[163]  Stephen J. Roberts,et al.  Dynamic Bayesian Combination of Multiple Imperfect Classifiers , 2012, Decision Making and Imperfection.

[164]  Angelika Dimoka,et al.  The Nature and Role of Feedback Text Comments in Online Marketplaces: Implications for Trust Building, Price Premiums, and Seller Differentiation , 2006, Inf. Syst. Res..

[165]  Ann Blandford,et al.  Designing for dabblers and deterring drop-outs in citizen science , 2014, CHI.

[166]  Jonathan Silvertown,et al.  Taxonomy: include social networking , 2010, Nature.

[167]  Miriam J. Metzger,et al.  The credibility of volunteered geographic information , 2008 .

[168]  Guillaume Touya,et al.  Quality Assessment of the French OpenStreetMap Dataset , 2010, Trans. GIS.

[169]  Oded Nov,et al.  Information Quality in Wikipedia: The Effects of Group Composition and Task Conflict , 2011, J. Manag. Inf. Syst..

[170]  Mary L. Gray,et al.  The Crowd is a Collaborative Network , 2016, CSCW.

[171]  T. C. Nicholas Graham,et al.  Beyond designing for motivation: the importance of context in gamification , 2014, CHI PLAY.

[172]  E. Howard,et al.  Spring recolonization rate of monarch butterflies in eastern North America: New estimates from citizen-science data , 2005 .

[173]  Jordan Raddick,et al.  Galaxy Zoo: Morphological Classification and Citizen Science , 2011, 1104.5513.

[174]  Margaret Kosmala,et al.  Assessing data quality in citizen science (preprint) , 2016, bioRxiv.

[175]  Roger Blake,et al.  Data and Information Quality: Research Themes and Evolving Patterns , 2015, AMCIS.

[176]  Steve Kelling,et al.  Data-intensive science applied to broad-scale citizen science. , 2012, Trends in ecology & evolution.

[177]  R. Bonney,et al.  Citizen Science: Public Participation in Environmental Research , 2012 .

[178]  Margaret Tan,et al.  Building an online collaborative platform to advance creativity , 2010, 4th IEEE International Conference on Digital Ecosystems and Technologies.

[179]  Sihem Amer-Yahia,et al.  A Survey of General-Purpose Crowdsourcing Techniques , 2016, IEEE Transactions on Knowledge and Data Engineering.

[180]  Manuel Arriaga,et al.  Distinction and Status Production on User-Generated Content Platforms: Using Bourdieu's Theory of Cultural Production to Understand Social Dynamics in Online Fields , 2014, Inf. Syst. Res..

[181]  Pengzhu Zhang,et al.  Text Analytics to Support Sense-Making in Social Media: A Language-Action Perspective , 2018, MIS Q..

[182]  Tone Bratteteig,et al.  Disentangling Participation: Power and Decision-making in Participatory Design , 2014 .

[183]  Isabelle Durance,et al.  Recommendations for the next generation of global freshwater biological monitoring tools , 2016 .

[184]  Richard Y. Wang,et al.  Data quality assessment , 2002, CACM.

[185]  Melinda Laituri,et al.  User-friendly web mapping: lessons from a citizen science website , 2010, Int. J. Geogr. Inf. Sci..

[186]  Kai R. Larsen,et al.  A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses , 2016, MIS Q..

[187]  Jerome Lewis,et al.  Making local knowledge matter: supporting non-literate people to monitor poaching in Congo , 2013, ACM DEV '13.

[188]  Paul Ralph,et al.  Proposing a theory of gamification effectiveness , 2014, ICSE Companion.

[189]  Adam Liwo,et al.  WeFold: A coopetition for protein structure prediction , 2014, Proteins.

[190]  Brice X Semmens,et al.  Conservation and management applications of the REEF volunteer fish monitoring program. , 2003, Environmental monitoring and assessment.

[191]  Roman Lukyanenko,et al.  Expecting the Unexpected: Effects of Data Collection Design Choices on the Quality of Crowdsourced User-Generated Content , 2019, MIS Q..

[192]  Yang W. Lee,et al.  Crafting Rules: Context-Reflective Data Quality Problem Solving , 2003, J. Manag. Inf. Syst..

[193]  Tanya Heikkila,et al.  Citizen Involvement and Performance Management in Special-Purpose Governments , 2007 .

[194]  Larissa L Bailey,et al.  Experimental investigation of false positive errors in auditory species occurrence surveys. , 2012, Ecological applications : a publication of the Ecological Society of America.

[195]  Andrew Gemino,et al.  A framework for empirical evaluation of conceptual modeling techniques , 2004, Requirements Engineering.

[196]  Jennifer Preece,et al.  Dynamic changes in motivation in collaborative citizen-science projects , 2012, CSCW.

[197]  Roman Lukyanenko,et al.  The Impact of Conceptual Modeling on Dataset Completeness: A Field Experiment , 2014, ICIS.

[198]  Jung P. Shim,et al.  Current Status, Issues, and Future of Bring Your Own Device (BYOD) , 2014, Commun. Assoc. Inf. Syst..

[199]  Hajo Hippner,et al.  Crowdsourcing , 2012, Business & Information Systems Engineering.

[200]  Paulo B. Góes,et al.  Editor's comments: design science research in top information systems journals , 2014 .

[201]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[202]  Jerome Lewis,et al.  Taking Participatory Citizen Science to Extremes , 2014, IEEE Pervasive Computing.

[203]  Jennifer Preece,et al.  Gamifying citizen science: a study of two user groups , 2014, CSCW Companion.

[204]  R. Bonney,et al.  Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy , 2009 .

[205]  Gerald C. Kane,et al.  Research Note - Content and Collaboration: An Affiliation Network Approach to Information Quality in Online Peer Production Communities , 2016, Inf. Syst. Res..

[206]  Keng Siau,et al.  Evaluation techniques for systems analysis and design modelling methods – a review and comparative analysis , 2011, Inf. Syst. J..

[207]  Sophie Lumineau,et al.  Fearfulness Affects Quail Maternal Care and Subsequent Offspring Development , 2014, PloS one.

[208]  M. Goodchild.  Citizens as sensors: the world of volunteered geography , 2007 .

[209]  Ron Weber,et al.  On the deep structure of information systems , 1995, Inf. Syst. J..

[210]  Loren G. Terveen,et al.  Quality is a verb: the operationalization of data quality in a citizen science community , 2011, Int. Sym. Wikis.

[211]  Paolo Missier,et al.  Data Quality at a Glance 6 Datenbank-spektrum 14/2005 , 2005 .

[212]  A. Janssens,et al.  Research Conducted Using Data Obtained through Online Communities: Ethical Implications of Methodological Limitations , 2012, PLoS medicine.

[213]  Anany Levitin,et al.  Quality dimensions of a conceptual view , 1995 .

[214]  Kevin Crowston,et al.  Purposeful gaming & socio-computational systems: a citizen science design case , 2012, GROUP.

[215]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[216]  Ron Kohavi,et al.  Seven pitfalls to avoid when running controlled experiments on the web , 2009, KDD.

[217]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[218]  Fred D. Davis,et al.  Dead Or Alive? The Development, Trajectory And Future Of Technology Adoption Research , 2007, J. Assoc. Inf. Syst..

[219]  Heiko Gewald,et al.  Determinants of Intention to Participate in Corporate BYOD-Programs: The Case of Digital Natives , 2020, Inf. Syst. Frontiers.

[220]  J. Clobert,et al.  Advantages of Volunteer‐Based Biodiversity Monitoring in Europe , 2009, Conservation biology : the journal of the Society for Conservation Biology.

[221]  James M. Lawrence.  Rediscovery of the threatened Stoffberg Widow butterfly, Dingana fraterna: the value of citizen scientists for African conservation , 2015, Journal of Insect Conservation.

[222]  Oded Nov,et al.  Dusting for science: motivation and participation of digital citizen science volunteers , 2011, iConference.

[223]  Shoshana Zuboff,et al.  In the Age of the Smart Machine: The Future of Work and Power , 1989 .

[224]  Sven Laumer,et al.  Research on information systems failures and successes: Status update and future directions , 2014, Information Systems Frontiers.

[225]  David P. Anderson,et al.  Scientists@Home: What Drives the Quantity and Quality of Online Citizen Science Participation? , 2014, PloS one.

[226]  Zhang De-xin.  Big Data Research , 2013 .

[227]  Omar Alonso,et al.  Challenges with Label Quality for Supervised Learning , 2015, JDIQ.

[228]  Kevin Crowston,et al.  Mechanisms for Data Quality and Validation in Citizen Science , 2011, 2011 IEEE Seventh International Conference on e-Science Workshops.

[229]  Cynthia Mathis Beath,et al.  Expanding the Frontiers of Information Systems Research: Introduction to the Special Issue , 2013, J. Assoc. Inf. Syst..

[230]  Rudy Hirschheim,et al.  Choosing Between Competing Design Ideals in Information Systems Development , 2001, Inf. Syst. Frontiers.

[231]  Paul Voosen.  Update: NASA confirms amateur astronomer has discovered a lost satellite , 2018 .

[232]  Eric J. Korpela,et al.  SETI@home, BOINC, and Volunteer Distributed Computing , 2012 .

[233]  Ilyong Park.  The Deep Structure of , 2020 .

[234]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[235]  Diane M. Strong,et al.  AIMQ: a methodology for information quality assessment , 2002, Inf. Manag..

[236]  Mike Michael,et al.  Science, Social Theory and Public Knowledge , 2003 .

[237]  Thomas Hilton,et al.  IS accreditation in AACSB colleges via ABET , 2007, J. Assoc. Inf. Syst..

[238]  Nigel Melville,et al.  Information Systems Innovation for Environmental Sustainability , 2010, MIS Q..

[239]  Roman Lukyanenko,et al.  Easier citizen science is better , 2011, Nature.

[240]  Morten Kyng,et al.  Making representations work , 1995, CACM.

[241]  Ilan Oshri,et al.  Re-presentation as Work Design in Outsourcing: A Semiotic View , 2018, MIS Q..

[242]  A. Ghezzi,et al.  Crowdsourcing: A Review and Suggestions for Future Research , 2018 .

[243]  Roman Lukyanenko,et al.  Participatory Design for User-generated Content: Understanding the challenges and moving forward , 2016, Scand. J. Inf. Syst..

[244]  James Howison,et al.  Beyond the organizational 'container': Conceptualizing 21st century sociotechnical work , 2014, Inf. Organ..

[245]  Roger H. L. Chiang,et al.  Big Data Research in Information Systems: Toward an Inclusive Research Agenda , 2016, J. Assoc. Inf. Syst..

[246]  Ofer Arazy,et al.  Heuristic Principles and Differential Judgments in the Assessment of Information Quality , 2017, J. Assoc. Inf. Syst..

[247]  Stuart E. Madnick,et al.  The Design and Implementation of a Corporate Householding Knowledge Processor to Improve Data Quality , 2003, J. Manag. Inf. Syst..