It's My Data! Tensions Among Stakeholders of a Learning Analytics Dashboard

Early warning dashboards in higher education analyze student data to enable early identification of underperforming students, allowing timely interventions by faculty and staff. To understand perceptions regarding the ethics and impact of such learning analytics applications, we conducted a multi-stakeholder analysis of an early-warning dashboard deployed at the University of Michigan through semi-structured interviews with the system's developers, academic advisors (the primary users), and students. We identify multiple tensions among and within the stakeholder groups, especially with regard to awareness, understanding, access and use of the system. Furthermore, ambiguity in data provenance and data quality result in differing levels of reliance and concerns about the system among academic advisors and students. While students see the system's benefits, they argue for more involvement, control, and informed consent regarding the use of student data. We discuss our findings' implications for the ethical design and deployment of learning analytics applications in higher education. Early warning dashboards in higher education analyze student data to enable early identification of underperforming students, allowing timely interventions by faculty and staff. To understand perceptions regarding the ethics and impact of such learning analytics applications, we conducted a multi-stakeholder analysis of an early-warning dashboard deployed at the University of Michigan through semi-structured interviews with the system's developers, academic advisors (the primary users), and students. We identify multiple tensions among and within the stakeholder groups, especially with regard to awareness, understanding, access, and use of the system. Furthermore, ambiguity in data provenance and data quality result in differing levels of reliance and concerns about the system among academic advisors and students. While students see the system's benefits, they argue for more involvement, control, and informed consent regarding the use of student data. We discuss our findings' implications for the ethical design and deployment of learning analytics applications in higher education.

[1]  Dirk Ifenthaler,et al.  Student perceptions of privacy principles for learning analytics , 2016, Educational Technology Research and Development.

[2]  Steven Lonn,et al.  Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems , 2009, Comput. Educ..

[3]  Jodi Aronson.  A Pragmatic View of Thematic Analysis , 1995 .

[4]  Helen Nissenbaum,et al.  Bias in computer systems , 1996, TOIS.

[5]  Hendrik Drachsler,et al.  The pulse of learning analytics understandings and expectations from the stakeholders , 2012, LAK.

[6]  G. Dishon.  New data, old tensions: Big data, personalized learning, and the challenges of progressive education , 2017 .

[7]  John Fritz,et al.  Classroom walls that talk: Using online course activity data of successful students to raise self-awareness of underperforming peers , 2011, Internet High. Educ..

[8]  Paul Prinsloo,et al.  Student perspectives on the use of their data: between intrusion, surveillance and care , 2015 .

[9]  P. Prinsloo,et al.  Learning Analytics , 2013 .

[10]  Matthew D. Pistilli,et al.  Course signals at Purdue: using learning analytics to increase student success , 2012, LAK.

[11]  Hélène Fournier,et al.  The value of learning analytics to networked learning on a personal learning environment , 2011, LAK.

[12]  Ke Zhang,et al.  Revealing Online Learning Behaviors and Activity Patterns and Making Predictions with Data Mining Techniques in Online Teaching , 2008 .

[13]  Yeonjeong Park,et al.  Development of the Learning Analytics Dashboard to Support Students' Learning Performance , 2015, J. Univers. Comput. Sci..

[14]  Anthony G. Picciano.  The Evolution of Big Data and Learning Analytics in American Higher Education , 2012 .

[15]  Susanne Bauer,et al.  Cutting Code. Software and Sociality , 2007 .

[16]  Mireille Hildebrandt,et al.  The Meaning and the Mining of Legal Texts , 2012 .

[17]  Hendrik Drachsler,et al.  Translating Learning into Numbers: A Generic Framework for Learning Analytics , 2012, J. Educ. Technol. Soc..

[18]  Paul Prinsloo,et al.  Student Data Privacy and Institutional Accountability in an Age of Surveillance , 2014 .

[19]  Katrien Verbert,et al.  Trends and issues in student-facing learning analytics reporting systems research , 2017, LAK.

[20]  John P. Campbell,et al.  Academic Analytics: A New Tool for a New Era. , 2007 .

[21]  E. Duval.  Attention please!: learning analytics for visualization and recommendation , 2011, LAK.

[22]  Gina Neff,et al.  Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science , 2017, Big Data.

[23]  Karen Holtzblatt,et al.  Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design , 2004, UBIQ.

[24]  Rebecca Ferguson,et al.  Setting learning analytics in context: overcoming the barriers to large-scale adoption , 2014, LAK '14.

[25]  David Beer,et al.  The social power of algorithms , 2017, The Social Power of Algorithms.

[26]  Laurie P. Dringus,et al.  Learning Analytics Considered Harmful. , 2012 .

[27]  Beverley G. Hope,et al.  Perceived Service Quality in a University Web Portal: Revising the E-Qual Instrument , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[28]  Paul Prinsloo,et al.  Student privacy self-management: implications for learning analytics , 2015, LAK.

[29]  Erik Duval,et al.  Recommender Systems for Technology Enhanced Learning ( RecSysTEL 2010 ) Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning , 2010 .

[30]  Ryan S. Baker,et al.  Educational Data Mining and Learning Analytics , 2014 .

[31]  Paul Prinsloo,et al.  An evaluation of policy frameworks for addressing ethical considerations in learning analytics , 2013, LAK '13.

[32]  Randolph G. Bias,et al.  Research Methods for Human-Computer Interaction , 2010, J. Assoc. Inf. Sci. Technol..

[33]  Florian Schaub,et al.  Achieving big data privacy in education , 2018, Theory and Research in Education.

[34]  Eli Pariser.  The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think , 2012 .

[35]  George Siemens,et al.  Learning Analytics , 2013 .

[36]  Matthew Clayton,et al.  Big data and the liberal conception of education , 2017 .

[37]  Dennis Zielke,et al.  Design and Implementation of a Learning Analytics Toolkit for Teachers , 2012, J. Educ. Technol. Soc..

[38]  Paul Gibbs,et al.  Using data to improve higher education , 2014 .

[39]  Steven Lonn,et al.  Investigating student motivation in the context of a learning analytics intervention during a summer bridge program , 2015, Comput. Hum. Behav..

[40]  Royce Kimmons,et al.  Instructor experiences with a social networking site in a higher education setting: expectations, frustrations, appropriation, and compartmentalization , 2013 .

[41]  Felix C. Freiling,et al.  Early warning system. , 2010, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[42]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[43]  Erik Duval,et al.  Dataset-driven research for improving recommender systems for learning , 2011, LAK.

[44]  Paul Dourish,et al.  The Appropriation of Interactive Technologies: Some Lessons from Placeless Documents , 2003, Computer Supported Cooperative Work (CSCW).

[45]  Doug Clow.  An overview of learning analytics , 2013 .

[46]  George Siemens,et al.  Current state and future trends: a citation network analysis of the learning analytics field , 2014, LAK.

[47]  Steven Lonn,et al.  Appropriation by unanticipated users: looking beyond design intent and expected use , 2013, CSCW.

[48]  Steven G. Jones,et al.  Ethical Decision-Making and Internet Research: Recommendations from the AoIR Ethics Working Committee , 2004 .

[49]  Carlos Delgado Kloos,et al.  ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform , 2015, Comput. Hum. Behav..

[50]  Steven Lonn,et al.  A Learning Management System-Based Early Warning System for Academic Advising in Undergraduate Engineering , 2014 .

[51]  Gina Schouten,et al.  On meeting students where they are: Teacher judgment and the use of data in higher education , 2017 .

[52]  Eli Pariser,et al.  The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think , 2012 .

[53]  Maren Scheffel,et al.  Quality Indicators for Learning Analytics , 2014, J. Educ. Technol. Soc..

[54]  Lee Rainie,et al.  Code-dependent: Pros and cons of the algorithm age , 2017 .

[55]  Hendrik Drachsler,et al.  Privacy and analytics: it's a DELICATE issue a checklist for trusted learning analytics , 2016, LAK.

[56]  Steven Lonn,et al.  Student explorer: a tool for supporting academic advising at scale , 2014, L@S.

[57]  Alyssa Friend Wise,et al.  Designing pedagogical interventions to support student use of learning analytics , 2014, LAK.

[58]  George Siemens,et al.  Ethical and privacy principles for learning analytics , 2014, Br. J. Educ. Technol..

[59]  Doug Clow,et al.  The learning analytics cycle: closing the loop effectively , 2012, LAK.