Brain and autonomic nervous system activity measurement in software engineering: A systematic literature review

Abstract In the past decade, brain and autonomic nervous system activity measurement received increasing attention in the study of software engineering (SE). This paper presents a systematic literature review (SLR) to survey the existing NeuroSE literature. Based on a rigorous search protocol, we identified 89 papers (hereafter denoted as NeuroSE papers). We analyzed these papers to develop a comprehensive understanding of who had published NeuroSE research and classified the contributions according to their type. The 47 articles presenting completed empirical research were analyzed in detail. The SLR revealed that the number of authors publishing NeuroSE research is still relatively small. The thematic focus so far has been on code comprehension, while code inspection, programming, and bug fixing have been less frequently studied. NeuroSE publications primarily used methods related to brain activity measurement (particularly fMRI and EEG), while methods related to the measurement of autonomic nervous system activity (e.g., pupil dilation, heart rate, skin conductance) received less attention. We also present details of how the empirical research was conducted, including stimuli and independent and dependent variables, and discuss implications for future research. The body of NeuroSE literature is still small. Yet, high quality contributions exist constituting a valuable basis for future studies.

[1]  Timothy E. J. Behrens,et al.  Tools of the trade: psychophysiological interactions and functional connectivity. , 2012, Social cognitive and affective neuroscience.

[2]  Daniel M. Germán,et al.  Quantifying programmers' mental workload during program comprehension based on cerebral blood flow measurement: a controlled experiment , 2014, ICSE Companion.

[3]  M. Crosby,et al.  Code Scanning Patterns in Program Comprehension , 2005 .

[4]  Karl J. Friston Functional and effective connectivity in neuroimaging: A synthesis , 1994 .

[5]  Tao Lin,et al.  Evaluating usability based on multimodal information: an empirical study , 2006, ICMI '06.

[6]  René Riedl,et al.  Adding background music as new stimuli of interest to information systems research , 2017, Eur. J. Inf. Syst..

[7]  Angelika Dimoka,et al.  On the Use of Neuropyhsiological Tools in IS Research: Developing a Research Agenda for NeuroIS , 2012, MIS Q..

[8]  Olusola Adesope,et al.  Measuring the impact of lexical and structural inconsistencies on developers’ cognitive load during bug localization , 2019, Empirical Software Engineering.

[9]  T. Lehtimäki,et al.  The Combined Effect of Common Genetic Risk Variants on Circulating Lipoproteins Is Evident in Childhood: A Longitudinal Analysis of the Cardiovascular Risk in Young Finns Study , 2016, PloS one.

[10]  Kleinner Farias,et al.  Using biometric data in software engineering: a systematic mapping study , 2020, Behav. Inf. Technol..

[11]  Anna Sidorova,et al.  Uncovering the Intellectual Core of the Information Systems Discipline , 2008, MIS Q..

[12]  Karen Blackmore,et al.  Using the Startle Eye-Blink to Measure Affect in Players , 2015 .

[13]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[14]  P. Venables,et al.  Publication recommendations for electrodermal measurements. , 1981 .

[15]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[16]  Andrew Begel Invited Talk: Fun with Software Developers and Biometrics , 2016, 2016 IEEE/ACM 1st International Workshop on Emotional Awareness in Software Engineering (SEmotion).

[17]  John Sweller,et al.  Cognitive Load Theory , 2020, Encyclopedia of Education and Information Technologies.

[18]  Björn Niehaves,et al.  Reconstructing the giant: On the importance of rigour in documenting the literature search process , 2009, ECIS.

[19]  Yann-Gaël Guéhéneuc,et al.  A systematic literature review on the usage of eye-tracking in software engineering , 2015, Inf. Softw. Technol..

[20]  S. Bunce,et al.  Functional near-infrared spectroscopy , 2006, IEEE Engineering in Medicine and Biology Magazine.

[21]  Peter C.-H. Cheng,et al.  A Survey on the Usage of Eye-Tracking in Computer Programming , 2018, ACM Comput. Surv..

[22]  H. van Steenbergen,et al.  Pupil dilation as an index of effort in cognitive control tasks: A review , 2018, Psychonomic Bulletin & Review.

[23]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[24]  Aidan Mooney,et al.  Examining the role of cognitive load when learning toprogram , 2015 .

[25]  Shihong Huang,et al.  Brainware: synergizing software systems and neural inputs , 2014, ICSE Companion.

[26]  Tapio Taipalus,et al.  Comparison of photoplethysmogram measured from wrist and finger and the effect of measurement location on pulse arrival time , 2018, Physiological measurement.

[27]  Danial Hooshyar,et al.  Mining biometric data to predict programmer expertise and task difficulty , 2017, Cluster Computing.

[28]  Edouard Machery What Is a Replication? , 2020, Philosophy of Science.

[29]  Marc Garbey,et al.  Measuring Mental Workload with EEG+fNIRS , 2017, Front. Hum. Neurosci..

[30]  Herman Tarasau,et al.  Problems in Experiment with Biological Signals in Software Engineering: The Case of the EEG , 2019, TOOLS.

[31]  Sven Apel,et al.  Simultaneous measurement of program comprehension with fMRI and eye tracking: a case study , 2018, ESEM.

[32]  Marco Ferrari,et al.  Functional Near-Infrared Spectroscopy (fNIRS) for Assessing Cerebral Cortex Function During Human Behavior in Natural/Social Situations: A Concise Review , 2019 .

[33]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[34]  Una-May O'Reilly,et al.  Comprehension of computer code relies primarily on domain-general executive brain regions , 2020, eLife.

[35]  James Shortle,et al.  The foundations , 2018, Celebrity Society.

[36]  Venera Arnaoudova,et al.  VITALSE: Visualizing Eye Tracking and Biometric Data , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).

[37]  Igor Crk,et al.  Toward using alpha and theta brain waves to quantify programmer expertise , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[38]  S. Lloyd-Fox Functional near infrared spectroscopy (fNIRS) , 2020, The Oxford Handbook of Developmental Cognitive Neuroscience.

[39]  João Durães,et al.  Spotting Problematic Code Lines using Nonintrusive Programmers' Biofeedback , 2019, 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE).

[40]  Barbara Weber,et al.  Learning process modeling phases from modeling interactions and eye tracking data , 2019, Data Knowl. Eng..

[41]  Chris Parnin,et al.  Dazed: Measuring the Cognitive Load of Solving Technical Interview Problems at the Whiteboard , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[42]  D. Gefen,et al.  Applying Functional Near Infrared (fNIR) Spectroscopy to Enhance MIS Research , 2014 .

[43]  Thomas Fritz,et al.  Leveraging Biometric Data to Boost Software Developer Productivity , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[44]  Michael S. Gazzaniga,et al.  Methods in mind , 2006 .

[45]  Jennifer S. Beer,et al.  Methods in social neuroscience , 2009 .

[46]  Sven Apel,et al.  Studying programming in the neuroage , 2020, Commun. ACM.

[47]  Dror G. Feitelson,et al.  How programmers read regular code: a controlled experiment using eye tracking , 2015, Empirical Software Engineering.

[48]  Bonita Sharif,et al.  Emotional Awareness in Software Development: Theory and Measurement , 2017, 2017 IEEE/ACM 2nd International Workshop on Emotion Awareness in Software Engineering (SEmotion).

[49]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[50]  A. Villringer,et al.  Non-invasive optical spectroscopy and imaging of human brain function , 1997, Trends in Neurosciences.

[51]  Angel Jiménez Molina,et al.  Using Psychophysiological Sensors to Assess Mental Workload During Web Browsing , 2018, Sensors.

[52]  Hugo F Posada-Quintero,et al.  Innovations in Electrodermal Activity Data Collection and Signal Processing: A Systematic Review , 2020, Sensors.

[53]  Stefan Wagner,et al.  Towards the Assessment of Stress and Emotional Responses of a Salutogenesis-Enhanced Software Tool Using Psychophysiological Measurements , 2017, 2017 IEEE/ACM 2nd International Workshop on Emotion Awareness in Software Engineering (SEmotion).

[54]  Giancarlo Succi,et al.  Understanding the Impact of Pair Programming on the Minds of Developers , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[55]  Marina Bedny,et al.  Computer code comprehension shares neural resources with formal logical inference in the fronto-parietal network , 2020, eLife.

[56]  Anjali Phukan,et al.  Measuring Usability via Biometrics , 2009, HCI.

[57]  R. R. Lekkala,et al.  A novel approach for comparison of heart rate variability derived from synchronously measured electrocardiogram and photoplethysmogram , 2017, 2017 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT).

[58]  René Riedl,et al.  Using Psycho-physiological Interaction Analysis with fMRI Data in IS Research: A Guideline , 2017, Commun. Assoc. Inf. Syst..

[59]  Software Engineering und Software Management 2018 , 2018, Software Engineering.

[60]  Dae-Shik Kim,et al.  Pattern-Based Granger Causality Mapping in fMRI , 2013, Brain Connect..

[61]  Nicole Novielli,et al.  A Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors , 2019, 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC).

[62]  N. Logothetis What we can do and what we cannot do with fMRI , 2008, Nature.

[63]  Thomas Fritz,et al.  Interruptibility of Software Developers and its Prediction Using Psycho-Physiological Sensors , 2015, CHI.

[64]  Andrew Begel,et al.  Using psycho-physiological measures to assess task difficulty in software development , 2014, ICSE.

[65]  Kazushi Ikeda,et al.  Expert Programmers Have Fine-Tuned Cortical Representations of Source Code , 2020, eNeuro.

[66]  Hidetake Uwano,et al.  Programmer's Electroencephalogram Who Found Implementation Strategy , 2016, 2016 4th Intl Conf on Applied Computing and Information Technology/3rd Intl Conf on Computational Science/Intelligence and Applied Informatics/1st Intl Conf on Big Data, Cloud Computing, Data Science & Engineering (ACIT-CSII-BCD).

[67]  Rick Kazman,et al.  Neurophysiological Impact of Software Design Processes on Software Developers , 2017, HCI.

[68]  Thierry Dutoit,et al.  A P300-based Quantitative Comparison between the Emotiv Epoc Headset and a Medical EEG Device , 2012, BioMed 2012.

[69]  Yan Xiao,et al.  Using Eye Tracking Technology to Analyze the Impact of Stylistic Inconsistency on Code Readability , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[70]  René Riedl,et al.  Design Blueprint for Stress-Sensitive Adaptive Enterprise Systems , 2017, Bus. Inf. Syst. Eng..

[71]  Yann-Gaël Guéhéneuc,et al.  Eye-Tracking Metrics in Software Engineering , 2015, 2015 Asia-Pacific Software Engineering Conference (APSEC).

[72]  Andrew T. Duchowski,et al.  Eye Tracking Methodology - Theory and Practice, Third Edition , 2003 .

[73]  Igor Crk,et al.  Assessing the contribution of the individual alpha frequency (IAF) in an EEG-based study of program comprehension , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[74]  Dimosthenis Kontogiorgos,et al.  Towards identifying programming expertise with the use of physiological measures , 2015 .

[75]  René Riedl,et al.  A Decade of NeuroIS Research: Status Quo, Challenges, and Future Directions , 2017, ICIS.

[76]  René Riedl,et al.  Fundamentals of NeuroIS , 2016, Studies in Neuroscience, Psychology and Behavioral Economics.

[77]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[78]  Dongchuan Yu,et al.  Variations of the Functional Brain Network Efficiency in a Young Clinical Sample within the Autism Spectrum: A fNIRS Investigation , 2018, Front. Physiol..

[79]  Joseph H. Goldberg,et al.  Measuring Software Screen Complexity: Relating Eye Tracking, Emotional Valence, and Subjective Ratings , 2014, Int. J. Hum. Comput. Interact..

[80]  Jan-Mathijs Schoffelen,et al.  A Tutorial Review of Functional Connectivity Analysis Methods and Their Interpretational Pitfalls , 2016, Front. Syst. Neurosci..

[81]  Alan R. Dennis,et al.  Consumer-Grade EEG Instruments: Insights on the Measurement Quality Based on a Literature Review and Implications for NeuroIS Research , 2020 .

[82]  Giulio Jacucci,et al.  The Psychophysiology Primer: A Guide to Methods and a Broad Review with a Focus on Human-Computer Interaction , 2016, Found. Trends Hum. Comput. Interact..

[83]  Angelika Dimoka,et al.  On the Foundations of NeuroIS: Reflections on the Gmunden Retreat 2009 , 2010, Commun. Assoc. Inf. Syst..

[84]  Thomas Fritz,et al.  Sensing and Supporting Software Developers' Focus , 2018, 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC).

[85]  Thomas Fritz,et al.  Stuck and Frustrated or in Flow and Happy: Sensing Developers' Emotions and Progress , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[86]  R. Turner,et al.  Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[87]  Daniel J. McDuff,et al.  Non-contact imaging of peripheral hemodynamics during cognitive and psychological stressors , 2020, Scientific Reports.

[88]  Hans Gruber,et al.  Eye Tracking Metrics in Software Engineering , 2018, ECSEE.

[89]  Giancarlo Succi,et al.  Toward a Better Understanding of How to Develop Software Under Stress - Drafting the Lines for Future Research , 2018, ENASE.

[90]  Cengiz Acartürk,et al.  Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures , 2019, Front. Hum. Neurosci..

[91]  Jonathan I. Maletic,et al.  iTrace: eye tracking infrastructure for development environments , 2018, ETRA.

[92]  P. Vlamos,et al.  Undergraduate Students' Brain Activity in Visual and Textual Programming. , 2020, Advances in experimental medicine and biology.

[93]  Mehmet Rasit Yuce,et al.  A survey on signals and systems in ambulatory blood pressure monitoring using pulse transit time , 2015, Physiological measurement.

[94]  Nicole Novielli,et al.  Sentiment and Emotion in Software Engineering , 2019, IEEE Softw..

[95]  Ananga Thapaliya EEG: identification of concentration level under pair programming , 2019, ITTCS.

[96]  Hidetake Uwano,et al.  Synchronized Analysis of Eye Movement and EEG during Program Comprehension , 2019, 2019 IEEE/ACM 6th International Workshop on Eye Movements in Programming (EMIP).

[97]  Nicole Novielli,et al.  Towards Recognizing the Emotions of Developers Using Biometrics: The Design of a Field Study , 2019, 2019 IEEE/ACM 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion).

[98]  Tianwei Yu,et al.  K-Profiles: A Nonlinear Clustering Method for Pattern Detection in High Dimensional Data , 2015, BioMed research international.

[99]  H. Helmholtz Ueber einige Gesetze der Vertheilung elektrischer Ströme in körperlichen Leitern mit Anwendung auf die thierisch‐elektrischen Versuche , 1853 .

[100]  Jinrui Zhang,et al.  FC-NIRS: A Functional Connectivity Analysis Tool for Near-Infrared Spectroscopy Data , 2015, BioMed research international.

[101]  M. Dawson,et al.  The electrodermal system , 2007 .

[102]  Hope H. Kean,et al.  Comprehension of computer code relies primarily on domain-general executive brain regions , 2020, bioRxiv.

[103]  Ramaswamy Palaniappan,et al.  Cognitive task difficulty analysis using EEG and data mining , 2017, 2017 Conference on Emerging Devices and Smart Systems (ICEDSS).

[104]  Westley Weimer,et al.  Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[105]  Shihong Huang,et al.  Incorporating Human Intention into Self-Adaptive Systems , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[106]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[107]  René Riedl,et al.  Analysis of Heart Rate Variability (HRV) Feature Robustness for Measuring Technostress , 2018, Information Systems and Neuroscience.

[108]  Sebastian C. Müller Measuring Software Developers' Perceived Difficulty with Biometric Sensors , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[109]  Norman Peitek,et al.  A Neuro-Cognitive Perspective of Program Comprehension , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[110]  Chris Parnin,et al.  Subvocalization - Toward Hearing the Inner Thoughts of Developers , 2011, 2011 IEEE 19th International Conference on Program Comprehension.

[111]  Gernot R. Müller-Putz,et al.  Electroencephalography (EEG) as a Research Tool in the Information Systems Discipline: Foundations, Measurement, and Applications , 2015, Commun. Assoc. Inf. Syst..

[112]  Makoto Kato,et al.  Blink-related momentary activation of the default mode network while viewing videos , 2012, Proceedings of the National Academy of Sciences.

[113]  Sven Apel,et al.  Neural Efficiency of Top-Down Program Comprehension , 2018, Software Engineering.

[114]  Chris Parnin,et al.  Can we predict stressful technical interview settings through eye-tracking? , 2018, EMIP@ETRA.

[115]  Henrique Madeira,et al.  WAP: Understanding the Brain at Software Debugging , 2016, 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE).

[116]  Ricardo Colomo Palacios,et al.  Taking the emotional pulse of software engineering - A systematic literature review of empirical studies , 2019, Inf. Softw. Technol..

[117]  Danial Hooshyar,et al.  Comparing Programming Language Comprehension between Novice and Expert Programmers Using EEG Analysis , 2016, 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE).

[118]  Sven Apel,et al.  Toward conjoint analysis of simultaneous eye-tracking and fMRI data for program-comprehension studies , 2018, EMIP@ETRA.

[119]  Andrew Begel,et al.  Affect Recognition in Code Review: An In-situ Biometric Study of Reviewer's Affect , 2020, J. Syst. Softw..

[120]  João Ricardo Sato,et al.  fNIRS Optodes’ Location Decider (fOLD): a toolbox for probe arrangement guided by brain regions-of-interest , 2018, Scientific Reports.

[121]  Henrique Madeira,et al.  Pupillography as Indicator of Programmers' Mental Effort and Cognitive Overload , 2019, 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[122]  N. Logothetis,et al.  Neurophysiological investigation of the basis of the fMRI signal , 2001, Nature.

[123]  Keith E. Nolan,et al.  The role of anxiety when learning to program: a systematic review of the literature , 2016, Koli Calling.

[124]  Ryad Titah,et al.  Precision is in the Eye of the Beholder: Application of Eye Fixation-Related Potentials to Information Systems Research , 2014, J. Assoc. Inf. Syst..

[125]  Fred D. Davis,et al.  ON THE USE OF NEUROPHYSIOLOGICAL TOOLS IN IS RESEARCH : DEVELOPING A RESEARCH AGENDA FOR NEUROIS 1 , 2012 .

[126]  Ken-ichi Matsumoto,et al.  Real-Time Monitoring of Neural State in Assessing and Improving Software Developers' Productivity , 2015, 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering.

[127]  Lefteris Angelis,et al.  Towards an affordable brain computer interface for the assessment of programmers' mental workload , 2018, Int. J. Hum. Comput. Stud..

[128]  WebsterJane,et al.  Analyzing the past to prepare for the future , 2002 .

[129]  René Riedl,et al.  Blood Pressure Measurement: A Classic of Stress Measurement and Its Role in Technostress Research , 2018 .

[130]  Nicole Novielli,et al.  Sensing Developers’ Emotions: The Design of a Replicated Experiment , 2018, 2018 IEEE/ACM 3rd International Workshop on Emotion Awareness in Software Engineering (SEmotion).

[131]  Venera Arnaoudova,et al.  The Effect of Poor Source Code Lexicon and Readability on Developers' Cognitive Load , 2018, 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC).

[132]  Fenna M. Krienen,et al.  Opportunities and limitations of intrinsic functional connectivity MRI , 2013, Nature Neuroscience.

[133]  G. Ben-Shakhar,et al.  Publication recommendations for electrodermal measurements. , 1981, Psychophysiology.

[134]  Hidetake Uwano,et al.  Brain activity measurement during program comprehension with NIRS , 2014, 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD).

[135]  Jonathan Klein,et al.  Frustrating the user on purpose: using biosignals in a pilot study to detect the user's emotional state , 1998, CHI Conference Summary.

[136]  Kurt Schneider,et al.  Attention in Software Maintenance: An Eye Tracking Study , 2019, 2019 IEEE/ACM 6th International Workshop on Eye Movements in Programming (EMIP).

[137]  M. Elgendi On the Analysis of Fingertip Photoplethysmogram Signals , 2012, Current cardiology reviews.

[138]  B. Velichkovsky,et al.  Effective Connectivity within the Default Mode Network: Dynamic Causal Modeling of Resting-State fMRI Data , 2016, Front. Hum. Neurosci..

[139]  Víctor M. González,et al.  Measuring Concentration While Programming with Low-Cost BCI Devices: Differences Between Debugging and Creativity Tasks , 2015, HCI.

[140]  P. Dayan Methods in Mind. Cognitive Neuroscience. , 2007 .

[141]  Thomas Leich,et al.  Understanding understanding source code with functional magnetic resonance imaging , 2014, ICSE.

[142]  Joseph D. Bronzino,et al.  The Biomedical Engineering Handbook , 1995 .

[143]  M. Bradley,et al.  Emotion, attention, and the startle reflex. , 1990, Psychological review.

[144]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[145]  Thomas Fritz,et al.  Sensing Interruptibility in the Office: A Field Study on the Use of Biometric and Computer Interaction Sensors , 2018, CHI.

[146]  Giancarlo Succi,et al.  Initial evaluation of the brain activity under different software development situations , 2019, SEKE.

[147]  Nicole Novielli,et al.  Introduction to the special issue on affect awareness in software engineering , 2019, J. Syst. Softw..

[148]  René Riedl,et al.  Are There Neural Gender Differences in Online Trust? An fMRI Study on the Perceived Trustworthiness of eBay Offers , 2010, MIS Q..

[149]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[150]  R. Lystad,et al.  Functional neuroimaging: a brief overview and feasibility for use in chiropractic research. , 2009, The Journal of the Canadian Chiropractic Association.

[151]  Venkataraman Ramesh,et al.  Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journals , 2002, J. Manag. Inf. Syst..

[152]  Sven Apel,et al.  Measuring neural efficiency of program comprehension , 2017, ESEC/SIGSOFT FSE.

[153]  Karl J. Friston Functional integration and inference in the brain , 2002, Progress in Neurobiology.

[154]  Michal R. Wrobel,et al.  Applicability of Emotion Recognition and Induction Methods to Study the Behavior of Programmers , 2018 .

[155]  Andrew T. Duchowski,et al.  The Low/High Index of Pupillary Activity , 2020, CHI.

[156]  B. Delman,et al.  Hippocampal subfield-specific connectivity findings in major depressive disorder: A 7 Tesla diffusion MRI study. , 2019, Journal of psychiatric research.

[157]  M. Murray,et al.  EEG source imaging , 2004, Clinical Neurophysiology.

[158]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[159]  Thomas Leich,et al.  Toward measuring program comprehension with functional magnetic resonance imaging , 2012, SIGSOFT FSE.

[160]  Bruno Carreiro da Silva,et al.  Measuring the Cognitive Load of Software Developers: A Systematic Mapping Study , 2019, 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC).

[161]  G. McArthur,et al.  Validation of the Emotiv EPOC® EEG gaming system for measuring research quality auditory ERPs , 2013, PeerJ.

[162]  Kai Puolamäki,et al.  Biosignals reflect pair-dynamics in collaborative work: EDA and ECG study of pair-programming in a classroom environment , 2018, Scientific Reports.

[163]  John C Gore,et al.  Assessing functional connectivity in the human brain by fMRI. , 2007, Magnetic resonance imaging.

[164]  Angelika Dimoka,et al.  How to Conduct a Functional Magnetic Resonance (fMRI) Study in Social Science Research , 2012, MIS Q..

[165]  Alan R. Hevner,et al.  Towards a NeuroIS Research Methodology: Intensifying the Discussion on Methods, Tools, and Measurement , 2014, J. Assoc. Inf. Syst..

[166]  Jonathan I. Maletic,et al.  Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[167]  Sven Apel,et al.  Beyond gaze: preliminary analysis of pupil dilation and blink rates in an fMRI study of program comprehension , 2018, EMIP@ETRA.

[168]  Antonio Fernández-Caballero,et al.  Artificial Neural Networks to Assess Emotional States from Brain-Computer Interface , 2018, Electronics.

[169]  Yu Yan,et al.  Detecting and comparing brain activity in short program comprehension using EEG , 2017, 2017 IEEE Frontiers in Education Conference (FIE).

[170]  Chris Parnin,et al.  Studying Sustained Attention and Cognitive States with Eye Tracking in Remote Technical Interviews , 2015 .

[171]  Luis Emilio Bruni,et al.  Gaze strategies can reveal the impact of source code features on the cognitive load of novice programmers , 2018 .

[172]  P. Carvalho,et al.  Software code complexity assessment using EEG features , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[173]  Jonathan I. Maletic,et al.  Studying developer gaze to empower software engineering research and practice , 2016, SIGSOFT FSE.

[174]  Hidetake Uwano,et al.  Time series analysis of programmer's EEG for debug state classification , 2019, Programming.

[175]  B. Cowley,et al.  Cognitive Collaboration Found in Cardiac Physiology: Study in Classroom Environment , 2016, PloS one.

[176]  Dylan D. Schmorrow,et al.  Augmented Cognition. Enhancing Cognition and Behavior in Complex Human Environments , 2017, Lecture Notes in Computer Science.

[177]  Masaki Nakanishi,et al.  Assessing the effects of voluntary and involuntary eyeblinks in independent components of electroencephalogram , 2016, Neurocomputing.

[178]  Hasan Ayaz,et al.  Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures , 2017, Front. Hum. Neurosci..

[179]  Sven Apel,et al.  CodersMUSE: Multi-Modal Data Exploration of Program-Comprehension Experiments , 2019, 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC).

[180]  Martin Raubal,et al.  The Index of Pupillary Activity: Measuring Cognitive Load vis-à-vis Task Difficulty with Pupil Oscillation , 2018, CHI.

[181]  Soussan Djamasbi,et al.  Eye Tracking and Web Experience , 2014 .

[182]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[183]  Alan R. Hevner,et al.  Advancing a NeuroIS research agenda with four areas of societal contributions , 2020, Eur. J. Inf. Syst..

[184]  J. Mazziotta,et al.  Brain Mapping: The Methods , 2002 .

[185]  Yu Huang,et al.  Distilling Neural Representations of Data Structure Manipulation using fMRI and fNIRS , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[186]  Thomas Fritz,et al.  Using (Bio)Metrics to Predict Code Quality Online , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[187]  Nicole Novielli,et al.  Recognizing Developers' Emotions while Programming , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[188]  Chris Parnin,et al.  A Cognitive Neuroscience Perspective on Memory for Programming Tasks , 2010, PPIG.

[189]  Bonita Sharif,et al.  iTrace: enabling eye tracking on software artifacts within the IDE to support software engineering tasks , 2015, ESEC/SIGSOFT FSE.

[190]  Joseph H. Goldberg,et al.  Relating Perceived Web Page Complexity to Emotional Valence and Eye Movement Metrics , 2012 .

[191]  Spyros Doukakis Exploring brain activity and transforming knowledge in visual and textual programming using neuroeducation approaches , 2019, AIMS neuroscience.

[192]  Thomas Leich,et al.  A Look into Programmers’ Heads , 2020, IEEE Transactions on Software Engineering.

[193]  Sarah Fakhoury Moving towards objective measures of program comprehension , 2018, ESEC/SIGSOFT FSE.

[194]  Henrique Madeira,et al.  Biofeedback Augmented Software Engineering: Monitoring of Programmers' Mental Effort , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER).

[195]  Yann-Gaël Guéhéneuc,et al.  A practical guide on conducting eye tracking studies in software engineering , 2020, Empirical Software Engineering.

[196]  Chiarella Sforza,et al.  Spontaneous blinking in healthy persons: an optoelectronic study of eyelid motion , 2008, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[197]  Yann-Gaël Guéhéneuc,et al.  An empirical study on the importance of source code entities for requirements traceability , 2015, Empirical Software Engineering.

[198]  Matthew J. Brookes,et al.  Methods in mind , 2013 .

[199]  Karl J. Friston,et al.  Modelling functional integration: a comparison of structural equation and dynamic causal models , 2004, NeuroImage.

[200]  Henrique Madeira,et al.  The role of the insula in intuitive expert bug detection in computer code: an fMRI study , 2018, Brain Imaging and Behavior.