DeePay: deep learning decodes EEG to predict consumer’s willingness to pay for neuromarketing

There is an increasing demand within consumer-neuroscience (or neuromarketing) for objective neural measures to quantify consumers’ subjective valuations and predict responses to marketing campaigns. However, the properties of EEG raise difficulties for these aims: small datasets, high dimensionality, elaborate manual feature extraction, intrinsic noise, and between-subject variations. We aimed to overcome these limitations by combining unique techniques of Deep Learning Networks (DLNs), while providing interpretable results for neuroscientific and decision-making insight. In this study, we developed a DLN to predict subjects’ willingness to pay (WTP) based on their EEG data. In each trial, 213 subjects observed a product’s image, from 72 possible products, and then reported their WTP for the product. The DLN employed EEG recordings from product observation to predict the corresponding reported WTP values. Our results showed 0.276 test root-mean-square-error and 75.09% test accuracy in predicting high vs. low WTP, surpassing other models and a manual feature extraction approach. Network visualizations provided the predictive frequencies of neural activity, their scalp distributions, and critical timepoints, shedding light on the neural mechanisms involved with evaluation. In conclusion, we show that DLNs may be the superior method to perform EEG-based predictions, to the benefit of decision-making researchers and marketing practitioners alike.

[1]  Doron Friedman,et al.  Machines learn neuromarketing: Improving preference prediction from self-reports using multiple EEG measures and machine learning , 2020 .

[2]  Nasim Alnuman,et al.  Classification of Products Preference from EEG Signals using SVM Classifier , 2020, 2020 12th International Conference on Information Technology and Electrical Engineering (ICITEE).

[3]  Varun Bajaj,et al.  Time–Frequency Representation and Convolutional Neural Network-Based Emotion Recognition , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Krishna P. Miyapuram,et al.  Understanding Consumer Preferences for Movie Trailers from EEG using Machine Learning , 2020, ArXiv.

[5]  Maarten A. S. Boksem,et al.  Measuring Neural Arousal for Advertisements and Its Relationship With Advertising Success , 2020, Frontiers in Neuroscience.

[6]  Hamed Jalaly Bidgoly,et al.  A survey on methods and challenges in EEG based authentication , 2020, Comput. Secur..

[7]  Luca Benini,et al.  EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain–Machine Interfaces , 2020, 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[8]  Chao Li,et al.  Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition , 2020, Inf. Process. Manag..

[9]  Marcel van Gerven,et al.  Explainable Deep Learning: A Field Guide for the Uninitiated , 2020, J. Artif. Intell. Res..

[10]  Padmavati Khandnor,et al.  A comparative analysis of signal processing and classification methods for different applications based on EEG signals , 2020 .

[11]  Sungho Jo,et al.  Deep Physiological Affect Network for the Recognition of Human Emotions , 2020, IEEE Transactions on Affective Computing.

[12]  J. Contreras-Vidal,et al.  An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding , 2020, Scientific Reports.

[13]  Partha Pratim Roy,et al.  Fusion of EEG response and sentiment analysis of products review to predict customer satisfaction , 2019, Inf. Fusion.

[14]  Muhammad Ghulam,et al.  Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion , 2019, Future Gener. Comput. Syst..

[15]  Yangsong Zhang,et al.  Predicting individual decision-making responses based on single-trial EEG , 2019, NeuroImage.

[16]  Jiaming Zhang,et al.  EEG Emotion Classification Using an Improved SincNet-Based Deep Learning Model , 2019, Brain sciences.

[17]  Gianluca Di Flumeri,et al.  Consumer Behaviour through the Eyes of Neurophysiological Measures: State-of-the-Art and Future Trends , 2019, Comput. Intell. Neurosci..

[18]  Xiaojun Bi,et al.  Deep Spatial-Temporal Neural Network for Classification of EEG-Based Motor Imagery , 2019, Proceedings of the 2019 International Conference on Artificial Intelligence and Computer Science.

[19]  Ihsan Ullah,et al.  Automatic Emotion Recognition (AER) System based on Two-Level Ensemble of Lightweight Deep CNN Models , 2019, ArXiv.

[20]  Mohammed Imamul Hassan Bhuiyan,et al.  End-to-end Sleep Staging with Raw Single Channel EEG using Deep Residual ConvNets , 2019, 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[21]  Radoslaw Martin Cichy,et al.  Deep Neural Networks as Scientific Models , 2019, Trends in Cognitive Sciences.

[22]  Y. F. Huang,et al.  Accurate EEG-Based Emotion Recognition on Combined Features Using Deep Convolutional Neural Networks , 2019, IEEE Access.

[23]  R. Knight,et al.  Individual EEG measures of attention, memory, and motivation predict population level TV viewership and Twitter engagement , 2019, PloS one.

[24]  U. Rajendra Acharya,et al.  Cascaded LSTM recurrent neural network for automated sleep stage classification using single-channel EEG signals , 2019, Comput. Biol. Medicine.

[25]  Mohammad A. Almogbel,et al.  Cognitive Workload Detection from Raw EEG-Signals of Vehicle Driver using Deep Learning , 2019, 2019 21st International Conference on Advanced Communication Technology (ICACT).

[26]  Mario Ignacio Chacon Murguia,et al.  Classification of multiple motor imagery using deep convolutional neural networks and spatial filters , 2019, Appl. Soft Comput..

[27]  Tiago H. Falk,et al.  Deep learning-based electroencephalography analysis: a systematic review , 2019, Journal of neural engineering.

[28]  Dino J Levy,et al.  A gateway to consumers' minds: Achievements, caveats, and prospects of electroencephalography-based prediction in neuromarketing. , 2018, Wiley interdisciplinary reviews. Cognitive science.

[29]  Gordon Cheng,et al.  Validating Deep Neural Networks for Online Decoding of Motor Imagery Movements from EEG Signals , 2018, Sensors.

[30]  Filip Karlo Dosilovic,et al.  Explainable artificial intelligence: A survey , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[31]  Joseph Ciorciari,et al.  Consumer neuroscience for marketing researchers , 2018 .

[32]  Laurent Vercueil,et al.  A convolutional neural network for sleep stage scoring from raw single-channel EEG , 2018, Biomed. Signal Process. Control..

[33]  T. Ramsøy,et al.  Frontal Brain Asymmetry and Willingness to Pay , 2018, Front. Neurosci..

[34]  Pan Wang,et al.  Using Support Vector Machine on EEG for Advertisement Impact Assessment , 2018, Front. Neurosci..

[35]  Neil Davey,et al.  The Correlation between EEG Signals as Measured in Different Positions on Scalp Varying with Distance , 2018, BICA.

[36]  Terry L. Childers,et al.  Applying EEG in consumer neuroscience , 2018 .

[37]  Joseph Picone,et al.  Deep Architectures for Automated Seizure Detection in Scalp EEGs , 2017, ArXiv.

[38]  Maria Theodorou,et al.  Your Brain on the Movies: A Computational Approach for Predicting Box-office Performance from Viewer’s Brain Responses to Movie Trailers , 2017, Front. Neuroinform..

[39]  Mohsen Pourahmadi,et al.  Reducing Brain Signal Noise in the Prediction of Economic Choices: A Case Study in Neuroeconomics , 2017, Front. Neurosci..

[40]  J. Guixeres,et al.  Consumer Neuroscience-Based Metrics Predict Recall, Liking and Viewing Rates in Online Advertising , 2017, Front. Psychol..

[41]  U. Rajendra Acharya,et al.  Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals , 2017, Comput. Biol. Medicine.

[42]  Debi Prosad Dogra,et al.  Analysis of EEG signals and its application to neuromarketing , 2017, Multimedia Tools and Applications.

[43]  Xue Li Lim,et al.  Neural signals of selective attention are modulated by subjective preferences and buying decisions in a virtual shopping task , 2017, Biological Psychology.

[44]  Klaus-Robert Müller,et al.  Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models , 2017, ArXiv.

[45]  Carolyn Yoon,et al.  When Brain Beats Behavior: Neuroforecasting Crowdfunding Outcomes , 2017, The Journal of Neuroscience.

[46]  Ming Hsu,et al.  Neuromarketing: Inside the Mind of the Consumer , 2017 .

[47]  Moran Cerf,et al.  A Ticket for Your Thoughts: Method for Predicting Content Recall and Sales Using Neural Similarity of Moviegoers , 2017 .

[48]  Nick Lee,et al.  This is your brain on neuromarketing: reflections on a decade of research , 2017 .

[49]  Cuntai Guan,et al.  Convolutional neural network-based transfer learning and knowledge distillation using multi-subject data in motor imagery BCI , 2017, 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER).

[50]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[51]  Chao Wu,et al.  DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw Single-Channel EEG , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[52]  Ugur Halici,et al.  A novel deep learning approach for classification of EEG motor imagery signals , 2017, Journal of neural engineering.

[53]  Shouqian Sun,et al.  Single-trial EEG classification of motor imagery using deep convolutional neural networks , 2017 .

[54]  Dario Pompili,et al.  Cloud-based deep learning of big EEG data for epileptic seizure prediction , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[55]  Brent Lance,et al.  EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces , 2016, Journal of neural engineering.

[56]  Saeid Sanei,et al.  Deep learning for epileptic intracranial EEG data , 2016, 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP).

[57]  Thomas E. Nichols,et al.  Scanning the horizon: towards transparent and reproducible neuroimaging research , 2016, Nature Reviews Neuroscience.

[58]  Fei Wang,et al.  Predicting Seizures from Electroencephalography Recordings: A Knowledge Transfer Strategy , 2016, 2016 IEEE International Conference on Healthcare Informatics (ICHI).

[59]  M. Falkenstein,et al.  ERP Correlates of Simulated Purchase Decisions , 2016, Front. Neurosci..

[60]  U. Karmarkar,et al.  Consumer Neuroscience: Advances in Understanding Consumer Psychology , 2016 .

[61]  Joelle Pineau,et al.  Learning Robust Features using Deep Learning for Automatic Seizure Detection , 2016, MLHC.

[62]  Fabio Babiloni,et al.  Gender and Age Related Effects While Watching TV Advertisements: An EEG Study , 2016, Comput. Intell. Neurosci..

[63]  Heesung Kwon,et al.  Single-trial EEG RSVP classification using convolutional neural networks , 2016, Defense + Security.

[64]  Tinoosh Mohsenin,et al.  Wearable seizure detection using convolutional neural networks with transfer learning , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[65]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[66]  Faisal Mushtaq,et al.  Randomised prior feedback modulates neural signals of outcome monitoring , 2016, NeuroImage.

[67]  Ran Manor,et al.  Convolutional Neural Network for Multi-Category Rapid Serial Visual Presentation BCI , 2015, Front. Comput. Neurosci..

[68]  Mohammed Yeasin,et al.  Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks , 2015, ICLR.

[69]  S. Woolgar,et al.  Neuromarketing in the making: Enactment and reflexive entanglement in an emerging field , 2015, BioSocieties.

[70]  Sebastian Stober,et al.  Deep Feature Learning for EEG Recordings , 2015, ArXiv.

[71]  Ming Hsu,et al.  The neuroscience of consumer choice , 2015, Current Opinion in Behavioral Sciences.

[72]  Shuicheng Yan,et al.  Parallel convolutional-linear neural network for motor imagery classification , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[73]  William H. Hampton,et al.  Predicting Advertising success beyond Traditional Measures: New Insights from Neurophysiological Methods and Market Response Modeling , 2015 .

[74]  Ariel Telpaz,et al.  Using EEG to Predict Consumers’ Future Choices , 2015 .

[75]  Ale Smidts,et al.  Brain Responses to Movie Trailers Predict Individual Preferences for Movies and Their Population-Wide Commercial Success , 2015 .

[76]  Scott A. Huettel,et al.  Consumer Neuroscience: Applications, Challenges, and Possible Solutions , 2015 .

[77]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[78]  Davide Baldo,et al.  Brain Waves Predict Success of New Fashion Products: A Practical Application for the Footwear Retailing Industry , 2015 .

[79]  Greg H. Proudfit The reward positivity: from basic research on reward to a biomarker for depression. , 2015, Psychophysiology.

[80]  Sebastian Stober,et al.  Using Convolutional Neural Networks to Recognize Rhythm Stimuli from Electroencephalography Recordings , 2014, NIPS.

[81]  Andrzej Cichocki,et al.  Deep Learning of Multifractal Attributes from Motor Imagery Induced EEG , 2014, ICONIP.

[82]  Pasin Israsena,et al.  EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation , 2014, TheScientificWorldJournal.

[83]  Lianghua He,et al.  A Deep Learning Method for Classification of EEG Data Based on Motor Imagery , 2014, ICIC.

[84]  John S. Johnson,et al.  Audience preferences are predicted by temporal reliability of neural processing , 2014, Nature Communications.

[85]  Shinobu Kitayama,et al.  Advancing consumer neuroscience , 2014 .

[86]  Jorge Henrique Caldeira de Oliveira,et al.  A review of studies on neuromarketing: practical results, techniques, contributions and limitations , 2014 .

[87]  M. Murugappan,et al.  Wireless EEG signals based Neuromarketing system using Fast Fourier Transform (FFT) , 2014, 2014 IEEE 10th International Colloquium on Signal Processing and its Applications.

[88]  Miguel P. Eckstein,et al.  Single-Trial Classification of Event-Related Potentials in Rapid Serial Visual Presentation Tasks Using Supervised Spatial Filtering , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[89]  Josep Marco-Pallarés,et al.  Electrophysiological correlates of anticipating improbable but desired events , 2013, NeuroImage.

[90]  Joseph W. Kable,et al.  The valuation system: A coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value , 2013, NeuroImage.

[91]  Jordan J. Louviere,et al.  Consumer neuroscience: Assessing the brain response to marketing stimuli using electroencephalogram (EEG) and eye tracking , 2013, Expert Syst. Appl..

[92]  Sanqing Hu,et al.  Electronic evaluation for video commercials by impression index , 2013, Cognitive Neurodynamics.

[93]  John M. Pearson,et al.  Rapid Brain Responses Independently Predict Gain Maximization and Loss Minimization during Economic Decision Making , 2013, The Journal of Neuroscience.

[94]  Niklas Ravaja,et al.  Predicting purchase decision: The role of hemispheric asymmetry over the frontal cortex. , 2013 .

[95]  Nick Lee,et al.  Neuromarketing and consumer neuroscience: contributions to neurology , 2013, BMC Neurology.

[96]  Dino J. Levy,et al.  The root of all value: a neural common currency for choice , 2012, Current Opinion in Neurobiology.

[97]  Ricardo Chavarriaga,et al.  Self-paced movement intention detection from human brain signals: Invasive and non-invasive EEG , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[98]  Gamini Dissanayake,et al.  Choice modeling and the brain: A study on the Electroencephalogram (EEG) of preferences , 2012, Expert Syst. Appl..

[99]  L. Parra,et al.  Human Neuroscience Original Research Article Correlated Components of Ongoing Eeg Point to Emotionally Laden Attention – a Possible Marker of Engagement? , 2022 .

[100]  Dino J. Levy,et al.  Comparing Apples and Oranges: Using Reward-Specific and Reward-General Subjective Value Representation in the Brain , 2011, The Journal of Neuroscience.

[101]  C. Padoa-Schioppa Neurobiology of economic choice: a good-based model. , 2011, Annual review of neuroscience.

[102]  Clay B. Holroyd,et al.  Reward positivity elicited by predictive cues , 2011, Neuroreport.

[103]  Hubert Cecotti,et al.  Convolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[104]  Laura Astolfi,et al.  Spectral EEG frontal asymmetries correlate with the experienced pleasantness of TV commercial advertisements , 2011, Medical & Biological Engineering & Computing.

[105]  M. Hallett,et al.  Prediction of human voluntary movement before it occurs , 2011, Clinical Neurophysiology.

[106]  Klaus Miller,et al.  How Should Consumers’ Willingness to Pay be Measured? An Empirical Comparison of State-of-the-Art Approaches , 2011 .

[107]  P. Glimcher,et al.  Choice from Non-Choice: Predicting Consumer Preferences from Blood Oxygenation Level-Dependent Signals Obtained during Passive Viewing , 2011, The Journal of Neuroscience.

[108]  A. Choromańska,et al.  Application of frontal EEG asymmetry to advertising research , 2010 .

[109]  Ian Krajbich,et al.  Visual fixations and the computation and comparison of value in simple choice , 2010, Nature Neuroscience.

[110]  F. Cincotti,et al.  Changes in Brain Activity During the Observation of TV Commercials by Using EEG, GSR and HR Measurements , 2010, Brain Topography.

[111]  D. Ariely,et al.  Neuromarketing: the hope and hype of neuroimaging in business , 2010, Nature Reviews Neuroscience.

[112]  Yann LeCun,et al.  Classification of patterns of EEG synchronization for seizure prediction , 2009, Clinical Neurophysiology.

[113]  J. O'Doherty,et al.  Evidence for a Common Representation of Decision Values for Dissimilar Goods in Human Ventromedial Prefrontal Cortex , 2009, The Journal of Neuroscience.

[114]  A. Choromańska,et al.  Analysis of Neurophysiological Reactions to Advertising Stimuli by Means of EEG and Galvanic Skin Response Measures , 2009 .

[115]  Don M. Tucker,et al.  Corticolimbic mechanisms in the control of trial and error learning , 2009, Brain Research.

[116]  F. Cincotti,et al.  Neural Basis for Brain Responses to TV Commercials: A High-Resolution EEG Study , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[117]  P. Glimcher,et al.  The neural correlates of subjective value during intertemporal choice , 2007, Nature Neuroscience.

[118]  Clay B. Holroyd,et al.  It's worse than you thought: the feedback negativity and violations of reward prediction in gambling tasks. , 2007, Psychophysiology.

[119]  J. O'Doherty,et al.  Orbitofrontal Cortex Encodes Willingness to Pay in Everyday Economic Transactions , 2007, The Journal of Neuroscience.

[120]  Jonathan R. Folstein,et al.  Influence of cognitive control and mismatch on the N2 component of the ERP: a review. , 2007, Psychophysiology.

[121]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[122]  Clay B. Holroyd,et al.  ERP correlates of feedback and reward processing in the presence and absence of response choice. , 2005, Cerebral cortex.

[123]  Atsushi Sato,et al.  Effects of value and reward magnitude on feedback negativity and P300 , 2005, Neuroreport.

[124]  H. Murohashi,et al.  The ERPs to feedback indicating monetary loss and gain on the game of modified “rock–paper–scissors” , 2005 .

[125]  Clay B. Holroyd,et al.  Brain potentials associated with expected and unexpected good and bad outcomes. , 2005, Psychophysiology.

[126]  Stéphane Robin,et al.  Revealing consumers' willingness-to-pay: A comparison of the BDM mechanism and the Vickrey auction , 2004 .

[127]  Alan Gevins,et al.  Attention and Brain Activity While Watching Television: Components of Viewer Engagement , 2004 .

[128]  A. Sanfey,et al.  Independent Coding of Reward Magnitude and Valence in the Human Brain , 2004, The Journal of Neuroscience.

[129]  Tim Ambler,et al.  The distributed neuronal systems supporting choice‐making in real‐life situations: differences between men and women when choosing groceries detected using magnetoencephalography , 2004, The European journal of neuroscience.

[130]  L. F. Haas Hans Berger (1873–1941), Richard Caton (1842–1926), and electroencephalography , 2003, Journal of neurology, neurosurgery, and psychiatry.

[131]  Clay B. Holroyd,et al.  The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. , 2002, Psychological review.

[132]  Adrian R. Willoughby,et al.  The Medial Frontal Cortex and the Rapid Processing of Monetary Gains and Losses , 2002, Science.

[133]  D. Malacara-Hernández,et al.  A Review of Methods for Measuring Corneal Topography , 2001, Optometry and vision science : official publication of the American Academy of Optometry.

[134]  R. Davidson,et al.  Prefrontal brain electrical asymmetry predicts the evaluation of affective stimuli , 2000, Neuropsychologia.

[135]  R. Davidson,et al.  Anterior electrophysiological asymmetries, emotion, and depression: conceptual and methodological conundrums. , 1998, Psychophysiology.

[136]  C. Braun,et al.  Event-Related Brain Potentials Following Incorrect Feedback in a Time-Estimation Task: Evidence for a Generic Neural System for Error Detection , 1997, Journal of Cognitive Neuroscience.

[137]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[138]  U. Karmarkar,et al.  Consumer Neuroscience: Past, Present, and Future , 2019 .

[139]  Wojciech Samek,et al.  Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , 2019, Explainable AI.

[140]  Jason Teo,et al.  Classification of Affective States via EEG and Deep Learning , 2018 .

[141]  Marina Schmid,et al.  An Introduction To The Event Related Potential Technique , 2016 .

[142]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[143]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[144]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .