A multisensory approach for the design of food and drink enhancing sonic systems

Everyday eating and drinking experiences involve multiple, interrelated, sensory inputs. However, when it comes to human-food interaction design (HFI) research, certain senses have received more attention than others have. Here, we focus on audition, a sense that has received limited attention in such context. In particular, we highlight the role of food/drink-related eating sounds, as a potential input for human-food interaction design. We review some of the few systems that have built on such sounds within food and drink contexts. We also present a multisensory design framework and discuss how the systematic connections that exist between the senses may provide some guidelines for the integration of eating sounds in HFI design. Finally, we present some key prospects that we foresee for research in technology design in HFI.

[1]  C. Spence.  Crossmodal correspondences: A tutorial review , 2011, Attention, perception & psychophysics.

[2]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[3]  John M. de Castro,et al.  Listening to music while eating is related to increases in people's food intake and meal duration , 2006, Appetite.

[4]  M. Bourne,et al.  CRISPNESS IN FOODS-A REVIEW , 1976 .

[5]  Adrian David Cheok,et al.  Crossmodal correspondences between taste and shape, and their implications for product packaging: A review , 2016 .

[6]  Charles Spence,et al.  Sensory expectations elicited by the sounds of opening the packaging and pouring a beverage , 2015, Flavour.

[7]  Lawrence E. Marks,et al.  Synesthesia: Strong and Weak , 2001 .

[8]  John Prescott,et al.  Multisensory processes in flavour perception and their influence on food choice , 2015 .

[9]  L. Marks.  The Unity of the Senses , 1978 .

[10]  John Prescott,et al.  Flavour as a psychological construct: implications for perceiving and measuring the sensory qualities of foods , 1999 .

[11]  Hidekazu Tanaka,et al.  Chewing jockey: augmented food texture by using sound based on the cross-modal effect , 2011, SA '11.

[12]  Masahiko Inami,et al.  Straw-Like User Interface (II): A New Method of Presenting Auditory Sensations for a More Natural Experience , 2008, EuroHaptics.

[13]  Ryan S. Elder,et al.  The crunch effect: Food sound salience as a consumption monitoring cue , 2016 .

[14]  Robin Dando,et al.  A crossmodal role for audition in taste perception. , 2015, Journal of experimental psychology. Human perception and performance.

[15]  C. Spence,et al.  The Perfect Meal: The Multisensory Science of Food and Dining , 2014 .

[16]  F. Marinetti.  The Founding and Manifesto of Futurism , 2016 .

[17]  C. Spence,et al.  Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes , 2013, Psychonomic bulletin & review.

[18]  Marc O Ernst,et al.  Natural auditory scene statistics shapes human spatial hearing , 2014, Proceedings of the National Academy of Sciences.

[19]  Karen B. Schloss,et al.  Music-to-Color Associations of Single-Line Piano Melodies in Non-synesthetes. , 2016, Multisensory research.

[20]  S. Fiske,et al.  The Structure of Value: Accounting for Taste George Mandler , 2014 .

[21]  Carlos Velasco,et al.  The Sound of Temperature: What Information do Pouring Sounds Convey Concerning the Temperature of a Beverage , 2013 .

[22]  Marc Leman,et al.  Does Music Influence the Multisensory Tasting Experience , 2015 .

[23]  Ali Israr,et al.  Touch, Taste, & Smell User Interfaces: The Future of Multisensory HCI , 2016, CHI Extended Abstracts.

[24]  Michael Haverkamp,et al.  Synesthetic approach for evaluation of the cross-sensory quality of multi-media applications , 2010, 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX).

[25]  R. E. Milliman,et al.  The Influence of Background Music on the Behavior of Restaurant Patrons , 1986 .

[26]  C. Spence.  Multisensory Flavor Perception , 2015, Cell.

[27]  L. Marks.  The Unity of the Senses: Interrelations Among the Modalities , 1978 .

[28]  C. Spence.  Eating with our ears: assessing the importance of the sounds of consumption on our perception and enjoyment of multisensory flavour experiences , 2015, Flavour.

[29]  Carlos Velasco,et al.  Music Influences Hedonic and Taste Ratings in Beer , 2016, Front. Psychol..

[30]  Marcia Grabowecky,et al.  Object-based auditory facilitation of visual search for pictures and words with frequent and rare targets. , 2011, Acta psychologica.

[31]  Geke D.S. Ludden,et al.  Surprise As a Design Strategy , 2008, Design Issues.

[32]  Adrian David Cheok,et al.  Sensory marketing in light of new technologies , 2015, Advances in Computer Entertainment.

[33]  Geke D.S. Ludden,et al.  Effects of visual-auditory incongruity on product expression and surprise , 2007 .

[34]  J. Vroomen,et al.  Intersensory binding across space and time: A tutorial review , 2013, Attention, Perception, & Psychophysics.

[35]  Charles Spence,et al.  The role of auditory cues in modulating the perceived crispness and staleness of potato chips , 2004 .

[36]  Nicolas Guéguen,et al.  Sound level of environmental music and drinking behavior: a field experiment with beer drinkers. , 2008, Alcoholism, clinical and experimental research.

[37]  Cesare Parise,et al.  Audiovisual Cross-Modal Correspondences in the General Population , 2013 .

[38]  Charles Spence,et al.  Assessing the Role of Sound in the Perception of Food and Drink , 2010 .

[39]  Koji Tsukada,et al.  EducaTableware: computer-augmented tableware to enhance the eating experiences , 2013, CHI Extended Abstracts.

[40]  Michael Haverkamp,et al.  Synesthetic Design: Handbook for a Multi-Sensory Approach , 2012 .

[41]  C. Spence,et al.  The multisensory perception of flavor , 2008, Consciousness and Cognition.

[42]  Jeremy N. Bailenson,et al.  Virtual Experiences, Physical Behaviors: The Effect of Presence on Imitation of an Eating Avatar , 2009, PRESENCE: Teleoperators and Virtual Environments.

[43]  C. Spence,et al.  Technology at the dining table , 2013, Flavour.

[44]  Michael Haverkamp,et al.  Synesthetic Design – Building Multi-sensory Arrangements , 2009 .

[45]  Geke D. S. Ludden,et al.  Surprise and humor in product design: Designing sensory metaphors in multiple modalities , 2012 .

[46]  M. Haverkamp,et al.  mayo 2015 CAN SYNESTHETIC PERCEPTION HELP TO DEFINE ATTRACTIVE PRODUCT DESIGN ? , 2015 .

[47]  Akio Yamamoto,et al.  Straw-like user interface: virtual experience of the sensation of drinking using a straw , 2006, ACE '06.

[48]  Charles Spence,et al.  Auditory contributions to flavour perception and feeding behaviour , 2012, Physiology & Behavior.

[49]  R. Lickliter,et al.  Intersensory redundancy educates selective attention in bobwhite quail embryos. , 2006, Developmental science.

[50]  R. Lickliter,et al.  Intersensory Redundancy Guides the Development of Selective Attention, Perception, and Cognition in Infancy , 2004 .