New Social Signals in a New Interaction World: The Next Frontier for Social Signal Processing

Social signal processing (SSP) is the domain aimed at the modeling, analysis, and synthesis of social behavior, particularly the nonverbal aspects. So far, the field has focused on face-to-face interactions, where it is possible to use the whole range of nonverbal cues that people have at their disposal to communicate (i.e., gestures, facial expressions, vocalizations, and so on). However, increasingly more interactions take place through communication technologies that limit the use of nonverbal cues (e.g., phone calls do not allow one to display facial expressions) or that require the adoption of artificial cues (e.g., “likes” on social media).

[1]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Richard Harper,et al.  Texture: Human Expression in the Age of Communications Overload , 2010 .

[3]  Björn Schuller,et al.  Computational Paralinguistics , 2013 .

[4]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[5]  Alessandro Vinciarelli,et al.  Automatic Role Recognition in Multiparty Conversations: An Approach Based on Turn Organization, Prosody, and Conditional Random Fields , 2012, IEEE Transactions on Multimedia.

[6]  Dirk Heylen,et al.  Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing , 2012, IEEE Transactions on Affective Computing.

[7]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[8]  Alex Pentland,et al.  Once Upon a Crime: Towards Crime Prediction from Demographics and Mobile Data , 2014, ICMI.

[9]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[10]  Kerstin Dautenhahn,et al.  Social Signal Processing in Social Robotics , 2017, Social Signal Processing.

[11]  I. Arminen,et al.  Mobile presence and intimacy—Reshaping social actions in mobile contextual configuration , 2009 .

[12]  A. Pentland.  Social Physics: how good ideas spread - the lessons from a new science , 2014 .

[13]  Alex Pentland,et al.  Pervasive Sensing to Model Political Opinions in Face-to-Face Networks , 2011, Pervasive.

[14]  Mohamed Chetouani,et al.  Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines , 2012, IEEE Transactions on Affective Computing.

[15]  Klaus R. Scherer,et al.  A psycho-ethological approach to social signal processing , 2012, Cognitive Processing.

[16]  Alessandro Vinciarelli,et al.  Negotiating over Mobile Phones: Calling or Being Called Can Make the Difference , 2014, Cognitive Computation.

[17]  Clifford Nass,et al.  Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship , 2005 .

[18]  Alex Pentland,et al.  Social sensing for epidemiological behavior change , 2010, UbiComp.

[19]  Maurizio Mancini,et al.  Implementing Expressive Gesture Synthesis for Embodied Conversational Agents , 2005, Gesture Workshop.

[20]  Zhigang Deng,et al.  Computer Facial Animation: A Survey , 2008 .

[21]  Maja Pantic,et al.  Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and tools , 2013, Image Vis. Comput..

[22]  Alessandro Perina,et al.  Unveiling the multimedia unconscious: implicit cognitive processes and multimedia content analysis , 2013, ACM Multimedia.

[23]  Roddy Cowie,et al.  Towards a conceptual framework of research on social signal processing , 2012, Journal on Multimodal User Interfaces.

[24]  E. Schegloff.  Analyzing Single Episodes of Interaction: An Exercise in Conversation Analysis , 1987 .

[25]  Alessandro Vinciarelli,et al.  A Survey of Personality Computing , 2014, IEEE Transactions on Affective Computing.

[26]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[27]  Vittorio Murino,et al.  Towards Computational Proxemics: Inferring Social Relations from Interpersonal Distances , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[28]  Radoslaw Niewiadomski,et al.  Laugh-aware virtual agent and its impact on user amusement , 2013, AAMAS.

[29]  J. Mccroskey,et al.  Nonverbal Behavior in Interpersonal Relations , 1987 .

[30]  A. Pentland.  Social Signal Processing [Exploratory DSP] , 2007, IEEE Signal Processing Magazine.

[31]  Chuohao Yeo,et al.  Modeling Dominance in Group Conversations Using Nonverbal Activity Cues , 2009, IEEE Transactions on Audio, Speech, and Language Processing.

[32]  N. House,et al.  Personal photography, digital technologies and the uses of the visual , 2011 .

[33]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Nathan Eagle,et al.  Reality Mining: Using Big Data to Engineer a Better World , 2014 .

[35]  Anna Esposito,et al.  When the Words are Not Everything: The Use of Laughter, Fillers, Back-Channel, Silence, and Overlapping Speech in Phone Calls , 2015, Front. ICT.

[36]  Hatice Gunes,et al.  A survey of perception and computation of human beauty , 2011, J-HGBU '11.

[37]  K. Scherer.  Personality inference from voice quality: The loud voice of extroversion. , 1978 .

[38]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[39]  Isabella Poggi,et al.  Social signals: a framework in terms of goals and beliefs , 2012, Cognitive Processing.