Using profile similarity to measure agreement in personality perception

Personality plays a very important role to generate conversational agents with believable interaction capabilities, as well as to build rich user models that consider their peculiarities and behavioural styles. To render or recognize personality, it is necessary to reliably compute agreement across ratings of personality, either to evaluate whether an agent’s personality is perceived as intended by human observers, to measure the reliability of the annotation of personality corpora, or to assess if users personality is correctly and/or consistently predicted by automatic recognizers. However, in the literature the automatic measurement of agreement is usually computed using observed agreement or kappa measures. This leads to a loss of information as agreement is generally computed trait-wise and then averaged, instead of considering personality as a profile with several dimensions and comparing the full profiles over all traits. In this paper we discuss the possibilities offered by a repertoire of profile similarity coefficients that can be used either with traits defined ad-hoc for the application domain or with standardized personality trait scores. To show the suitability of our proposal, we have used it with the SSPNet Speaker Personality Corpus, and show that these coefficients provide valuable information that complements state-of-the-art evaluation approaches.