Darling or Babygirl ? Investigating Stylistic Bias in Sentiment Analysis

Sentiment analysis is increasingly used for a range of applications from customer service to opinion mining. Stylistic bias arises when text generated by different groups of people expressing the same underlying content receive disparate treatment. Using three lexical alignment techniques, we find that standard sentiment models produce significantly different sentiment scores for word pairs that mainly differ stylistically. We suggest a simple align and substitute method to automatically generate examples of potentially undesirable biases in black-box models in order to better facilitate identification and mitigation of differential treatment based on stylistic variation.