Addressing Annotation Complexity: The Case of Annotating Ideological Perspective in Egyptian Social Media

Automatically detecting the stance of people toward political and ideological topics –namely their “Ideological Perspective”– from social media is a rapidly growing research area with a wide range of applications. Research in such a field faces several challenges among which is the lack of annotated corpora and associated guidelines for collecting annotations. The problem is even more pronounced in situations where there is no clear taxonomy for the common community perspectives and ideologies. The challenges are exacerbated when the communities where we need to gather these annotations are in a state of turmoil causing subjectivity and intimidation to be factors in the annotation process. Accordingly, we present the process for creating a robust and succinct set of guidelines for annotating “Egyptian Ideological Perspectives”. We collect social media data discussing Egyptian politics and develop an iterative feedback annotation framework refining the annotation task and associated guidelines attempting to circumvent both weaknesses. Our efforts lead to a significant increase in inter-annotator agreement measures from 75.7% to 92% overall agreement.