Manual semantic annotations: User evaluation of interface and interaction designs

Abstract Semantic annotation is the process by which existing texts receive a mark-up that allows automatic identification of named entities (e.g., to distinguish between a turkey bird and the country Turkey). Manual annotation is useful both as stand-alone process in a domain-specific setting or as post-processing for automatic annotation algorithms. Development of most annotation tools strongly focuses on providing novel functionality — end-user evaluations of interface and user interaction are rare. This article reports on the results of a series of user studies executed to explore how non-expert users understand the process of enriching texts with semantic annotations. We find that these users can easily create simple semantic annotations (e.g., assigning concepts to text passages) but have difficulties understanding complex semantic annotations (e.g., assigning semantic identifiers to text passages). We present the results of three user studies on manual semantic annotation and discusses the lessons learned about both the semantic enrichment process and our methodology of exposing non-experts to semantic enrichment.