Meteor/Sopremo: An Extensible Query Language and Operator Model

Recently, quite a few query and scripting languages for MapReduce-based systems have been developed to ease formulating complex data analysis tasks. However, existing tools mainly provide basic operators for rather simple analyses, such as aggregating or ltering. Analytic functionality for advanced applications, such as data cleansing or information extraction can only be embedded in user-dened functions where the semantics is hidden from the query compiler and optimizer. In this paper, we present a language that treats application-specic functions as rst-class operators, so that operator semantics can be evaluated and exploited for optimization at compile time. We present Sopremo, a semantically rich operator model, and Meteor, an extensible query language that is grounded in Sopremo. Sopremo also provides a programming framework that allows users to easily develop and integrate extensions with their respective operators and instantiations. Meteor’s syntax is operator-oriented and uses a Json-like data model to support applications that analyze semi- and unstructured data. Meteor queries are translated into data ow programs of operator instantiations, i.e., concrete implementations of the involved Sopremo operators. Using a real-world example, we show how operators from dierent applications can be combined for writing complex analytical queries.