Exact convex formulations of network-oriented optimal operator placement

Data processing tasks are increasingly spread across the internet to account for the spatially distributed nature of many data sources. In order to use network resources efficiently, subtasks need to be distributed in the network so data can be filtered close to the data sources. Previous approaches to this operator placement problem relied on various heuristics to constrain the complexity of the problem. In this paper, we propose two generic integer constrained problem formulations: a topology aware version which provides a placement including the specific network links as well as an end-to-end delay aware version which relies on the routing capabilities of the network. A linear programming relaxation for both versions is provided which allows exact and efficient solution using common solvers.

[1]  Olga Papaemmanouil,et al.  Supporting Generic Cost Models for Wide-Area Stream Processing , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[2]  Yanif Ahmad,et al.  Networked Query Processing for Distributed Stream-Based Applications , 2004, VLDB.

[3]  Frank Dürr,et al.  Fulfilling end-to-end latency constraints in large-scale streaming environments , 2011, 30th IEEE International Performance Computing and Communications Conference.

[4]  David K. Smith Theory of Linear and Integer Programming , 1987 .

[5]  Kurt Rothermel,et al.  Multilevel Predictions for the Aggregation of Data in Global Sensor Networks , 2010, 2010 IEEE/ACM 14th International Symposium on Distributed Simulation and Real Time Applications.

[6]  Ling Shi,et al.  Kalman Filtering Over Graphs: Theory and Applications , 2009, IEEE Transactions on Automatic Control.

[7]  Frank Dürr,et al.  A System for Distributed Context Reasoning , 2010, 2010 Sixth International Conference on Autonomic and Autonomous Systems.

[8]  G. Dantzig,et al.  Integral Extreme Points , 1968 .

[9]  Ying Li,et al.  Placement Strategies for Internet-Scale Data Stream Systems , 2008, IEEE Internet Computing.

[10]  Margo I. Seltzer,et al.  Network-Aware Operator Placement for Stream-Processing Systems , 2006, 22nd International Conference on Data Engineering (ICDE'06).

[11]  Navendu Jain,et al.  Adaptive Control of Extreme-scale Stream Processing Systems , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[12]  Jennifer Widom,et al.  Operator placement for in-network stream query processing , 2005, PODS.