OODIDA: On-board/Off-board Distributed Data Analytics for Connected Vehicles

Connected vehicles may produce gigabytes of data per hour, which makes centralized data processing impractical at the fleet level. In addition, there are the problems of distributing tasks to edge devices and processing them efficiently. Our solution to this problem is OODIDA (On-board/off-board Distributed Data Analytics), which is a platform that tackles both task distribution to connected vehicles as well as concurrent execution of large-scale tasks on arbitrary subsets of clients. Its message-passing infrastructure has been implemented in Erlang/OTP, while the end points are language-agnostic. OODIDA is highly scalable and able to process a significant volume of data on resource-constrained clients.

[1]  Philip H. Enslow.  What is a "Distributed" Data Processing System? , 1978, Computer.

[2]  Jared Flatow,et al.  Disco: a computing platform for large-scale data analytics , 2011, Erlang '11.

[3]  Jignesh M. Patel,et al.  Storm@twitter , 2014, SIGMOD Conference.

[4]  Maurizio Morisio,et al.  Connected Car , 2016, ACM Comput. Surv..

[5]  Bruno Sericola,et al.  Distributed deep learning on edge-devices: Feasibility via adaptive compression , 2017, 2017 IEEE 16th International Symposium on Network Computing and Applications (NCA).

[6]  Peter Richtárik,et al.  Federated Optimization: Distributed Machine Learning for On-Device Intelligence , 2016, ArXiv.

[7]  Mats Jirstrand,et al.  Functional Federated Learning in Erlang (ffl-erl) , 2018, WFLP.

[8]  Omer Tene Jules Polonetsky,et al.  Privacy in the Age of Big Data: A Time for Big Decisions , 2012 .

[9]  R. K. Bunkar,et al.  Data Security and Privacy Protection Issues in Cloud Computing , 2014 .

[10]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[11]  Viktória Fördós,et al.  Towards secure Erlang systems , 2018, Erlang Workshop.

[12]  Blaise Agüera y Arcas,et al.  Federated Learning of Deep Networks using Model Averaging , 2016, ArXiv.

[13]  Pramod K. Varshney,et al.  Robust Federated Learning Using ADMM in the Presence of Data Falsifying Byzantines , 2017, ArXiv.

[14]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[15]  Allan L. Scherr.  Distributed Data Processing , 1978, IBM Syst. J..

[16]  Pramod K. Varshney,et al.  Robust Decentralized Learning Using ADMM With Unreliable Agents , 2017, IEEE Transactions on Signal Processing.

[17]  Simon Marlow,et al.  Parallel and Concurrent Programming in Haskell , 2013, CEFP.

[18]  Konstantinos Sagonas.  Experience from Developing the Dialyzer: A Static Analysis Tool Detecting Defects in Erlang Applications , 2005 .

[19]  Robert Harper,et al.  Practical Foundations for Programming Languages , 2012 .

[20]  Robert Stephens,et al.  A survey of stream processing , 1997, Acta Informatica.

[21]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[22]  Carl Hewitt,et al.  A Universal Modular ACTOR Formalism for Artificial Intelligence , 1973, IJCAI.