QuantCloud: Big Data Infrastructure for Quantitative Finance on the Cloud

In this paper, we present the QuantCloud infrastructure, designed for performing big data analytics in modern quantitative finance. Through analyzing market observations, quantitative finance (QF) utilizes mathematical models to search for subtle patterns and inefficiencies in financial markets to improve prospective profits. To discover profitable signals in anticipation of volatile trading patterns amid a global market, analytics are carried out on Exabyte-scale market metadata with a complex process in pursuit of a microsecond or even a nanosecond of data processing advantage. This objective motivates the development of innovative tools to address challenges for handling high volume, velocity, and variety investment instruments. Inspired by this need, we developed QuantCloud by employing large-scale SSD-backed datastore, various parallel processing algorithms, and portability in Cloud computing. QuantCloud bridges the gap between model computing techniques and financial data-driven research. The large volume of market data is structured in an SSD-backed datastore, and a daemon reacts to provide the Data-on-Demand services. Multiple client services process user requests in a parallel mode and query on-demand datasets from the datastore through Internet connections. We benchmark QuantCloud performance on a 40-core, 1TB-memory computer and a 5-TB SSD-backed datastore. We use NYSE TAQ data from the fourth quarter of 2014 as our market data. The results indicate data-access application latency as low as 3.6 nanoseconds per message, sustained throughput for parallel data processing as high as 74 million messages per second, and completion of 11 petabyte-level data analytics within 53 minutes. Our results demonstrate that the aggregated contributions of our infrastructure, parallel algorithms, and sophisticated implementations offer the algorithmic trading and financial engineering community new hope and numeric insights for their research and development.

[1]  M. Lewis.  Flash Boys: A Wall Street Revolt , 2014 .

[2]  Lawrence Harris,et al.  Equity Trading in the 21st Century , 2010 .

[3]  Allen Carrion.  Very Fast Money: High-Frequency Trading on the NASDAQ , 2013 .

[4]  Veda C. Storey,et al.  Business Intelligence and Analytics: From Big Data to Big Impact , 2012, MIS Q..

[5]  N. B. Anuar,et al.  The rise of "big data" on cloud computing: Review and open research issues , 2015, Inf. Syst..

[6]  Bin Fang,et al.  Big Data in Finance , 2016 .

[7]  Ed Thorp.  A Perspective on Quantitative Finance: Models for Beating the Market , 2004 .

[8]  Pierre Giot,et al.  Market Models: A Guide to Financial Data Analysis , 2003 .

[9]  Jae-Myung Kim,et al.  A case for flash memory ssd in enterprise database applications , 2008, SIGMOD Conference.

[10]  Alexander J. McNeil,et al.  Quantitative Risk Management: Concepts, Techniques and Tools : Concepts, Techniques and Tools , 2015 .

[11]  Maureen O'Hara,et al.  What Is a Quote? , 2010, The Journal of Trading.

[12]  Carol Alexander,et al.  Market Models: A Guide to Financial Data Analysis , 2001 .

[13]  Clara Vega,et al.  Rise of the Machines: Algorithmic Trading in the Foreign Exchange Market: Rise of the Machines , 2014 .

[14]  A. Menkveld.  High frequency trading and the new market makers , 2013 .

[15]  Dzmitry Kliazovich,et al.  GreenCloud: a packet-level simulator of energy-aware cloud computing data centers , 2010, The Journal of Supercomputing.

[16]  Samee Ullah Khan,et al.  Future Generation Computer Systems ( ) – Future Generation Computer Systems towards Secure Mobile Cloud Computing: a Survey , 2022 .

[17]  Alexander Zeier,et al.  In-Memory Data Management: Technology and Applications , 2012 .

[18]  Kingsley Y. L. Fong,et al.  Algorithmic Trading and Market Quality: International Evidence , 2015, Journal of Financial and Quantitative Analysis.

[19]  Meikang Qiu,et al.  Design and Architecture of Dell Acceleration Appliances for Database (DAAD): A Practical Approach with High Availability Guaranteed , 2015, 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems.

[20]  Gerhard Stahl,et al.  Applied Quantitative Finance , 2002 .

[21]  Peng Zhang,et al.  Interlacing Bypass Rings to Torus Networks for More Efficient Networks , 2011, IEEE Transactions on Parallel and Distributed Systems.

[22]  Li Zhang,et al.  Analysis of Linpack and power efficiencies of the world's TOP500 supercomputers , 2013, Parallel Comput..

[23]  Robert Kissell,et al.  Algorithmic Decision-Making Framework , 2005, Algorithmic Trading Methods.

[24]  Peng Zhang,et al.  Quantitative Data Analysis in Finance , 2017, Handbook of Big Data Technologies.

[25]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[26]  Dzmitry Kliazovich,et al.  GreenCloud: A Packet-Level Simulator of Energy-Aware Cloud Computing Data Centers , 2010, GLOBECOM.

[27]  Clara Vega,et al.  Rise of the Machines : Algorithmic Trading in the Foreign Exchange Market , 2009 .

[28]  Rajkumar Buyya,et al.  Article in Press Future Generation Computer Systems ( ) – Future Generation Computer Systems Cloud Computing and Emerging It Platforms: Vision, Hype, and Reality for Delivering Computing as the 5th Utility , 2022 .