This paper describes a novel approach to detect correlation from data streams in the context of MobiMine -- an experimental mobile data mining system. It presents a brief description of the MobiMine and identifies the problem of detecting dependencies among stocks from incrementally observed financial data streams. This is a non-trivial problem since the stock-market data is inherently noisy and small incremental volumes of data makes the estimation process more vulnerable to noise. This paper presents EDS, a technique to estimate the correlation matrix from data streams by exploiting some properties of the distribution of eigenvalues for random matrices. It separates the "information" from the "noise" by comparing the eigen-spectrum generated from the observed data with that of random matrices. The comparison immediately leads to a decomposition of the covariance matrix into two matrices: one capturing the "noise" and the other capturing useful "information." The paper also presents experimental results using Nasdaq 100 stock data.
[1]
Joseph S. Zirilli.
Financial Prediction Using Neural Networks
,
1996
.
[2]
W. Andrew,et al.
LO, and A.
,
1988
.
[3]
M. Stephanov,et al.
Random Matrices
,
2005,
hep-ph/0509286.
[4]
Lei Liu,et al.
MobiMine: monitoring the stock market from a PDA
,
2002,
SKDD.
[5]
Dag Jonsson.
Some limit theorems for the eigenvalues of a sample covariance matrix
,
1982
.
[6]
Chu-Sing Yang,et al.
Adaptive and Fault-Tolerant Wormhole Routing in Hpyercube Network
,
1993,
J. Inf. Sci. Eng..
[7]
T. Andersen.
THE ECONOMETRICS OF FINANCIAL MARKETS
,
1998,
Econometric Theory.
[8]
N. Baba,et al.
An intelligent forecasting system of stock price using neural networks
,
1992,
[Proceedings 1992] IJCNN International Joint Conference on Neural Networks.
[9]
E. Michael Azoff,et al.
Neural Network Time Series: Forecasting of Financial Markets
,
1994
.