Blind source recovery: theoretical formulations, implementations and application to cdma communication systems

Blind Source Recovery (BSR) is an autonomous (or unsupervised) stochastic adaptation approach that denotes recovering signals from measurements in environments that may include convolution, transients, and even possible nonlinearity. The primary goal of BSR is to recover original source signals, as best as possible, even in the absence of precise environment identifiability. A discrete-time optimization framework for BSR has been adopted based on minimization of the Kullback-Lieblar Divergence, subject to the constraints of a state-space representation, using the Riemannian contra-variant gradient adaptation. The modeling of the environment and its representation is vital in the proposed framework. The adoption of the state space framework allows for the derivation of more general update laws capable of catering to most known filtering paradigms. For the interesting case of multi-variable linear time-invariant dynamic BSR, parametric update laws for discrete-time canonical feedforward and feedback state space configurations have been derived. Higher Order Statistics have been explored to obtain adaptation algorithms that are purely blind of actual source distributions. The BSR cases of undercomplete and overcomplete mixtures have also been investigated. A new two-stage BSR algebraic algorithm for sparse static overcomplete mixtures has been developed. Further, the BSR framework has been successfully formulated for multi-user detection in modern CDMA wireless communication networks. Promising new results clearly demonstrate the effectiveness and practicality of the formulated approach. Computer Simulations have been widely conducted during the course of this research to evaluate the performance of all the developed algorithms for a variety of scenarios. In most cases, the performance verification has been done using actual speech and communication data in the presence of noise. In summary the main contributions of this thesis are (1) Info-theoretic update laws for a multi-variable dynamic state space network were investigated. Natural gradient based BSR update laws for the cases of minimum phase and non-minimum phase mixing environments as well as both feedforward and feedback canonical state space configurations were developed and implemented. (2) An exploration and extension of parametric source distribution models and derivation of adaptive score-functions (or non-linearities) for the BSR of sources with multiple source distributions. (3) The development of the Algebraic ICA (AICA) Algorithm, which is a new ICA algorithm for blind, sparse, static mixing matrix recovery. It enables the BSR from overcomplete mixtures of speech and other sparse distributions using a combination of AICA and interior point linear programming (IP-LP) techniques. (4) Development and simulation of new Blind Multi-user Detection algorithms for DS-CDMA and WCDMA wireless communication networks based on the BSR framework.

[1]  Fathi M. A. Salam,et al.  Implementation and test results of a chip for the separation of mixed signals , 1995, Proceedings of ISCAS'95 - International Symposium on Circuits and Systems.

[2]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[3]  G. Erten,et al.  Formulation and algorithms for blind signal recovery , 1997, Proceedings of 40th Midwest Symposium on Circuits and Systems. Dedicated to the Memory of Professor Mac Van Valkenburg.

[4]  Tapani Ristaniemi,et al.  Advanced ICA-based receivers for block fading DS-CDMA channels , 2002, Signal Process..

[5]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[6]  Reimann,et al.  Unsupervised learning by examples: On-line versus off-line. , 1996, Physical review letters.

[7]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[8]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[9]  Kari Torkkola,et al.  Blind separation of convolved sources based on information maximization , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[10]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[11]  Pentti Leppanen,et al.  The Influence of Initial-Phases of a PN Code Set on the Performance ofan Asynchronous DS-CDMA System , 2000 .

[12]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[13]  Yunnan Wu,et al.  Spreading code assignment in an ad hoc DS-CDMA wireless network , 2002, 2002 IEEE International Conference on Communications. Conference Proceedings. ICC 2002 (Cat. No.02CH37333).

[14]  Ralph Linsker,et al.  A Local Learning Rule That Enables Information Maximization for Arbitrary Input Distributions , 1997, Neural Computation.

[15]  A. Benveniste,et al.  Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications , 1980 .

[16]  Fathi M. A. Salam,et al.  Blind source recovery using an adaptive generalized Gaussian score function , 2002, The 2002 45th Midwest Symposium on Circuits and Systems, 2002. MWSCAS-2002..

[17]  M.J. Heikkila.  A novel blind adaptive algorithm for channel equalization in WCDMA downlink , 2001, 12th IEEE International Symposium on Personal, Indoor and Mobile Radio Communications. PIMRC 2001. Proceedings (Cat. No.01TH8598).

[18]  Shoko Araki,et al.  The fundamental limitation of frequency domain blind source separation for convolutive mixtures of speech , 2003, IEEE Trans. Speech Audio Process..

[19]  H. Vincent Poor,et al.  Blind equalization and multiuser detection in dispersive CDMA channels , 1998, IEEE Trans. Commun..

[20]  Juha Karhunen,et al.  Neural approaches to independent component analysis and source separation , 1996, ESANN.

[21]  Upamanyu Madhow,et al.  Blind adaptive multiuser detection , 1995, IEEE Trans. Inf. Theory.

[22]  Shun-ichi Amari,et al.  A Theory of Adaptive Pattern Classifiers , 1967, IEEE Trans. Electron. Comput..

[23]  A. J. Bell,et al.  A Unifying Information-Theoretic Framework for Independent Component Analysis , 2000 .

[24]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[25]  Fathi M. A. Salam,et al.  Exact entropy series representation for blind source separation , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[26]  Visa Koivunen,et al.  Adaptive Score Functions for Maximum Likelihood ICA , 2002, J. VLSI Signal Process..

[27]  Gerhard Fettweis,et al.  Performance of adaptive chip equalization for the WCDMA downlink in fast changing environments , 2002, IEEE Seventh International Symposium on Spread Spectrum Techniques and Applications,.

[28]  R. Lambert.  Multichannel blind deconvolution: FIR matrix algebra and separation of multipath mixtures , 1996 .

[29]  Néstor Parga,et al.  Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches , 1997, Neural Computation.

[30]  Fathi M. Salem,et al.  Blind Source Recovery: A State-Space Formulation , 2003 .

[31]  Khurram Waheed,et al.  Cascaded structures for blind source recovery , 2002, The 2002 45th Midwest Symposium on Circuits and Systems, 2002. MWSCAS-2002..

[32]  Fathi M. Salem,et al.  New hyperbolic source density models for blind source recovery score functions , 2003, Proceedings of the 2003 International Symposium on Circuits and Systems, 2003. ISCAS '03..

[33]  Fathi M. Salem,et al.  Algebraic independent component analysis , 2003, IEEE International Conference on Robotics, Intelligent Systems and Signal Processing, 2003. Proceedings. 2003.

[34]  C. Fyfe,et al.  Generalised independent component analysis through unsupervised learning with emergent Bussgang properties , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[35]  Fathi M. Salem,et al.  Algebraic independent component analysis: an approach for separation of overcomplete speech mixtures , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[36]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[37]  Andrzej Cichocki,et al.  Relationships between instantaneous blind source separation and multichannel blind deconvolution , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[38]  Fathi M. A. Salam,et al.  Voice extraction by on-line signal separation and recovery , 1999 .

[39]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[40]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[41]  Fathi M. Salem,et al.  Blind Source Recovery: A Framework in the State Space , 2003, J. Mach. Learn. Res..

[42]  Te-Won Lee,et al.  Blind Separation of Delayed and Convolved Sources , 1996, NIPS.

[43]  E. Polak,et al.  System Theory , 1963 .

[44]  Wing C. Kwong,et al.  Optical CDMA-based ATM switches supporting equal-bit-rate and variable-bit-rate communication services , 1998, IEEE GLOBECOM 1998 (Cat. NO. 98CH36250).

[45]  Fathi M. A. Salam,et al.  Algorithms for blind signal separation and recovery in static and dynamic environments , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[46]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[47]  Fabian J. Theis,et al.  Geometric overcomplete ICA , 2002, ESANN.

[48]  Antti Toskala,et al.  WCDMA for UMTS: Radio Access for Third Generation Mobile Communications , 2000 .

[49]  Fathi M. A. Salam,et al.  Separation of mixed signals in dynamic environments: formulation and some implementation , 1994, Proceedings of 1994 37th Midwest Symposium on Circuits and Systems.

[50]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[51]  H. Elders-Boll,et al.  Performance evaluation of advanced receivers for WCDMA downlink detection , 2002, The 5th International Symposium on Wireless Personal Multimedia Communications.

[52]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[53]  Bruno O. Shubert,et al.  Random variables and stochastic processes , 1979 .

[54]  Fathi M. A. Salam,et al.  An adaptive network for blind separation of independent signals , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[55]  H. Vincent Poor,et al.  Subspace methods for blind joint channel estimation and multiuser detection in CDMA systems , 2000, Wirel. Networks.

[56]  Fathi M. Salem,et al.  Adaptive RAKE-blind source recovery algorithms for 3GPP UMTS/WCDMA downlink receivers , 2003, IEEE International Conference on Robotics, Intelligent Systems and Signal Processing, 2003. Proceedings. 2003.

[57]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[58]  Frank L. Lewis,et al.  Optimal Control , 1986 .

[59]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[60]  Fathi M. Salem,et al.  ALGEBRAIC OVERCOMPLETE INDEPENDENT COMPONENT ANALYSIS , 2003 .

[61]  Noboru Murata,et al.  A stable and robust ICA algorithm based on t-distribution and generalized Gaussian distribution models , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[62]  Derong Liu,et al.  Qualitative Analysis and Synthesis of Recurrent Neural Networks , 2002 .

[63]  Mark A. Girolami,et al.  An Alternative Perspective on Adaptive Independent Component Analysis Algorithms , 1998, Neural Computation.

[64]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[65]  Reinhold Orglmeister,et al.  Blind source separation of real world signals , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[66]  Khurram Waheed,et al.  State space blind source recovery for mixtures of multiple source distributions , 2002, 2002 IEEE International Symposium on Circuits and Systems. Proceedings (Cat. No.02CH37353).

[67]  M. Kendall,et al.  The advanced theory of statistics , 1945 .

[68]  Tapani Ristaniemi,et al.  Blind separation of convolved mixtures for CDMA systems , 2000, 2000 10th European Signal Processing Conference.

[69]  Fathi M. A. Salam,et al.  Voice output extraction by signal separation , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[70]  Khurram Waheed,et al.  State Space Feedforward and Feedback Structures for Blind Source Recovery , 2001 .

[71]  Thomas P. von Hoff,et al.  Blind separation of signals with mixed kurtosis signs using threshold activation functions , 2001, IEEE Trans. Neural Networks.

[72]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[73]  Shun-ichi Amari,et al.  Neural Network Models for Blind Separation of Time Delayed and Convolved Signals , 1997 .

[74]  Fathi M. A. Salam,et al.  The state space framework for blind dynamic signal extraction and recovery , 1999, ISCAS'99. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems VLSI (Cat. No.99CH36349).

[75]  E. Oja,et al.  Independent Component Analysis , 2013 .

[76]  Aapo Hyvärinen,et al.  A Fast Fixed-Point Algorithm for Independent Component Analysis , 1997, Neural Computation.

[77]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[78]  S. Kullback,et al.  Topics in statistical information theory , 1987 .

[79]  Fathi M. A. Salam,et al.  Blind source recovery: algorithms for static and dynamic environments , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[80]  A. Cichocki,et al.  Blind Deconvolution of Dynamical Systems:A State-Space Approach , 2000 .

[81]  Sergio Verdú,et al.  Minimum probability of error for asynchronous Gaussian multiple-access channels , 1986, IEEE Trans. Inf. Theory.

[82]  I︠a︡. Z. T︠S︡ypkin,et al.  Foundations of the theory of learning systems , 1973 .

[83]  Russell H. Lambert,et al.  Blind separation of multiple speakers in a multipath environment , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[84]  Khurram Waheed,et al.  Blind source recovery: some implementation and performance issues , 2001, Proceedings of the 44th IEEE 2001 Midwest Symposium on Circuits and Systems. MWSCAS 2001 (Cat. No.01CH37257).

[85]  Shun-ichi Amari,et al.  Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient , 1996, NIPS.

[86]  Mark Girolami,et al.  Extraction of independent signal sources using a deflationary exploratory projection pursuit network , 1997 .

[87]  Fathi M. A. Salam,et al.  Real time separation of audio signals using digital signal processors , 1997, Proceedings of 40th Midwest Symposium on Circuits and Systems. Dedicated to the Memory of Professor Mac Van Valkenburg.

[88]  Andrzej Cichocki,et al.  Robust learning algorithm for blind separation of signals , 1994 .

[89]  S.C. Douglas,et al.  Multichannel blind deconvolution and equalization using the natural gradient , 1997, First IEEE Signal Processing Workshop on Signal Processing Advances in Wireless Communications.

[90]  J. Lilleberg,et al.  Downlink channel decorrelation in CDMA systems with long codes , 1999, 1999 IEEE 49th Vehicular Technology Conference (Cat. No.99CH36363).