The roles of majorization and generalized triangular decomposition in communication and signal processing

The main contribution of this thesis is toward the use of majorization and generalized triangular decomposition (GTD) to the theory and many applications of signal processing. In particular, the focus is on developing new signal processing methods based on these mathematical tools for digital communication, data compression, and filter bank design. The first part of the thesis focuses on transceiver design for multiple-input multiple-output (MIMO) communications. The first problem considered is the joint optimization of transceivers with linear precoders, decision feedback equalizers (DFEs), and bit allocation schemes for frequency flat MIMO channels. We show that the generalized triangular decomposition offers an optimal family of solutions to this problem. This general framework incorporates many existing designs, such as the optimal linear transceiver, the ZF-VBLAST system, and the geometric mean decomposition (GMD) transceiver, as special cases. It also predicts many novel optimal solutions that have not been observed before. We also discuss the use of each of these theoretical solutions under practical considerations. In addition to total power constraints, we also consider the transceiver optimization under individual power constraints and other linear constraints on the transmitting covariance matrix, which includes a more realistic individual power constraint on each antenna. We show the use of semidefinite programming (SDP), and the theory of majorization again provides a general framework for optimizing the linear transceivers as well as the DFE transceivers. The transceiver design for frequency selective MIMO channels is then considered. Block diagonal GMD (BD-GMD), which is a special instance of GTD with block diagonal structure in one of the semi-unitary matrices, is used to design transceivers that have many desirable properties in both performance and computation. The second part of the thesis focuses on signal processing algorithms for data compressions and filter bank designs. We revisit the classical transform coding problem (for optimizing the theoretical coding gain in the high bit rate regime) from the view point of GTD and majorization theory. A general family of optimal transform coders is introduced based on GTD. This family includes the Karhunen-Loeve transform (KLT), and the prediction-based lower triangular transform (PLT) as special cases. The coding gain of the entire family, with optimal bit allocation, is maximized and equal to those of the KLT and the PLT. Other special cases of the GTD-TC are the GMD (geometric mean decomposition) and the BID (bidiagonal transform). The GMD in particular has the property that the optimum bit allocation is a uniform allocation. We also propose using dither quantization in the GMD transform coder. Under the uniform bit loading scheme, it is shown that the proposed dithered GMD transform coders perform significantly better than the original GMD coder in the low rate regime. Another important signal processing problem, namely the filter bank optimization based on the knowledge of input signal statistics, is then considered. GTD and the theory of majorization are again used to give a new look to this classical problem. We propose GTD filter banks as subband coders for optimizing the theoretical coding gain. The orthonormal GTD filter bank and the biorthogonal GTD filter bank are discussed in detail. We show that in both cases there are two fundamental properties in the optimal solutions, namely, total decorrelation and spectrum equalization. The optimal solutions can be obtained by performing the frequency dependent GTD on the Cholesky factor of the input power spectrum density matrices. We also show that in both theory and numerical simulations, the optimal GTD subband coders have superior performance than optimal traditional subband coders. In addition, the uniform bit loading scheme can be used in the optimal biorthogonal GTD coders with no loss of optimality. This solves the granularity problem in the conventional optimum bit loading formula. The use of the GTD filter banks in frequency selective MIMO communication systems is also discussed. Finally, the connection between the GTD filter bank and the traditional filter bank is clearly indicated. (Abstract shortened by UMI.)