Lie group representations and harmonic polynomials of a matrix variable

The first part of this paper deals with problems concerning the symmetric algebra of complex-valued polynomial functions on the complex vector space of n by k matrices. In this context, a generalization of the socalled "classical separation of variables theorem" for the symmetric algebra is obtained. The second part is devoted to the study of certain linear representations, on the above linear space (the symmetric algebra) and its subspaces, of the complex general linear group of order k and of its subgroups, namely, the unitary group, and the real and complex special orthogonal groups. The results of the first part lead to generalizations of several well-known theorems in the theory of group representations. The above representation, of the real special orthogonal group, which arises from the right action of this group on the underlying vector space (of the symmetric algebra) of matrices, possesses interesting properties when restricted to the Stiefel manifold. The latter is defined as the orbit (under the action of the real special orthogonal group) of the n by k matrix formed by the first n row vectors of the canonical basis of the k-dimensional real Euclidean space. Thus the last part of this paper is involved with questions in harmonic analysis on this Stiefel manifold. In particular, an interesting orthogonal decomposition of the complex Hilbert space consisting of all square-integrable functions on the Stiefel manifold is also obtained. Introduction. Let Eo = Rn xk and E = Cn xk with k > n. Let G SO(k) and G = SO(k,C). The group GL(n, C) operates on E to the left; the group GL(k, C) and its subgroups U(k), G, and Go act linearly on E to the right. When n = 1 and k > 2 the following theorems are well known: THEOREM 0.1. The representations of U(k) and its complexification Received by the editors August 16, 1974. AMS (MOS) subject classifications (1970). Primary 13F20, 22E30, 22E45; Secondary 31C05, 33A45, 43A85.