Bessel functions and representation theory. I

Abstract In this paper a general theory of operator-valued Bessel functions is presented. These functions arise naturally in representation theory in the context of metaplectic representations, discrete series, and limits of discrete series for certain semi-simple Lie groups. In general, Bessel functions J λ are associated to the action by automorphisms of a compact group U on a locally compact abelian group X , and are indexed by the irreducible representations λ of U that appear in the primary decomposition of the regular representation of U on L 2 ( X ). Then on the λ-primary constituent of L 2 ( X ), the Fourier transform is described by the Hankel transform corresponding to J λ . More detailed information is available in the case in which ( U , X ) is an orthogonal transformation group which possesses a system of polar coordinates. In particular, when X = F k × n , F a real finite-dimensional division algebra, with k ⩾ 2 n and O ( k , F ), the representations λ of U are induced in a certain sense from representations π of GL( n , F ). This leads to a characterization of J λ as a reduced Bessel function defined on the component of 1 in GL( n , F ) and to the connection between metaplectic representations and holomorphic discrete series for the group of biholomorphic automorphisms of the Siegel upper half-plane in the complexification of F n × n .