Recent developments in machine learning and signal processing have resulted in many new techniques that are able to effectively capture the intrinsic yet complex properties of hyperspectral imagery (HSI). Tasks ranging from anomaly detection to classification can now be solved by taking advantage of very efficient algorithms which have their roots in representation theory and computational approximation. Time–frequency methods are one example of such techniques. They provide means to analyze and extract the spectral content from data. On the other hand, hierarchical methods such as neural networks (NNs) incorporate spatial information across scales and model multiple levels of dependencies between spectral features. Both of these approaches have recently been proven to provide significant advances in the spectral-spatial classification of HSI. The 3-D Fourier scattering transform, which is introduced in this article, is an amalgamation of time–frequency representations with NN architectures. It leverages the benefits provided by the short-time Fourier transform with the numerical efficiency of deep learning network structures. We test the proposed method on several standard hyperspectral data sets, and we present results that indicate that the 3-D Fourier scattering transform is highly effective at representing spectral content when compared with other state-of-the-art spectral-spatial classification methods.