Restructured Audio Encoder for Improved Computational Efficiency