A Deeper Look at FFT and Winograd Convolutions

Since convolutional layers are computationally expensive and dominate the total execution time of modern deep ConvNets [13, 16, 18, 19], many efforts have been made to improve the performance of the convolutional primitives for CPUs [1, 7, 20, 25, 27], GPUs [4, 8, 15, 21] or both [26]. Initially, several approaches using FFT–based convolutions were proposed [15, 21, 25, 26]. Recent work by Lavin et al. on Winograd–based convolutions [14] demonstrated a great speedup, which shifted the focus from FFT–based to Winograd–based implementations, as it became widely accepted that the Winograd–based approach provides greater reduction in the number of operations required by the algorithm, especially for small kernels (e.g. 3 × 3). A well optimized manycore CPU implementation [3, 12] of the Winograd approach can improve the performances by more than 3X. The main reduction in operations in the Winograd method, compared to FFT, comes from the fact that it works with real numbers. However, due to its numerical instability, the Winograd method can only use small tile (transform) sizes [7, 14, 22], which result in a larger amount of required data movement to and from memory. In contrast, the FFT–based method does not suffer from such instability, thus larger tile sizes can be used, which can partially reduce the number of required operations and greatly reduce the amount of data movements; these savings can, in certain cases, offset the increase in the number of operations due to complex arithmetic. These observations raise the question, under what conditions the Winograd-based approach performs better than the FFT– based approach and vice versa, and how to compare the two approaches.