A theoretical argument for complex-valued convolutional networks

A complex-valued convolutional network (convnet) implements the repeated application of the following composition of three operations, recursively applying the composition to an input vector of nonnegative real numbers: (1) convolution with several complex-valued vectors followed by (2) taking the absolute value of every entry of the resulting vectors followed by (3) local averaging. For processing real-valued random vectors, complex-valued convnets can be viewed as “data-driven multiscale windowed power spectra,” “data-driven multiscale windowed absolute spectra,” “data-driven multiwavelet absolute values,” or (in their most general configuration) “data-driven nonlinear multiwavelet packets.” Indeed, complex-valued convnets can calculate multiscale windowed spectra when the convnet filters are windowed complex-valued exponentials. Standard real-valued convnets, using rectified linear units (ReLUs), sigmoidal (for example, logistic or tanh) nonlinearities, max. pooling, etc., do not obviously exhibit the same exact correspondence with data-driven wavelets (whereas for complex-valued convnets, the correspondence is much more than just a vague analogy). This note develops “data-driven multiscale windowed spectra” for certain stochastic processes that are common in the modeling of time series (such as audio) and natural images (including patterns and textures). We motivate the construction of such multiscale spectra in the form of “local averages of multiwavelet absolute values” or (in the most general configuration) “nonlinear multiwavelet packets” and connect these to certain “complex-valued convolutional networks.” A textbook treatment of all concepts and terms used above and below is given by [12]. Further information is available in the original work of [7], [15], [5], [4], [19], [16], [9], [20], and [18], for example. The work of [8], [13], [17], [2], and [3] also develops complex-valued convolutional networks (convnets). Renormalization group theory and its connection to convnets is discussed by [14]; this connection is incredibly insightful, though we leave further discussion to the cited work. Our exposition relies on nothing but the basic signal processing treated by [12]. For simplicity, we first limit consideration to the special case of a doubly infinite sequence of nonnegative random variables Xk, where k ranges over the integers. This input data will be the result of convolving an unmeasured independent and identically distributed (i.i.d.) sequence Zk, where k ranges over the integers, with an unknown sequence of real numbers fk, where k ranges over the integers (this latter sequence is known as a “filter,” whereas the i.i.d. sequence is known as “white noise”):