Implicit dynamic regularization in deep networks

Square loss has been observed to perform well in classification tasks. However, a theoretical justification is lacking, unlike the cross-entropy case. Here we discuss several observations on the dynamics of gradient flow under the square loss in ReLU networks. We show how convergence to a solution with the absolute minimum norm is expected when normalization techniques such as Batch Normalization (BN) or Weight Normalization (WN) are used together with zero-initial conditions on the layers weights. This is similar to the behavior of linear degenerate networks under gradient descent (GD), though the reason for zero-initial conditions is different. The main property of the minimizer that bounds its expected error is its norm: we prove that among all the interpolating solutions, the ones associated with smaller Frobenius norms of the weight matrices have better margin and better bounds on the expected classification error. The theory yields several predictions, including role of BN and weight decay, aspects of Papyan, Han and Donoho’s Neural Collapse and the constraints induced by BN on the network weights. This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. Implicit dynamic regularization in deep networks Tomaso Poggio and Qianli Liao Abstract Square loss has been observed to perform well in classification tasks. However, a theoretical justification is lacking, unlike the cross-entropy [1] case for which an asymptotic analysis has been proposed (see [2] and [3] and references therein). Here we discuss several observations on the dynamics of gradient flow under the square loss in ReLU networks. We show how convergence to a solution with the absolute minimum norm is expected when normalization techniques such as Batch Normalization[4] (BN) or Weight Normalization[5] (WN) are used together with zero-initial conditions on the layers weights. This is similar to the behavior of linear degenerate networks under gradient descent (GD), though the reason for zero-initial conditions is different. The main property of the minimizer that bounds its expected error is its norm: we prove that among all the interpolating solutions, the ones associated with smaller Frobenius norms of the weight matrices have better margin and better bounds on the expected classification error. The theory yields several predictions, including the joint role of BN and weight decay, aspects of Papyan, Han and Donoho’s Neural Collapse and the constraints induced by BN on the network weights.Square loss has been observed to perform well in classification tasks. However, a theoretical justification is lacking, unlike the cross-entropy [1] case for which an asymptotic analysis has been proposed (see [2] and [3] and references therein). Here we discuss several observations on the dynamics of gradient flow under the square loss in ReLU networks. We show how convergence to a solution with the absolute minimum norm is expected when normalization techniques such as Batch Normalization[4] (BN) or Weight Normalization[5] (WN) are used together with zero-initial conditions on the layers weights. This is similar to the behavior of linear degenerate networks under gradient descent (GD), though the reason for zero-initial conditions is different. The main property of the minimizer that bounds its expected error is its norm: we prove that among all the interpolating solutions, the ones associated with smaller Frobenius norms of the weight matrices have better margin and better bounds on the expected classification error. The theory yields several predictions, including the joint role of BN and weight decay, aspects of Papyan, Han and Donoho’s Neural Collapse and the constraints induced by BN on the network weights.