The performance of deep neural networks is often attributed to their automated, task-related feature construction. It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples. Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error. For several flatness measures, this correlation has been empirically validated. However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization due to a lack of invariance with respect to reparameterizations. We propose a natural modification of existing flatness measures that results in invariance to reparameterization.
[1]
Razvan Pascanu,et al.
Sharp Minima Can Generalize For Deep Nets
,
2017,
ICML.
[2]
Jürgen Schmidhuber,et al.
Flat Minima
,
1997,
Neural Computation.
[3]
Shankar Krishnan,et al.
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
,
2019,
ICML.
[4]
Jorge Nocedal,et al.
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
,
2016,
ICLR.
[5]
Trac D. Tran,et al.
A Scale Invariant Flatness Measure for Deep Network Minima
,
2019,
ArXiv.
[6]
Huan Wang,et al.
Identifying Generalization Properties in Neural Networks
,
2018,
ArXiv.
[7]
Jürgen Schmidhuber,et al.
Simplifying Neural Nets by Discovering Flat Minima
,
1994,
NIPS.
[8]
Jascha Sohl-Dickstein,et al.
Sensitivity and Generalization in Neural Networks: an Empirical Study
,
2018,
ICLR.