Structure Inducing Pre-Training

We present a theoretical analysis from first principles that establishes a novel connection between relational inductive bias of pre-training and fine-tuning performance while providing an extended view on general pre-training models. We further explore how existing pretraining methods impose relational inductive biases, finding that the vast majority of existing approaches focus almost exclusively on modelling relationships in an intra-sample manner, rather than a per-sample manner. We build upon these findings with simulations and empirical studies on standard benchmarks spanning 3 data modalities and 10 downstream tasks. These investigations validate our theoretical analyses, and provides a recipe to produce new pre-training methods which incorporate provably richer inductive biases than do existing methods in line with user specified relational graphs.