The search for new materials, based on computational screening, relies on methods that accurately predict, in an automatic manner, total energy, atomic-scale geometries, and other fundamental characteristics of materials. Many technologically important material properties directly stem from the electronic structure of a material, but the usual workhorse for total energies, namely density-functional theory, is plagued by fundamental shortcomings and errors from approximate exchange-correlation functionals in its prediction of the electronic structure. At variance, the $GW$ method is currently the state-of-the-art {\em ab initio} approach for accurate electronic structure. It is mostly used to perturbatively correct density-functional theory results, but is however computationally demanding and also requires expert knowledge to give accurate results. Accordingly, it is not presently used in high-throughput screening: fully automatized algorithms for setting up the calculations and determining convergence are lacking. In this work we develop such a method and, as a first application, use it to validate the accuracy of $G_0W_0$ using the PBE starting point, and the Godby-Needs plasmon pole model ($G_0W_0^\textrm{GN}$@PBE), on a set of about 80 solids. The results of the automatic convergence study utilized provides valuable insights. Indeed, we find correlations between computational parameters that can be used to further improve the automatization of $GW$ calculations. Moreover, we find that $G_0W_0^\textrm{GN}$@PBE shows a correlation between the PBE and the $G_0W_0^\textrm{GN}$@PBE gaps that is much stronger than that between $GW$ and experimental gaps. However, the $G_0W_0^\textrm{GN}$@PBE gaps still describe the experimental gaps more accurately than a linear model based on the PBE gaps.