Scalability of optical devices is a major challenge for quantum optics and quantum cryptography fields. However, non-linear optical processes such as second harmonic generation (SHG) and parametric-down conversion become very inefficient when the active medium is reduced to the nanoscale. Enhancement strategies are therefore mandatory. Here, we first investigate the role of plasmonic resonances in single aluminum nanostructures allowing doubly resonant and mode-matched conditions. We show that the SHG rate can be 36-fold enhanced compared to non-resonant structures. We further infer the origin of the nonlinearity by quantitatively comparing simulated and measured SHG maps obtained by scanning the antennas under a tightly focused beam. The SHG response of a KTP nano-crystal and its modification by the proximity of a plasmonics antenna can then be confidently modeled. We show that the harmonic photon production yield is comparable for a bare nano-crystal and a doubly resonant aluminum antenna, despite the centro-symmetric nature of the latter. Combining the nonlinearity of the KTP crystal and the field enhancements provided by the plasmonic structure at both fundamental and harmonic frequency, we demonstrate that the SHG signal can be magnified by more than two orders of magnitude. The anticipated efficiency of the hybrid nonlinear plasmonic structures is compared to experiments performed at the single structure level, emphasizing the crucial role of the nanocrystal orientation.