Controllable Pure Dephasing Pathways in Single Site-Controlled Pyramidal Quantum Dot – Nanocavity System