Imperfect Quantum Photonic Neural Networks