Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation