Constraining Sources of Ultra High Energy Cosmic Rays Using High Energy Observations with the Fermi Satellite