Preventing rapid energy loss from electron-hole pairs to phonons in graphene quantum dots