Is a Semi-Classical Theory of Gravity Viable?