The geometric β-function in curved space-time under operator regularization