Nonlocal gravity with worldline inversion symmetry