Solitonic symmetry beyond homotopy: invertibility from bordism and non-invertibility from TQFT