Non-linear duality invariant partially massless models?