Beyond Uniform Lipschitz Condition in Differentially Private Optimization