Turning a first order quantum phase transition continuous by fluctuations: general flow equations and application to the nematic transition in two-dimensional electron systems