Destructive effect of fluctuations on the performance of a Brownian gyrator