Optimal minimax bounds for time and ensemble averages for the incompressible Navier-Stokes equations