Diminishing returns and tradeoffs constrain the laboratory optimization of an enzyme