Universal Complexity Bounds Based on Value Iteration and Application to Entropy Games