Essentially doubly stochastic matrices