Local inversion of matrices with sparse inverses