Fast inversion algorithms for a class of block structured matrices