Semi-invariant Matrices over Finite Groups