Modular Invariance and the Fusion Algebra