Conditional Conformer: Improving Speaker Modulation For Single And Multi-User Speech Enhancement