Energy and bandwidth efficiency optimization of quantum-enabled optical communication channels