Bouc's conjecture on $B$-groups