Fermionic CFTs at large charge and large N