Groups with finitely many normalizers of non-periodic subgroups