Collective cooling and self-organization of atoms in a cavity