The one-particle spectral function and its orbital composition are investigated in a three-orbital model for the undoped parent compounds of the iron-based superconductors. In the realistic parameter regime, where results fit experimental data best, it is observed that the magnetization in the $xz$ and $yz$ orbitals are markedly different and the Fermi surface presents mostly $xz$ character, as recently observed in photoemission experiments [T. Shimojima et al., Phys. Rev. Lett. 104, 057002 (2010)]. Since the ferro-orbital order in this regime is at most a few percent, these results are mainly driven by the magnetic order. An analogous analysis for a five-orbital model leads to similar conclusions.