Intrinsic normalized emittance growth in laser-driven electron accelerators