Comparative study of nonideal beam effects in high gain harmonic generation and self-seeded free electron lasers