Structure theory for ensemble controllability, observability, and duality