Mode separation of magnetotelluric responses in three-dimensional environments