Azimuthal inhomogeneity of turbulence structure and its impact on intermittent particle transport in linear magnetized plasmas