Controlling multimode coupling by boundary-wave scattering