Organization of Energetic Particles by the Solar Wind Structure During the Declining to Minimum Phase of Solar Cycle 23