An ultra-high sensitivity refractive index sensor is proposed and designed by integrating plasmonic Mach-Zehnder interferometer and the optical fiber structure together. The configuration, the numerical simulations and the performance of the proposed structure have been analyzed and discussed by the finite element method. At the working wavelength of 1.55 μm, the designed sensor is optimized to obtain the sensing sensitivity 20141.7 nm/RIU. Furthermore, it is found that with the increase of the working wavelength, the sensitivity will also increase, thus the proposed sensor has the potential to work in the near-infrared band. This sensor not only shows a high sensitivity to refractive index detection, but also is highly applicable in the detection of liquid concentration, gas concentration or temperature.