In this paper, an operator-based nonlinear vibration control of a flexible arm experimental system using piezoelectric actuator with hysteresis is discussed. The piezoelectric actuator exhibits hysteretic effects in the output displacement responses, which can cause inaccuracy and undesirable oscillations, where Prandtl-Ishlinskii(PI) model is used to describe the hysteresis. In order to compensate the hysteretic effects, using the model, and based on the concept of Lipschiz operators and robust right coprime factorization, nonlinear vibration controllers are proposed to the experimental system. Further, a tracking operator design method is given to ensure the tracking performance of the considered system. Finally, simulation results and experimental results are presented to support the effectiveness of the proposed design method.