Direct-driven electro-hydraulic systems have a wide range of applications owing to their advantages of energy-savings and relatively high control flexibilities in comparison with classic variable displacement pump-controlled hydraulic systems. However, the control accuracy is limited by the inherent nonlinear hydraulic dynamics. Additionally, the pump flow rate may become nonlinear at low pump speeds, causing large pressure-related flow deviations; thereby, limiting the improvement of motion control accuracy. Unfortunately, the pump flow nonlinearity has been ignored or oversimplified without effective modeling in most studies so far. To improve the control accuracy of direct-drive hydraulic systems, a high-precision control strategy must be designed to deal with the nonlinear characteristics and resolve the issue of nonlinear pump flow at low speeds. This article proposes an adaptive robust motion control strategy for a direct-driven electro-hydraulic system with adaptive pump flow rate model compensation. A backstepping integrated direct/indirect adaptive robust controller is designed to deal with the dynamic nonlinearities and uncertainties, which guarantees the stability of the entire hydraulic system. Furthermore, a parameterized polynomial fitting modeling strategy is proposed to accurately describe the nonlinear characteristics of the pump flow rate. Therefore, the uncertain parameters are adjusted in real-time, achieving satisfactory parameter estimations and model compensation for asymptotic motion tracking. Theoretical proof and comparative experiments demonstrate the advantages of the proposed control strategy with adaptive polynomial fitting model compensation for high-precision motion control.