In this paper, a new memristor chaotic system is designed based on Chua’s memristor chaotic system. To get the complete picture of the brain of a three-dimensional chaotic attractor, red and blue 3D glasses is used to observe the chaotic attractor, and using Lyapunov exponent spectrum, SE complexity and C0 complexity to analyze dynamical characteristics of new memristor chaotic system. The results illustrate that the chaotic state of the new memristor chaotic system is distributed over a large parameter range, which shows that the new memristor chaotic system is more suitable for image encryption applications. To verify the image encryption application of the new memristor chaotic system, a novel image encryption algorithm is designed based on the new memristor chaotic system and DNA variation. The security performances of the designed algorithm indicate that the proposed algorithm can effectively encrypt image and has better security performance.