Cartoonization as a special type of artistic style transfer is a difficult image processing task. The current existing artistic style transfer methods cannot generate satisfactory cartoon-style images due to that artistic style images often have delicate strokes and rich hierarchical color changes while cartoon-style images have smooth surfaces without obvious color changes, and sharp edges. To this end, we propose a cartoon loss based generative adversarial network (CartoonLossGAN) for cartoonization. Particularly, we first reuse the encoder part of the discriminator to build a compact generative adversarial network (GAN) based cartoonization architecture. Then we propose a novel cartoon loss function for the architecture. It can imitate the process of sketching to learn the smooth surface of the cartoon image, and imitate the coloring process to learn the coloring of the cartoon image. Furthermore, we also propose an initialization strategy, which is used in the scenario of reusing the discriminator to make our model training easier and more stable. Extensive experimental results demonstrate that our proposed CartoonLossGAN can generate fantastic cartoon-style images, and outperforms four representative methods.