The great success of convolutional neural networks (CNNs) in image classification benefits from the powerful feature learning abilities of their architectures. However, arbitrarily constructing these architectures is very costly in terms of manpower and computation resources. Recently, particle swarm optimization (PSO), as a promising evolutionary computation (EC) method, has been used to automatically search for the CNN architectures and achieved encouraging results on image classification, but these existing methods are not well-designed and/or do not have good search capabilities. In this paper, a flexible variable-length PSO algorithm is proposed for automated CNN architecture design for image classification tasks. Particularly, an improved encoding scheme is proposed to truly break the fixed-length representation constraint of PSO and encode the parameters in a more meaningful way. In addition, a novel velocity and position updating approach is developed to update variable-length particles. Experiments on four benchmark datasets are carried out to confirm the superiority of the proposed algorithm. It is shown that the proposed method leads to better results comparing to the state-of-the-art algorithms in terms of classification performance, parameter size and computational complexity.