Human age estimation from face image has drawn increasing research attention due to its many meaningful applications such as demographics analysis and surveillance monitoring. However, most existing methods directly extract age-specific features for age estimation and ignore age-related gender information. In this paper, we propose a simplified deep learning network for age estimation by simultaneously learning aging and potential gender features. Specifically, we first learn the potential gender information from face images. Then, we employ a two-stream convolutional neural network to simultaneously learn and concatenate the aging and gender latent appearance features. Third, we feed the multi-type features into a compact convolution network, named AgeNetwork, to further learn the age-specific features. Finally, we use a deep regression function to estimate the detailed ages. Extensive experimental results demonstrate the promising effectiveness and efficiency of our proposed method in comparison with state-of-the-arts.