Saliency detection is an important and challenging research topic due to the variety and complexity of the background and saliency regions. In this paper, we present a novel unsupervised saliency detection approach by exploiting the grouping and compactness characteristics of the high-level semantic features. First, for the high-level semantic feature, the elastic net based hypergraph model is adopted to discover the group structure relationships of salient regional points, and the calculation of the spatial distribution is constructed to detect the compactness of the saliency regions. Next, the grouping-based and compactness-based saliency maps are improved by a propagation algorithm. The propagation process uses an enhanced similarity matrix, which fuses the low-level deep feature and the high-level semantic feature through cross diffusion. Results on four benchmark datasets with pixel-wise accurate labeling demonstrate the effectiveness of the proposed method. Particularly, the proposed unsupervised method achieves competitive performance with deep learning-based methods.