Sampling operation is a bottleneck of the hierarchical point cloud learning. Existing learnable sampling methods generate a “soft” virtual subset in the training phase, thus distorting the original underlying shape and losing 3D geometric information. In this paper, we propose a novel end-to-end discrete sampling method, named Straight Sampling, to output a “hard” authentic subset with the assistance of Straight Through Estimator. Equipped with Straight Sampling, a hierarchical architecture is developed to learn an effective representation. By grouping and pooling the sampled points in 3D Euclidean space, the network benefits from semantic features as well as 3D geometric information to achieve state-of-the-art performance.