Voxel-based methods have been widely used in point cloud 3D object detection. These methods usually transform points into voxels while suffering from information loss during point cloud voxelization. To address this problem, we propose a novel one-stage Voxelization Information Compensation Network (VIC-Net), which has the ability of loss-free feature extraction. The whole framework consists of a point branch for geometry detail extraction and a voxel branch for efficient proposals generation. Firstly, PointNet++ is adopted to efficiently encode geometry structure features from the raw point clouds. Then based on the encoded point features, two Point2Voxel (P2V) feature fusion modules are proposed to fuse point features with a voxel backbone, including Local P2V and Multi-Scale P2V. The P2V modules respectively integrate local detail features and multi-scale semantic contexts into a sparse voxel backbone. Thirdly, an auxiliary reconstruction loss is employed on the point branch to explicitly guide the point backbone to be aware of real geometry structures. In addition, we extend VIC-Net to a two-stage approach, namely VIC-RCNN, which further utilizes the fine geometry features to refine object locations. Experiments on the KITTI dataset demonstrate that our proposed VIC-Net outperforms other onestage methods and our two-stage method VIC-RCNN achieves new state-of-the-art performance.