The deficiencies of existing polyp detection methods remain: i) They primarily depend on the manually extracted features and require considerable amounts of preprocessing. ii) Most traditional methods cannot specify the location of the polyps in colonoscopy images, especially for the polyps with variable size. In order to derive the improvement and lift the accuracy, we propose a novel and scalable detection algorithm based on deep neural networks-an improved Faster Regionbased Convolutional neural networks (Faster R-CNN)-by increasing the fusion of feature maps at different levels. It can be employed to detect and locate polyps, and even achieve a multi-object task for polyps in the future. The experimental consequences demonstrate that the best version among improved algorithms achieves 97.13% accuracy on the CVC-ClinicDB database, overtaking the previous methods.