Small targets detection is one of the key technical problems in these fields such as maritime surveillance and disaster rescue. In order to find the target as early as possible, the small target detection algorithm based on sea-sky line extraction is generally adopted, and the accurate extraction of the sea-sky line becomes the basis of the small target detection in the maritime. For the panoramic sea images acquired by the catadioptric panoramic vision system, the sea-sky line extraction algorithm is studied in this paper. The specificity of panoramic vision system creating images makes sea-sky line in a panoramic image an approximate circle. A detection algorithm based on gradient direction is proposed to extract accurately the circular sea-sky line. Experimental results show the efficiency and superiority of the proposed method in this paper.