In this paper, we present a segmentation and classification method for thyroid follicular neoplasm based on the combination of the prior-based level set method and deep convolutional neural network (DCNN).The proposed method aims to discriminate thyroid follicular adenoma (TFA) and follicular thyroid carcinoma (FTC) in ultrasound images. From their appearance, these two kinds of tumors have almost similar shapes, sizes and contrasts. So it is difficult to distinguish them even by ultrasound specialists. Due to the complex background in thyroid ultrasound images, before distinguishing TFA and FTC, we need to segment the lesions from the whole image for each patient. The main challenge of the segmentation is that the images often have weak edges, heterogeneous regions, and heavy noise. The main issue of classification is that the accuracy will depend on the extracted features from the segmentation results. To solve these problems, we conduct the two tasks, i.e., the segmentation and classification, by a cascaded learning architecture. For the segmentation, in order to get more accurate results, we exploit the Res-U-net framework and the prior-based level set method to enhance their respective abilities. Then, the classification network is trained by sharing shallow layers of the segmentation network. By testing the proposed method on the real patient data, the results show that it is able to segment the lesion areas in thyroid ultrasound images with 92.06% for Dice score and distinguish TFA and FTC with 94.00% of classification accuracy.