This article proposes an automatic approach for segmenting inclusion defects in composite materials inspected by lock-in thermography (LT) in a heat source invariant way. In the proposed pipeline, temperature maps are preliminarily processed to enhance the contrast between sound and defective areas, reduce the acquisition noise, and mitigate the effect of nonuniform heat excitation. Then, the enhanced sequences are grouped in highly local sets, the input of a convolution residual neural network, which segments the defects focusing on the temporal evolution of the temperature. Experiments have been run on a dataset acquired from a CFRP specimen tested under five different heat source configurations. Here, a brief study of signal variations associated with the heat source configuration is presented. Then, an ablation study has been presented to compare different architectures of increasing complexity. The best results are achieved by using the proposed preliminary processing, the residual layers, and an input set of temperature signals of size $3 \times 3$ . This network can reach a global intersection over union (IOU) of 0.78, while defects are segmented with an IOU value of 0.58. In addition, the balanced segmentation accuracy of the test reaches 94.78%, with a maximum recall of the defective class of 91.41%. Further cross validation changing the source configurations of the training and test sets produces comparable results of IOU and balanced accuracy, which are on average equal to 0.71 and 84.02%, respectively. A final evaluation of the trained network is performed on a new dataset augmented through a vertical stretch. Results are still comparable with an average IOU and balanced accuracy of 0.74 and 87.15%, respectively. These results prove the capability of the method to segment inclusion defects of any shape without being affected by the configuration of the heating sources used in the LT inspections.