4D thoracic images constructed from free-breathing 2D slice acquisitions based on dynamic magnetic resonance imaging (dMRI) provide clinicians the capability of examining the dynamic function of the left and right lungs, left and right hemi-diaphragms, and left and right chest wall separately for thoracic insufficiency syndrome (TIS) treatment [1]. There are two shortcomings of the existing 4D construction methods [2]: a) the respiratory phase corresponding to end expiration (EE) and end inspiration (EI) need to be manually identified in the dMRI sequence; b) abnormal breathing signals due to non-tidal breathing cannot be detected automatically which affects the construction process. Since the typical 2D dynamic MRI acquisition contains ~3000 slices per patient, handling these tasks manually is very labor intensive. In this study, we propose a deep-learning-based framework for addressing both problems via convolutional neural networks (CNNs) [3] and Long Short-Term Memory (LSTM) [4] models. A CNN is used to extract the motion characteristics from the respiratory dMRI sequences to automatically identify contiguous sequences of slices representing exhalation and inhalation processes. EE and EI annotations are subsequently completed by comparing the changes in the direction of motion of the diaphragm. A LSTM network is used for detecting abnormal respiratory signals by exploiting the non-uniform motion feature sequence of abnormal breathing motions. Experimental results show the mean error of labeling EE and EI is ~0.3 dMRI time point unit (much less than one time point). The accuracy of abnormal cycle detection reaches 80.0%. The proposed approach achieves results highly comparable to manual labeling in accuracy but with close to full automation of the whole process. The framework proposed here can be readily adapted to other modalities and dynamic imaging applications.