Neonatal encephalopathy (NE) is the clinical manifestation of disordered neonatal brain function. As a leading cause of neonatal mortality and lifetime neurological disability, NE affects 2 to 6 infants per 1000 live births worldwide. It has been recognized that early intervention can potentially prevent NE or reduce the severity of negative consequences (e.g., cerebral palsy and epilepsy) caused by NE. However, most studies focus on predicting NE several hours before the NE event occurs (e.g., right after childbirth), which does not provide sufficient time to institute effective interventions. A prior study leveraged regularized logistic regression models along with electronic health records (EHRs) to predict NE before childbirth. The models were trained and tested on balanced data (equal number of cases and controls) and achieved a good performance. However, the predictive performance of these models was poor on imbalanced dataset with a prevalence of 0.2% to 0.6%. We hypothesize that deep learning along with word embedding can capture temporal patterns existing in EHRs, which can significantly improve the prediction of NE on imbalanced data. In this paper, we introduce a framework that incorporates deep learning, word embedding, and EHR data to predict NE before childbirth. The framework is evaluated with 104 NE and 31,054 non-NE newborns in a large academic medical center. We design 12 predictive models and the best model achieve an AUC of 0.93.