Traditional digital holographic imaging algorithms need multiple iterations to obtain focused reconstructed image, which is time-consuming. In terms of phase retrieval, there is also the problem of phase compensation in addition to focusing task. Here, a new method is proposed for fast digital focus, where we use U-type convolutional neural network (U-net) to recover the original phase of microscopic samples. Generated data sets are used to simulate different degrees of defocused image, and verify that the U-net can restore the original phase to a great extent and realize phase compensation at the same time. We apply this method in the construction of real-time off-axis digital holographic microscope and obtain great breakthroughs in imaging speed.