Heterogeneous structural responses recovery based on multi-modal deep learning