Accurate origin-destination traffic prediction (ODTP) is a persistent problem in network management. Due to the structural nature of networks, the spatial correlations are critical for effective perdition. In this paper, the ODTP problem is built onto the graph domain by mapping OD traffic into graph-structured data involving the topology information. Moreover, due to the flow characteristics of OD traffic, the complex spatial-temporal (ST) correlations should not be limited at one-hop neighbors or consecutive time steps. To benefit from this observation, we propose a novel graph learning based method, called multi-scale spatial-temporal graph neural network (MSTNN). In MSTNN, spatial and temporal extractors are designed to capture multi-scale spatial and temporal correlations. Specifically, the spatial extractor employs the graph attention mechanism to capture the time-varying spatial correlations of nodes and their multi-hop neighbors. In the temporal extractor, we design gated dilated convolution layers with different dilation factors, each of which represents a different granularity of time, to exploit the multi-scale temporal correlations in both adjacent and non-adjacent time steps. After cascading the spatial and temporal extractors, the multi-scale ST correlations are weighted fused. Simulations on two real-world datasets show that MSTNN outperforms existing approaches that work well in various prediction tasks.