Virtual network embedding (VNE) that instantiates virtualized networks on a substrate infrastructure, is one of the key research problems for network virtualization. Most existing VNE approaches, however, focus on the current virtual network request (VNR) and treat all VNRs equally, which disregard the long-term impact and waste many resources on the process of embedding infeasible VNRs (i.e., VNRs that cannot be embedded completely). To address these problems, a proactive virtual network embedding algorithm based on hierarchical reinforcement learning, VNE-HRL, is proposed in this paper. Within our framework, the VNE task is performed by a two-level agent that considers both the long-term impact of a VNR and the short-term effect of an embedding action. For each processing, a high-level agent aims to select a currently feasible VNR with the maximum long-term reward from a window-based batch, and a low-level agent is assigned to embed the selected VNR on a substrate infrastructure by performing a series of embedding actions. Extensive simulation results indicate that our algorithm best performance on most metrics compared with existing state-of-the-art solutions, with up to 9.92% and 33.03% improvement on acceptance ratio and average revenue.