Digital Object Identifier: 10.1109/ TNNLS.2013.2247627 new adaptive dynamic programming approach was developed by integrating a reference network that provides an internal goal representation to help the systems learning and optimization. The reference network was built on top of the critic network to form a dual critic network design that contains detailed internal goal representation to help approximate the value function. This internal goal signal, working as the reinforcement signal for the critic network in the design, is adaptively generated by the reference network and can also be adjusted automatically. In this way, an alternative choice was provided rather than crafting the reinforcement signal manually from prior knowledge. Besides, the online action-dependent heuristic dynamic programming (ADHDP) design was employed and the detailed design of the dual critic network structure was presented. Lyapunov stability analysis was also presented to support the proposed structure from a theoretical point of view. Furthermore, a virtual reality platform was developed to demonstrate the real-time simulation of the approach under different disturbance situations. The overall adaptive learning performance has been tested on two tracking control benchmarks with a tracking filter.”