Machine learning related research in transient control has drawn considerable attention with the rapid increase in data measurement from power grids. Two key components, the control algorithm and system structure, work together to determine the control performance. The design of control laws, the selection of phase measurement units, the allocation of power resources, and the scheduling of communication topology in limited cyber-physical resources need to be considered. Many existing scheduling or planning schemes specialized for control structure are designed based on various linearized analytical models or the optimization of steady states. However, the transient dynamics of power grids are nonlinear and parts of these dynamics are usually unknown. Linearized analytical models cannot represent the transient dynamics of power grids with large disturbances. This article proposes a sparse neural network based reinforcement learning scheme to optimize the control system structure for the transient stability enhancement of power grids with energy storage systems. One adjustable group sparse weight matrix is introduced to formulate both control structure and actor–critic networks. This strategy enables the proposed scheme to simultaneously schedule the control system structure and design the control laws by online learning without solving any combinational optimization problems or requiring any linearized analytical models. The sufficient conditions of learning stability, control stability, and group sparsity are thoroughly studied by mathematical analysis. The proposed scheme is simulated on an IEEE 118-bus test system for verification. The simulation results confirm the feasibility, advantages, and adaptability of the proposed method.