Abstract Topology optimization offers great potential to design periodic structures with desired bandgap properties. This paper proposes a novel Subset Simulation (SS) based topology optimization framework by integrating SS with generative adversarial network (GAN). First, the topology optimization problem is reformulated as a rare event simulation problem in reliability analysis, where the optimal solutions are analogously the rare event samples close to failure. Then SS, which has been developed for efficient simulation of rare events in reliability analysis, is used to effectively find the optimal topologies. In each iteration of SS, to address the challenge of simulating samples from high-dimensional design space (stemming from discretization of the unit cell to represent different topologies), this paper proposes to use GANs to learn an implicit model for the underlying high-dimensional failure distribution based on existing failure samples (i.e., topologies with higher objective function values) from the previous iteration in SS, and then use the trained GANs to directly and efficiently generate failure samples (i.e., new promising topologies). Overall, the proposed SS-based and GAN-guided topology optimization algorithm can facilitate efficient topology optimization of periodic structures. The effectiveness and efficiency of the proposed approach are demonstrated through topology optimization of 2D periodic structures.