Chemical space is impractically large and conventional structure-based virtual screening techniques cannot be used to simply search through the entire space to discover effective bioactive molecules. To address this shortcoming we propose a generative adversarial network to generate, rather than search, diverse three-dimensional ligand shapes complementary to the pocket. Furthermore, we show that the generated molecule shapes can be decoded using a shape-captioning network into a sequence of SMILES enabling directly structure-based de novo drug design. We evaluate the quality of the method by both structure- (Docking) and ligand-based (QSAR) virtual screening methods. For both evaluation approaches we observed enrichment compared to random sampling from initial chemical space of ZINC drug-like compounds.