We demonstrate the realization of topological band structures by exploiting the intrinsic spin-orbit coupling of dipolar interactions in combination with broken time-reversal symmetry. The system is based on polar molecules trapped in a deep optical lattice, where the dynamics of rotational excitations follows a hopping Hamiltonian which is determined by the dipolar exchange interactions. We find topological bands with Chern number $C=2$ on the square lattice, while a very rich structure of different topological bands appears on the honeycomb lattice. We show that the system is robust against missing molecules. For certain parameters we obtain flat bands, providing a promising candidate for the realization of hard-core bosonic fractional Chern insulators.