People changing residence within their own country is the most common type of migration and is one of the main driving forces of a country's demography. Yet, due to a lack of data and tools, some key characteristics of internal migration such as the propensity to remain in the same city or to return to previous locations are frequently ignored. Here, a network-based model of city-to-city migration is constructed, where the movement of individuals is modelled using the frequency of distinct network motifs. We fit this model to longitudinal data on 3.3 million workers in Colombia, including 1.4 million migrations, and compare the motif frequency based on migration and return rates between men and women, and between distinct age and income groups. Results show that the majority of people do not move in general, but that there is a small group which exhibits frequent migration, particularly the young and male. Nearly three out of four times that a person moves, they are going back to a previous city, although women and mature people are less likely to move and more likely to return if they actually move. At a city level, different onward and return migration patterns are observed, whereby people from small secondary towns are more likely to leave and not return than people from large metropolitan areas like Bogotá or Medellín.