A new method to build the (inverse) distance ladder