Low-mass black holes as the remnants of primordial black hole formation