The Yarkovsky thermal force on small asteroids and their fragments - Choosing the right albedo