Fundamental limits to attractive and repulsive Casimir-Polder forces