Color-kinematics duality for pure Yang-Mills and gravity at one and two loops