A geometric framework for the kinematics of crystals with defects