A gauge model of amorphous solids containing defects II. Chern-Simons free energy