The infinite-range clock spin glass model: an investigation of the relevance of reflection symmetry