Unitarity and vacuum stability constraints on the couplings of color octet scalars