Rate distortion in image coding from embedded optimization constraints in vector quantization