Method for image coding by rate-distortion adaptive zerotree-basedresidual vector quantization and system for effecting same