Least squares filters in canonical coordinates for transform coding, filtering, and quantizing