Linearity of response is one of the most important features of a measurement system. Linearity implies that accurate linewidths can be obtained from measured values knowing only the slope and offset of the data with respect to reference data taken with another, presumably more accurate, instrument. A first-order linear regression of the data yields the slope, offset, and estimate of the goodness of fit. Ideally, the slope is near unity, so that the magnification scales of the two instruments agree. The offset is considerably less important since, in IC process control, absolute changes in linewidth are often of more concern than the linewidths themselves. This paper demonstrates by simulation and experiment that the linearity (R-Squared) of an optical microscope depends not only upon the characteristics of the tool but also upon the characteristics of the object being measured. In virtually all optical microscopes, transparent structures support waveguide resonant eigenmodes which are strongly affected by geometry and contribute substantially to non- linearities in response. For isolated lines, nonlinearities are found to occur especially at certain widths where the eigenfunctions change rapidly with small change in width. The theory of these singular points is presented. The authors demonstrate that the coherence microscope, which uses both phase and amplitude information, has a potential advantage over brightfield and confocal microscopes in dealing with these problems. The introduction of a 'complex phase filter' in the measurement algorithm greatly reduces unwanted phase noise and its concomitant contribution to non-linearity. The ability to simulate the optical images and resulting measurement non-linearities offers promise in improving understanding and accuracy of optical metrology.