Precision limits of waveform recovery and analysis in a signal processing oscilloscope