Digital signal processing techniques in Nyquist-WDM transmission systems

In single-carrier wavelength-division multiplexing (WDM) systems, the spectral efficiency can be increased by reducing the channel spacing through digital signal processing (DSP). Two major issues with using tight filtering are cross talk between channels and inter-symbol interference (ISI) within a channel. By fulfilling the Nyquist criterion, Nyquist spectral-shaped WDM systems can achieve narrow channel spacings close to the symbol rate $$(\hbox {R}_{\mathrm{S}})$$(RS) with negligible cross talk and ISI. In principle, DSP can generate any signals with arbitrary waveforms and spectrum shapes. However, the complexity of DSP is limited by its cost and power consumption. It is necessary to optimize the DSP to achieve the required performance at a minimum complexity. In this paper, we first introduced the background of digital signal processing for Nyquist spectral shaping in optical fiber WDM systems. Then, we investigated the use of digital finite impulse response (FIR) filters to generate Nyquist-WDM 16-ary quadrature amplitude modulation (16QAM) signals with the raised-cosine (RC) and root-raised-cosine (RRC) shape spectra. The system performance of both the RC and RRC spectra is also examined. Moreover, we explored the various methods to reduce digital-to-analog converter (DAC) sampling speed, such as using super-Gaussian electrical filters (E-filter) and spectral pre-emphasis. We also discussed receiver-side matched filter design for Nyquist-WDM receiver optimization.