Symmetric design of encoder-decoder networks is common in deep learning. For almost all segmentation problems, the output segmentation is vastly less complex compared to the input image. However, the effect of the size of the decoder on segmentation performance has not been investigated in literature. This work investigates the effect of reducing decoder size on binary segmentation performance in a medical imaging application. To this end, we propose a methodology to reduce the size of the decoder in encoder-decoder networks, where residual skip connections are employed in combination with a 1x1 convolution instead of concatenations (as employed by U-Net) to achieve models with asymmetric design. The results on the ISIC2017 data set show that the amount of trainable parameters in the decoder can be reduced by up to a factor 100 compared to standard U-Net, while retaining segmentation performance. Additionally, the reduced amount of trainable decoder parameters in the proposed models leads to inference times up to 3 times faster compared to standard U-Net.