We present the optical design of the ELT polarimeter in the context of the Phase-A study for HIRES. It is well known that in order to reduce the instrumental polarization and cross-talk, the optimal position for a polarimeter along the optical path of a telescope is the rotationally symmetric focus. In the particular case of ELT this is represented by the intermediate focus (IF) below M4 which is not directly accessible and needs therefore a reimaging to a safety distance of at least 500 mm. The design of a transfer optics unit for such location is challenging due to the constraint of having an allowed vignetting area of maximum 5 arc min. We focus in our paper on two optical design solutions. The first one is deploying a double Cassegrain system to reimage the IF, which includes the polarization optics and feeds the other ELT mirrors, redirecting the ordinary and extraordinary beams to the front end module (FE) onto the Nasmyth focus. This module comprises components for sky derotation, atmospheric dispersion correction (ADC), wavelength splitting in two bands (UBVRI, zYJH), field stabilization and conversion to f/20, dispatching the light into two pairs of fiber bundles to feed the HIRES spectrograph. The other solution considers a fiber based compact IF module, using a Schwarzschild Collimator with Foster prism, ADC and beam splitters for the two spectral bands. The two polarized beams are sent by pupil imaging through four separate long fibers to the fiber link module of the spectrograph. There we convert the output fiber f ratio from f/2.5 to f/20.