Rayleigh's optimization approach for the approximate solution of integral equations for vibration problems