Hill-type formula for Hamiltonian system with Lagrangian boundary conditions