Analysis of the capture reaction using the shell model embedded in the continuum