The continuous spectrum of the relative trace formula for GL(3) over a quadratic extension