On initial-boundary value problem for nonlinear viscoelastic wave system with damping and source terms