Exponential and polynomial stability of a wave equation for boundary memory damping with singular kernels