Onset of random matrix behavior in scrambling systems