Towards the fast scrambling conjecture