Long and Short Time Asymptotics of the Two-Time Distribution in Local Random Growth