Detection of curved surface using augmented reality system