Modeling the instability of carbon nanotubes: from continuum mechanics to molecular dynamics