Mechanisms of Self-Ordering in Nonplanar Epitaxy of Semiconductor Nanostructures