Octupole instability of super- and hyperdeformed nuclei