Rim instability of bursting thin smectic films