Analysis of self-excited transverse combustion instability in a rectangular model rocket combustor