Competing ν = 5/2 fractional quantum Hall states in confined geometry