High-spin states in {sup 161}Er have been studied experimentally using the {sup 150}Nd({sup 16}O,5n) reaction at a beam energy of 86 MeV. The 5/2{sup +}[642], 3/2{sup -}[521], and 11/2{sup -}[505] bands are extended up to high-spin states, and particularly the {alpha}=-1/2 branch of the ground state 3/2{sup -}[521] band is revised significantly. The relatively enhanced E1 transitions from the 3/2{sup -}[521] band to the 5/2{sup +}[642] band are observed. The band properties are analyzed within the framework of a triaxial particle-rotor model, and near-prolate shape and triaxial deformation are proposed to the 3/2{sup -}[521] and 5/2{sup +}[642] bands, respectively. Signature inversion occurs in the 3/2{sup -}[521] band after the band crossing in {sup 161}Er, and the systematics of the signature inversion associated with the 3/2{sup -}[521] configuration are discussed. By analyzing the properties of the relatively enhanced E1 transitions, it is found that the R(E1/E2) values show angular momentum dependence before the band crossing, and these enhanced E1 transitions could be attributed to octupole softness.