The anharmonic effect in thermoelectrics has been a central topic for decades in both condensed matter physics and material science. However, despite the long-believed strong and complex anharmonicity in the Bi2Te3−xSex series, experimental verification of anharmonicity and its evolution with doping remains elusive. We fill this important gap with high-resolution, temperature-dependent Raman spectroscopy in high-quality single crystals of Bi2Te3, Bi2Te2Se, and Bi2Se3 over the temperature range from 4 to 293 K. Klemens’s model was employed to explain the renormalization of their phonon linewidths. The phonon energies of Bi2Se3 and Bi2Te3 are analyzed in detail from three aspects: lattice expansion, cubic anharmonicity, and quartic anharmonicity. For the first time, we explain the evolution of anharmonicity in various phonon modes and across the series. In particular, we find that the interplay between cubic and quartic anharmonicity is governed by their distinct dependence on the phonon density of states, providing insights into anomalous anharmonicity designing of new thermoelectrics.