Open quantum systems with nonlinear environmental backactions: Extended dissipaton theory versus core-system hierarchy construction