Nonunitary entanglement dynamics in continuous-variable systems