Nonunitary transformations in quantum optics