Breakdown of Polaritons in Nanophotonic Systems