We study the entanglement spectrum in the many body localizing and thermalizing phases of one and two dimensional Hamiltonian systems, and periodically driven `Floquet' systems. We focus on the level statistics of the entanglement spectrum as obtained through numerical diagonalization, finding structure beyond that revealed by more limited measures such as entanglement entropy. In the thermalizing phase the entanglement spectrum obeys level statistics governed by an appropriate random matrix ensemble. For Hamiltonian systems this can be viewed as evidence in favor of a strong version of the eigenstate thermalization hypothesis (ETH). Similar results are also obtained for Floquet systems, where they constitute a result `beyond ETH', and show that the corrections to ETH governing the Floquet entanglement spectrum have statistical properties governed by a random matrix ensemble. The particular random matrix ensemble governing the Floquet entanglement spectrum depends on the symmetries of the Floquet drive, and therefore can depend on the choice of origin of time. In the many body localized phase the entanglement spectrum is also found to show level repulsion, following a semi-Poisson distribution (in contrast to the energy spectrum, which follows a Poisson distribution). This semi-Poisson distribution is found to come mainly from states at high entanglement energies. The observed level repulsion only occurs for interacting localized phases. We also demonstrate that equivalent results can be obtained by calculating with a single typical eigenstate, or by averaging over a microcanonical energy window - a surprising result in the localized phase. This discovery of new structure in the pattern of entanglement of localized and thermalizing phases may open up new lines of attack on many body localization, thermalization, and the localization transition.