Molecular dynamics simulations of all-atom and coarse-grained lipid bilayer models are increasingly used to obtain insights useful for understanding the structural dynamics of these assemblies. In this context, one crucial point concerns the comparison of the performance and accuracy of classical force fields (FFs), which sometimes remains elusive. To date, the assessments performed on different classical potentials are mostly based on the comparison with experimental observables, which typically regard average properties. However, local differences of structure and dynamics, which are poorly captured by average measurements, can make a difference, but these are non-trivial to catch. Here we propose an agnostic way to compare different FFs at different resolutions (atomistic, united-atom, and coarse-grained), by means of a high-dimensional similarity metrics built on the framework of Smooth Overlap of Atomic Positions (SOAP). We compare and classify a set of 13 force fields, modeling 1-palmitoyl-2-oleoyl-sn-glycero-3-phosphocholine (POPC) bilayers. Our SOAP kernels-based metrics allows us to compare, discriminate and correlate different force fields at different model resolutions in an unbiased, high-dimensional way. This also captures differences between FFs in modeling non-average events (originating from local transitions), such as for example the liquid-to-gel phase transition in dipalmitoylphosphatidylcholine (DPPC) bilayers, for which our metrics allows to identify nucleation centers for the phase transition, highlighting some intrinsic resolution limitations in implicit vs. explicit solvent force fields.