On operator growth and emergent Poincaré symmetries