Probing the nonlinear structure of general relativity with black hole binaries