Backreaction from inhomogeneous matter fields during large-scale structure formation