Primordial black holes and stochastic inflation beyond slow roll: I -- noise matrix elements