Optimization of solid atomic magnetometer with frequency modulation method on ensembles of nitrogen vacancy color centers in diamond