Is modularity robust to misfits? A formal test