Dibaryons cannot be the dark matter