Quantifying the fragility of unprotected quadratic band crossing points