Mean-field limit of Non-exchangeable interacting diffusions with singular kernels