Refined BPS invariants of 6d SCFTs from anomalies and modularity