Note on ghost-free matter couplings in massive gravity and multigravity