Role of dense matter in collective supernova neutrino transformations