Evolution of complex magnetic phases and metal-insulator transition through Nb substitution in La0.5Sr