Studies of radiative penguin decays at BaBar