Search for rare B meson decays at the BaBar experiment