Rigidity and flexibility for handlebody groups