Can galaxy growth be sustained through HI-rich minor mergers?