Performance of a prototype active veto system using liquid scintillator for a dark matter search experiment