This paper presents a distributed multi-modality sensor network concept for vehicle classification within perimeter of a surveillance system. This perimeter surveillance concept represents a "Virtual RF Fence" consisting of remotely located electro-optic surveillance cameras and a standoff range radar system. The perimeter surveillance system vigilantly monitors the field and each time a vehicle crosses the virtual RF fence it informs the surveillance cameras to actively monitor the activity of vehicles as it passes through the field. This paper describes the methodologies applied for processing the EO imagery data including target vehicle segmentation from background, vehicle shadow elimination, vehicle feature vector generation, and a neural network approach for vehicle classification. A metric is also proposed for evaluation of performance of the vehicle classification technique.