Experimental Evaluation of the New Automated Team Composition System (ATCS) by Formation of Optimal Teams for an Engineering Research Task

This paper describes an experimental study conducted in partnership with the UCLA School of Engineering and Applied Science and designed to demonstrate and evaluate in a real-world situation the use of the new Automated Team Composition System (ATCS) being developed by Perceptronics Solutions under a US Army Research Institute (ARI) Small Business Innovative Research (SBIR) project. The ATCS is an innovative PC-based tool that supports rapid team formation in critical incident preparedness and response as well as in other applications. The ATCS uses advanced multiagent system software technology to help command and/or decision making groups solve the daunting organizational problem of optimally assigning personnel to mission-oriented teams. The team performance criteria used by the Multiagent System derive from the analysis of specific mission features as well as from the psychological literature. The present study used archival data from a UCLA (University of California Los Angeles) engineering course in which nominally student teams prepare a research report on a societal problem involving both technical and ethical issues. Data on approximately 700 students and 130 teams were first examined to determine the relationships among student characteristics (including major, gender, ethnicity and essay and test scores) and team performance (as defined by scores on several components of the research report). Analysis of the data revealed a number of significant, marginally significant and trend relationships between team member characteristics and team performance. These results, in combination with other information about course goals, were used to program the ATCS to form optimal 5-person teams with specific role assignments. The ATCS was able within 1 minute to recommend teams from the candidate population that met all the programmed criteria and had strong face validity as well -- a task well beyond human capabilities. The study showed that the ACTS can be readily used in a real-world situation in which rapid team formation is required and where knowledge about the team mission can be combined with knowledge about the factors influencing team performance. Previous ATCS demonstrations have used on-the-spot questionnaires to gather attribute and qualification data about team candidates in anti-terrorist exercises. In the present evaluation we used candidate data from a stored data base, which we see as a major advantage in future applications of the ATCS tools, and also used a non-military, non-security situation to demonstrate the broad applicability of the technology.