Automated Assessment of Systems Engineering Competencies

Systems engineering and technical leadership (SETL) is a multidisciplinary practice that is as much an art as a science. While a traditional model of education can teach the fundamental body of knowledge, it is not until this knowledge is put into practice in an integrated, real world environment that a systems engineer can develop the necessary insights and wisdom to become proficient. Organizations and enterprises not only need to improve the existing workforce to enable them to keep up with the demands of the work place, but also require a better approach to assess and evaluate the competencies and learnings of prospective and practicing systems engineering practitioners. Learning assessment is a critical component of accelerated learning. It is imperative to understand individual learning and the efficacy of the various learning experiences. This is critical both in determining the capabilities of the learner, but also enable the continual improvement of the capabilities of the learning experience. This paper describes a set of Automated Learning Assessment Tools (ALATs) that measure a subject’s proficiency in a set of systems engineering competencies and the efficacy simulated learning experiences through analysis of the data recorded throughout the learners’ participation in a simulation experience. The vehicle that it uses is the Systems Engineering Experience Accelerator which is a new approach to developing the systems engineering and technical leadership workforce, aimed at accelerating experience assimilation through immersive, simulated learning situations where learners solve realistic problems. A prototype technology infrastructure and experience content has been developed, piloted, and evaluated. Traditionally, learning assessment has been done through examinations and experts’ reviews and opinions on students’ work which requires substantial effort. In addition, most approaches emphasize comparing learners’ performance against those of the experts’ and less about the evaluation the actual learning performance of the individuals. Though simulation has been widely adapted by systems engineering learning, it has yet to be used to assess learner competencies and learnings performance in systems engineering and technical leadership learning. The ALATs described in this paper address these issues. This paper describes the evaluation of the capabilities of these tools through their performance in a number of pilot studies. Evidence of systems engineering competencies and learning trajectories is analyzed, compared and contrasted from the perspective of the learner’s performance, behaviors, self-evaluation and finally expert assessments. The limitations and strengths of the various approaches are discussed. Finally, areas of future research in pilot studies and learning assessment tool capabilities are described.