Lessons learned during requirements acquisition for COTS systems

Our customer was the procurement executive (PE) of the U.K. Ministry of Defense (MoD). The PE wanted new methods and software tools with which to manage requirements of a new naval platform that would take 20 years to develop. A consortium of four members (including the three authors) was asked to recommend commercial requirements engineering methods and software tools or products for trial by the PE. The consortium was chosen for its knowledge of and experience in MoD procedures and state-ofthe-art requirements engineering methods and software tools. The consortium had 11 weeks to make a recommendation to the PE. This led to considerable time pressures. However, it was decided that, where possible, standard commercial product selection procedures would be adhered to. Requirements acquisition involved both documents and stakeholders. Five meetings with six to 10 stakeholders took place over a three-week period, each divided into a review of the current requirements document and acquisition of new requirements. The final requirements document contained 133 atomic requirement statements in 22 basic hierarchies. To save time, market research was undertaken in parallel with requirements acquisition. An initial version of the requirements document was drafted as a questionnaire and sent to more than 30 candidate suppliers to determine the coverage of their products. A short list of six candidate products was produced from supplier responses to the questionnaire. Next, a set of 35 complex test cases for product evaluation was developed from the final requirements document. Five of the six short-listed suppliers demonstrated their products against the 35 test cases. During each evaluation, product compliance with these requirements was recorded by three members of the consortium using both quantitative scores and qualitative comments. After each evaluation, the final product-requirement compliance scores were agreed to by consortium members. These scores were then investigated using a range of analytic techniques. As a result, trial use of two requirements engineering software tools was recommended.