World-class product certification using Erlang

It is now ten years ago since the decision was made to apply the functional programming language Erlang to real production projects at Ericsson. In late 1995, development on the Open Telecom Platform (OTP) started, and in mid 1996 the AXD 301 project became the first user of OTP. The AXD 301 Multi-service Switch was released in October 1998, and later became "the heart of ENGINE", Ericsson's leading Voice over Packet solution.In those early days of Erlang programming, high-level tools for development and testing were not really available, and programmers used mainly the Emacs editor and the Erlang shell. Still, anecdotal evidence suggested a 4-10x productivity increase compared to mainstream programming techniques.Through the years, significant progress has been made, especially in the area of automated testing of Erlang programs. The OTP team designed an Erlang-based test suite execution environment, were developers can easily write their own automated test suites, and now performs nightly builds where more than one thousand test cases are executed on ten different platforms. OTP designers can view the outcome in web-based test reports as they come to work the next day. Each corrected bug results in a new test case that is incorporated into the ever-growing test suite. Thus, this world-class middleware is certified to telecom-class quality without a dedicated test team!The AXD 301 project uses OTP's test environment, and executes more than 10,000 automated test cases before each major release. Designers and testers compose their own test suites, and the designers carry out function tests with little or no help from the Integration and Certification team. Each test case can be run both in a simulated environment on the designer's workstation and in the test lab on real hardware. In order to provide stimuli to the system, the testers often design their own traffic generators in Erlang.To analyze the faults that occur, Erlang offers an increasing wealth of debugging options. Beyond the symbolic error messages, which are often sufficient to locate the fault, Erlang developers are able to dynamically turn on tracing on message passing, scheduling events, garbage collections, selected function calls, etc.This paper demonstrates how Erlang's declarative syntax and pattern matching provide an outstanding environment for test suite development.