Building an Experience Base for Software Engineering: A Report on the First CeBASE eWorkshop

New information is obtained by research and disseminated by papers in conferences and journals. The synthesis of knowledge depends upon social discourse among the experts in a given domain to discuss the impact of this new information. Meetings among such experts are, however, expensive and time consuming. In this paper we discuss the organization of CeBASE, a center whose goal is the collection and dissemination of empirically-based software engineering knowledge, and the concept of the online workshop or eWorkshop as a way to use the Internet to minimize the needs of face-to-face meetings. We discuss the design of our eWorkshop and give the results of one eWorkshop that discussed the impact of defect reduction strategies. 1 Building an Experience Base for Software Engineering Software development is a peopleand knowledge-intensive activity; it is a rapidly changing field, and although it is slowly maturing, many activities are still ad hoc and depend upon personal experiences. In order to cope with such restrictions as firm deadlines and shrinking budgets, software-developing organizations need assistance in setting up and running increasingly critical projects. In order to reach their goals, software development teams need to understand and choose the right models and techniques to support their projects. They must answer key questions: what is the best life-cycle process model to choose for this particular project (from waterfall to extreme programming)? What is an appropriate balance of effort between inspections and testing in a specific context? And what are the benefits, if any, to buy a readily available software component instead of developing it? These questions are not easy to answer. In some cases the knowledge exists to answer such questions; in other cases it does not, so instead on relying on knowledge and experience, we must trust our instincts. In order to support this decision-making activity, we need to develop empirically based software models in a systematic way, covering all aspects from high-level lifecycle models to low-level techniques, in which the effects of process decisions are well understood. However, context plays an important role as most projects and organizations differ. Consequently, the knowledge must be formulated relative to the development context and project goals. The Center for Empirically-Based Software Engineering (CeBASE)1 was organized to support this goal. CeBASE will accumulate empirical models in order to provide validated guidelines for selecting techniques and models, recommend areas for research, and support software engineering education. CeBASE’s objective is to transform software engineering from a fad-based practice to an engineering-based discipline in which development processes are selected based on what is known about their effects on products, through synthesis, derivation, organization, and dissemination of empirical knowledge on software development and evolution phenomenology. CeBASE is a new National Science Foundation-sponsored research center led by personnel with extensive industry and government experience, including Professors Barry Boehm (University of Southern California), Scott Henninger (University of Nebraska Lincoln) Rayford Vaughn (Mississippi State University) and co-author Victor Basili (University of Maryland and Fraunhofer Center for Experimental Software Engineering – Maryland). Their experience includes major improvements in software productivity, quality, and cycle time on highly complex software-intensive systems. Within CeBASE we are integrating existing data, methods, and models (including the Experience Factory, Goal-Question-Metric approach, Spiral life cycle Model, Model-Based (System) Architecting and Software Engineering (MBASE), stakeholder Win-Win requirements engineering, COCOMO cost and schedule modeling, and perspective-based reviewing) with the best commercial and government practices (such as the CMMI, Rational Unified Process, Independent Expert Reviews/Architecture Review Boards, SEI Product Line, Risk, and Architecture practices). The result will be tailorable methods for defining, managing, and continuously improving future software intensive systems involving challenges such as COTS based development, systems distribution and mobility, and tradeoffs among cost, schedule, performance, dependability, interoperability, usability, and other factors. The initial focus of CeBASE is on two high-leverage areas of software engineering, defect reduction and COTS based development. CeBASE collects, documents, and disseminates knowledge on software engineering gained from experiments, case studies, observations, and real world projects. While some of this empirical knowledge might be well known by the community, it has not yet been documented. Although this knowledge is believed to be generally applicable, the effects of its application have never been systematically investigated making it difficult to discern when it is useful. Some of this knowledge is distributed among many individuals, which means that we need to gather the pieces together and facilitate the collection and management of collective knowledge. Meetings among experts discussing their findings and recording their discussions are a classical method for creating and disseminating knowledge. By analyzing such discus1 http://www.CeBASE.org sions new knowledge can be created and the results can be shared. This is generally achieved by holding workshops. Workshops, however, possess limitations: 1) experts are spread all over the world and would have to travel, and 2) workshops are usually oral presentations and discussions, which are generally not captured for further analysis. To overcome these problems we designed the concept of the eWorkshop, using the facilities of the Internet. This paper describes the process of running an eWorkshop and the results from the first eWorkshop, which was held March 16, 2001. 2 Conducting an eWorkshop The eWorkshop is an on-line meeting, which replaces the usual face-to-face workshop. While it uses a Web-based chat-application, it is structured to accommodate the needs of a workshop without becoming an unconstrained on-line chat discussion. The goal is to synthesize new knowledge from a group of experts as an efficient and inexpensive method in order to populate the CeBASE experience base. The idea behind the eWorkshop was to use simple collaboration tools, thus minimizing potential technical problems and decreasing the time it would take to learn the tools. Simultaneously, we wanted to set up a support team and control room to ensure that there would be as few disturbances as possible once the eWorkshop was running. 2.1 eWorkshop Process Organization of the workshop follows a strict protocol: 1. Choose a topic of discussion. The topic under discussion is first determined. 2. Invite participants. Participants are invited and instructed to log into the eWorkshop using a Web browser at the appointed time. 3. Distribute Pre-meeting information sheet. To direct the discussion, preliminary information about the topic is presented to the participants, who send in a premeeting information sheet. This is used to guide the discussion during the meeting. 4. Establish meeting codes – for meeting analysis. The workshop organizers analyze the information sheets to develop a taxonomy of issues to be discussed. 5. Publish synthesized info from pre-meeting sheets. An analysis of the information sheets are given by the eWorkshop team and distributed to each participant before the meeting. 6. Schedule pre-meeting training on tools. A preliminary work session is scheduled to give meeting participants a chance to try out the software so that the meeting can proceed smoothly. 7. Set up control room. Several individuals (described later) actually run the meeting. While most participants are in their own offices looking at a computer screen, the meeting organizers need to coordinate their activities among several roles. 8. Conduct meeting. At the appointed time, the participants use their Web browser to log into the chat tool and the meeting is underway. 9. Post-meeting analysis and synthesis. The meeting organizers keep a script of the meeting. This is analyzed to extract knowledge for the knowledge base.