Organisational Size Metrics in IS Research: A Critical Survey of the Literature 1989-2000

A number of disciplines pursue research into organisations. This organisational research serves to improve knowledge regarding the interaction, behaviour and direction of humans and groups. Many of these disciplines use proprietary methods for and approaches to such research. Because Information Systems (IS) has drawn on several of these disciplines for foundation, a number of research approaches exist for examining organisations within the IS domain. Organisational size measurement, as one research approach in the IS literature, has received considerable application but little critical examination. This study examines six leading IS journals over an eleven year period in order to document and classify the metrics used for organisational size measurement in the IS research literature. The results show a large number of metrics in scholarly use, with studies offering little supportive discussion regarding the application of these metrics. The findings raise a number of issues that are out of the scope of this study: these issues merit further research. INTRODUCTION Theoretical multiplicity is common in many disciplines. For example, authors in domains such as accounting (Watts and Zimmerman 1979), science (Cat 1995) and empirical finance (Fama 1965) have thoroughly documented (and, at times, vigorously defended) competing theories in their respective disciplines. Competing theories abound in the IS discipline also. Information Systems, having grown out of Management Science, Computer Science, Psychology and Organisational Science (Claver et al. 2000) has a rich variety of research approaches available to its scholars: a number of authors note this (Keen 1980, Massetti 1998). This diversity, however, can mean that competing and conflicting theories exist for the analysis of a given phenomenon. Consider, for example, the many approaches to comparing system development success (Olle et al. 1988), quantifying software engineering productivity (Fenton and Neil 1999) or frameworks for strategic systems implementation (Lee and Adams 1990). While circumstances such as this may not be uncommon in scholarly environments, competition between methods can serve to undermine the validity, reliability and comparability of both the scholarly analysis and practical application of research. In general, the steady resolution of such conflict benefits both practitioners and researchers alike. This paper is concerned with better understanding the nature and use of the organisational size metric in information systems research and is motivated by several important issues. First, whereas many research metrics have received considerable analytical attention in the literature, organisational size has received relatively little rigourous scholarly concern. Despite this, the metric continues to receive critical application in many studies and the continual application of the size metric suggests an a priori need to better understand its nature. Second, while some studies have successfully relied on organisational size, other studies have delivered inconclusive results, prompting some researchers to call for a reassessment of the phenomenon (Duncan 1995). Third, many practitioner models (such as those concerning productivity assessments) make judgments based on organisational size. However, models developed using large companies may have tenuous application in small organisation environments (Fayad and Laitinen 1997). The discrepancy associated with size analysis coupled with the popularity of the research area suggests that the sober re-assessment of the metric makes fertile ground for research. To quote March and Smith (1995), “Metrics define what we are trying to accomplish. They are used to assess the performance of an artifact. Lack of metrics and failure to measure artifact performance according to established criteria result in an inability to effectively judge research efforts”. A study such as this represents a significant undertaking, and this paper documents the first stage of such a study by determining those size metrics that are in use in IS research. This leads to this paper’s central research question: 1 The difference between conditions of theoretical diversity and theoretical conflict should be noted. The former is argued by authors such as Cheon et al. (1993) to demonstrate disciplinary maturity and cohesion; the latter represents disagreement and discrepancy. Indeed, some disciplines thrive on such theoretical dichotomy (Palvia and Nosek 1993). Proceedings of the Twelfth Australasian Conference on Information Systems What Metrics Do IS Researchers Use To Measure Organisational Size? The rest of this paper is structured as follows. First, the paper gives an overview of research metrics and measurement. Then, the paper discusses the importance of size measurement in IS research, making specific reference to extant research into small and large organisational differences. The research approach and results are then detailed. Limitations and an overview of the next stage of the study are then considered. METRICS AND MEASUREMENT IN RESEARCH The nature of scientific enquiry is that observations are made about phenomena. This study is concerned with observations about the relationship between reality (for example, the true, unobserved size of an organisation) and the scientific research construct used to measure this reality (for example, the measure of the perceived size of an organisation). In order to conduct sound science, researchers are generally bound by two main requirements. First, research should be internally and externally valid: observations are comparable within and between experiments. Second, research should be reliable: similar observations can be made across experiments (Cook and Campbell 1979). Accurate measurement underpins both validity and reliability. The use of inaccurate measurement methods compromise validity by compromising the researcher’s ability to make intra-study comparisons. Likewise, inaccurate measurement also compromises a researcher’s ability to make inter se comparisons to other studies. While many argue that research can never be completely valid and reliable, agreement upon consistent measurement methods lies at the heard of all science, including IS as a social science. IS researchers and practitioners alike are typically interested in being able to quantify and measure a given phenomenon, largely so that these phenomena can be compared across time or space. This is one of the fundamental reasons for which systems analysts use Entity Relationship Modeling and Dataflow Diagrams when examining information systems: these methods provide analysts with a standardised way of representing the phenomenon under examination. Similarly, the analysis of semantic and relational data models assist in the examination of database effectiveness (Chan et al. 1991). A range of studies in IS pursue the development and analysis of metrics, for instance software development (Fenton and Neil 1999), use of hypertext interfaces (Otter and Johnson 2000) and parallel system efficiency (Cremonesi et al. 1999). In addition, the accurate measurement of system effectiveness (Srinivasan 1985), system performance (Brancheau and Wetherbe 1987) and system success (Delone and McLean 1992) remain areas of substantial debate in the IS literature. These observations underline the importance of standardised metrics in research pertaining to the analysis of the real world. ORGANISATIONAL SIZE IN THE IS LITERATURE Examination of the behaviour of organisations (and, macroscopically, groups of organisations) remains a pivotal component of many research domains: this scholarly attention is echoed in the IS research literature (Lucas and Baroudi 1994). This organisational research comes in a variety of forms, including social organisations, governmental organisations, managerial groups and commercial organisations, with particular attention given to the analysis of social organisations (Holsapple and Luo 1996). The examination and use of size in the IS literature has also received widespread attention. Size, as a measure of capacity, has been applied to a variety of environments: Igbaria and Greenhaus (1991) examined computer departments (number of IT employees), Kappelman et al. (1998) and Wrigley and Dexter (1991) examined software inventories and information system size respectively (lines of code), Gopal and Sanders (1997) examined software piracy clubs (number of members), Mukhophadyay and Kekre (1995) examined production plants (thousands of vehicles produced per year), Choudhury (1998) examined airlines (number of aircraft component purchase orders per year), and Schwartz and Wood (1993) examined email administrative domains (number of email users). The number of studies involving the size metric suggests two issues. First, it emphasises the variety of research environments in which size metrics are applied. Second, it emphasises the range of potential metrics available to the IS researcher. A number of authors make similar observations (Jenkins 1985, March and Smith 1995). Given the importance of size and organisations, organisational size has been held as the most important characteristic in the analysis of technology (Lind et al. 1989) and it has been found to apply equally well to large and small business groups (Raymond 1985). This research has made a number of observations concerning small and large organisations. Some studies suggest that technology adopters tend to be larger than non-adopters (Montazemi 1989). possibly because larger businesses can allocate greater financial and personnel resources to the adoption and use of new technology. Larger organisations, it is argued, typically have more complex developmental approaches (Raymond 1991) and greater risk (Ivancevich et al. 1998), often requiring greater information support networks (Yap 1990) or products such as CASE tools (Hayley and Lyman 1990). Larger Procee