The Many Faces of Systems Research - and How to Evaluate Them

Improper evaluation of systems papers may result in the loss or delay in publication of possibly important research. This paper posits that systems research papers may be evaluated in one or more of the three dimensions of science, engineering and art. Examples of these dimensions are provided, and methods for evaluating papers based on these dimensions are suggested. In the dimension of science, papers can be judged by how well they actually follow the scientific method, and by the inclusion of proofs or statistical measures of the significance of results. In engineering, the applicability and utility of the research in solving real world problems is the main metric. Finally, we argue that art be considered as a paper category evaluated based on elegance, simplicity, and beauty.

[1]  Roger M. Needham,et al.  On the duality of operating system structures , 1979, OPSR.

[2]  David A. Patterson,et al.  How to Have a Bad Career in Research/Academia , 2005 .

[3]  David Mazières,et al.  Tangler: a censorship-resistant publishing system based on document entanglements , 2001, CCS '01.

[4]  Steven McCanne,et al.  The BSD Packet Filter: A New Architecture for User-level Packet Capture , 1993, USENIX Winter.

[5]  GhemawatSanjay,et al.  The Google file system , 2003 .

[6]  Nancy A. Lynch,et al.  Impossibility of distributed consensus with one faulty process , 1983, PODS '83.

[7]  David A. Patterson,et al.  Towards Availability Benchmarks: A Case Study of Software RAID Systems , 2000, USENIX Annual Technical Conference, General Track.

[8]  Roy Williams.  Context, Content and Commodities: e-Learning Objects. i , 2003 .

[9]  Howard Gobioff,et al.  The Google file system , 2003, SOSP '03.

[10]  Bill Broyles.  Notes , 1907, The Classical Review.

[11]  Glenn D. Berkheimer.  Using Conceptual Change Research To Reason about Curriculum. Research Series No. 195. , 1989 .

[12]  Miguel Oom Temudo de Castro,et al.  Practical Byzantine fault tolerance , 1999, OSDI '99.

[13]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[14]  Krishna P. Gummadi,et al.  The impact of DHT routing geometry on resilience and proximity , 2003, SIGCOMM '03.

[15]  Edsger W. Dijkstra,et al.  The structure of the “THE”-multiprogramming system , 1968, CACM.

[16]  Jean Arlat,et al.  A Framework for Dependability Benchmarking , 2002 .

[17]  Martín Abadi,et al.  A logic of authentication , 1989, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[18]  David D. Redell,et al.  An evaluation of the ninth SOSP submissions or how (and how not) to write a good systems paper , 1983, ACM SIGOPS Oper. Syst. Rev..