Research methods in computing: what are they, and how should we teach them?

Despite a lack of consensus on the nature of Computing Research Methods (CRM), a growing number of programs are exploring models and content for CRM courses. This report is one step in a participatory design process to develop a general framework for thinking about and teaching CRM.We introduce a novel sense-making structure for teaching CRM. That structure consists of a road map to the CRM literature, a framework grounded in questions rather than answers, and two CRM skill sets: core skills and specific skills. We integrate our structure with a model for the process a learner goes through on the way to becoming an expert computing researcher and offer example learning activities that represent a growing repository of course materials meant to aid those wishing to teach research skills to computing students.Our model is designed to ground discussion of teaching CRM and to serve as a roadmap for institutions, faculty, students and research communities addressing the transition from student to fully enfranchised member of a computing research community of practice. To that end, we offer several possible scenarios for using our model.In computing, research methods have traditionally been passed from advisor to student via apprenticeship. Establishing a richer pedagogy for training researchers in computing will benefit all (see Figure 1).

[1]  B. Bloom Taxonomy of educational objectives , 1956 .

[2]  Donald H. Drury,et al.  A Historiographical Examination of Information Systems , 1999, Commun. Assoc. Inf. Syst..

[3]  Venkataraman Ramesh,et al.  A unified classification system for research in the computing disciplines , 2005, Inf. Softw. Technol..

[4]  Brenda Dervin,et al.  Sense-making theory and practice: an overview of user interests in knowledge seeking and use , 1998, J. Knowl. Manag..

[5]  David Passig,et al.  A Taxonomy of Future Higher Thinking Skills , 2003, Informatics Educ..

[6]  Shamus P. Smith,et al.  Augmenting descriptive scenario analysis for improvements in human reliability design , 2002, SAC '02.

[7]  Christopher D. Hundhausen,et al.  Integrating algorithm visualization technology into an undergraduate algorithms course: ethnographic studies of a social constructivist approach , 2002, Comput. Educ..

[8]  Justin Zobel Writing for Computer Science, 2nd ed , 2004 .

[9]  Raymond Lister Mixed methods: positivists are from Mars, constructivists are from Venus , 2005, SGCS.

[10]  Varun Grover,et al.  Analyzing methodological rigor of MIS survey research from 1980-1989 , 1993, Inf. Manag..

[11]  Martha Kyrillidou,et al.  Pilot testing the DigiQUALTM protocol: lessons learned , 2006, Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '06).

[12]  Venkataraman Ramesh,et al.  Research in computer science: an empirical study , 2004, J. Syst. Softw..

[13]  Steve Sato,et al.  Methods & tools: Playacting and focus troupes:: theater techniques for creating quick, intense, immersive, and engaging focus group sessions , 1999, INTR.

[14]  Phil Turner,et al.  Designing Interactive Systems. , 2005 .

[15]  Richard T. Snodgrass,et al.  Forensic analysis of database tampering , 2008, TODS.

[16]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[17]  Geri Gay,et al.  Concept mapping: an innovative approach to digital library design and evaluation , 1995, SIGO.

[18]  Vincent S. Lai,et al.  Exploring the research in information technology implementation , 1997, Inf. Manag..

[19]  John S. Murnane,et al.  An empirical study of junior secondary students' expression of algorithms in natural language , 2002 .

[20]  Mary Elizabeth Raven,et al.  Using contextual inquiry to learn about your audiences , 1996, ASTR.

[21]  G. Michael Schneider,et al.  A new model for a required senior research experience , 2002, SGCS.

[22]  Penny Hagen,et al.  Emerging research methods for understanding mobile technology use , 2005, OZCHI.

[23]  Varun Grover,et al.  The evolution of empirical research in IS: A study in IS maturity , 1993, Inf. Manag..

[24]  Kimmo Tuominen,et al.  Multiperspective digital libraries: The implications of constructionism for the development of digital libraries , 2003, J. Assoc. Inf. Sci. Technol..

[25]  R. Goede,et al.  The applicability of grounded theory as research methodology in studies on the use of methodologies in IS practices , 2003 .

[26]  Sunny Consolvo,et al.  Evaluating an ambient display for the home , 2005, CHI Extended Abstracts.

[27]  Robin Jeffries,et al.  Comparative expert reviews , 2003, CHI Extended Abstracts.

[28]  Geoffrey E. Mills,et al.  Educational Research: Competencies for Analysis and Application , 1995 .

[29]  Ian H. Witten,et al.  Getting research students started: a tale of two courses , 1993, SIGCSE '93.

[30]  Rosío Alvarez,et al.  Tell me a good story: using narrative analysis to examine information requirements interviews during an ERP implementation , 2002, DATB.

[31]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[32]  Peter J. Denning,et al.  Great principles in computing curricula , 2004, SIGCSE '04.

[33]  Douglas R. Vogel,et al.  MIS research: a profile of leading journals and universities , 1984, DATB.

[34]  Gary Marchionini,et al.  Evaluating hypermedia and learning: methods and results from the Perseus Project , 1994, TOIS.

[35]  Karen Ward,et al.  The fifty-four day thesis proposal: first experiences with a research course , 2004 .

[36]  Inspec Crc Press Handbooks Elsevier Engineering Information , 2003 .

[37]  Cristina Gena,et al.  Methods and techniques for the evaluation of user-adaptive systems , 2005, The Knowledge Engineering Review.

[38]  Guy Latouche Algorithmic Analysis of a Multiprogramming-Multiprocessor Computer System , 1981, JACM.

[39]  Joey F. George,et al.  Exploring the software engineering component in MIS research , 1995, CACM.

[40]  Lauri Malmi,et al.  Constructing a core literature for computing education research , 2005, SGCS.

[41]  D. C. Phillips,et al.  On the Science of Education Design Studies , 2003 .

[42]  Soon Ang,et al.  Turnover of IT professionals: a quantitative analysis of the literature , 2003, SIGMIS CPR '03.

[43]  Yvonne Dittrich,et al.  Personas is not applicable: local remedies interpreted in a wider context , 2004, PDC 04.

[44]  Mordechai Ben-Ari,et al.  Situated Learning in Computer Science Education , 2004, Comput. Sci. Educ..

[45]  Barbara L. McCombs,et al.  Self-Regulated Learning and Academic Achievement: A Phenomenological View , 1989 .

[46]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[47]  Ross A. Gagliano,et al.  Mathematical modeling and Ada simulation of some synchronization processes , 1987, ANSS '87.

[48]  Peggy A. Ertmer,et al.  The expert learner: Strategic, self-regulated, and reflective , 1996 .

[49]  Clifford A. Shaffer Experiences teaching a graduate research methods course , 2006, SGCS.

[50]  Robin Jeffries,et al.  Applying cognitive walkthroughs to more complex user interfaces: experiences, issues, and recommendations , 1992, CHI.

[51]  D. Tall The Cognitive Development of Proof: Is Mathematical Proof For All or For Some? , 1998 .

[52]  Khaled El Emam,et al.  An empirical evaluation of the G/Q/M method , 1993, CASCON.

[53]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[54]  Patricia Carlson,et al.  A Review of MIS Research and Disciplinary Development , 1992, J. Manag. Inf. Syst..

[55]  Margaret-Anne D. Storey,et al.  Theories, Methods and Tools in Program Comprehension: Past, Present and Future , 2005, IWPC.

[56]  Alan Fekete,et al.  Preparation for research: instruction in interpreting and evaluating research , 1996, SIGCSE '96.

[57]  David Beymer,et al.  Wide vs. Narrow Paragraphs: An Eye Tracking Analysis , 2005, INTERACT.

[58]  Martin Höst,et al.  A case study investigating the characteristics of verification and validation activities in the software development process , 2003, 2003 Proceedings 29th Euromicro Conference.

[59]  Ned Kock,et al.  The three threats of action research: a discussion of methodological antidotes in the context of an information systems study , 2004, Decis. Support Syst..

[60]  Paul R. Cohen,et al.  Empirical methods for artificial intelligence , 1995, IEEE Expert.

[61]  Richard J. Shavelson,et al.  Statistical Reasoning for the Behavioral Sciences , 1980 .

[62]  Gregory D. Abowd,et al.  Human-Computer Interaction (3rd Edition) , 2003 .

[63]  Richard L. Van Horn,et al.  Empirical studies of management information systems , 1973, DATB.

[64]  Alex Paramythis,et al.  A Decomposition Model for the Layered Evaluation of Interactive Adaptive Systems , 2005, User Modeling.

[65]  David F. Redmiles,et al.  Extracting usability information from user interface events , 2000, CSUR.

[66]  Menno D. T. de Jong,et al.  Does think aloud work?: how do we know? , 2006, CHI Extended Abstracts.

[67]  Jesper Simonsen,et al.  Make room for ethnography in design!: overlooked collaborative and educational prospects , 1998, ASTR.

[68]  Scott Hamilton,et al.  Knowledge Utilization Among MIS Researchers , 1982, MIS Q..

[69]  Detmar W. Straub,et al.  Normative standards for IS research , 1994, DATB.

[70]  Wanda J. Orlikowski,et al.  Studying Information Technology in Organizations: Research Approaches and Assumptions , 1991, Inf. Syst. Res..

[71]  Grant Braught,et al.  Disequilibration for teaching the scientific method in computer science , 2002, SIGCSE '02.

[72]  Ali F. Farhoomand,et al.  Scientific progress of management information systems , 1987, DATB.

[73]  David Chenho Kung,et al.  Software requirements and acceptance testing , 1997, Ann. Softw. Eng..

[74]  Tor Stålhane,et al.  Trade-off Analysis in Web Development , 2005, 3-WoSQ.

[75]  John Impagliazzo,et al.  Computing Curricula 2005: The Overview Report , 2006, SIGCSE '06.

[76]  Shihong Huang,et al.  Towards a documentation maturity model , 2003, SIGDOC '03.

[77]  Stephanie Rosenbaum,et al.  Alternative methods for field usability research , 2003, SIGDOC '03.

[78]  Gautam Biswas,et al.  Applications of qualitative modeling to knowledge-based risk assessment studies , 1989, IEA/AIE '89.

[79]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[80]  Venkataraman Ramesh,et al.  An analysis of research in computing disciplines , 2004, CACM.

[81]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[82]  Mikael B. Skov,et al.  A review of research methods in children's technology design , 2005, IDC '05.

[83]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[84]  Robert F. Testa,et al.  Educational Research: Competencies for Analysis and Application , 1979 .

[85]  Nelishia Pillay,et al.  A genetic programming system for the induction of iterative solution algorithms to novice procedural programming problems , 2005 .