Challenges to ensuring quality in qualitative research: A procedural view

Qualitative research methods are increasingly being used in engineering education research. In this context there is an ongoing discourse in the community around ways of ensuring interpretive research quality. This paper presents a process-oriented framework of research quality that was developed while undertaking a study that was recently published in the Journal of Engineering Education. Drawing on the concept of Total Quality Management (TQM), the framework consists of two components i) a procedural view of the research process, broadly defined as Making Data and Handling Data, and ii) a flexible typology of fundamental processes of validation (theoretical, procedural, communicative, pragmatic) and the notion of process reliability. Both of these aspects of the framework are illustrated with examples from the aforementioned study. Future work is planned to further develop the conceptual framework as a language for the engineering education community to engage in a discourse around shared, contextual and flexible understandings of research quality. Introduction: Questions of quality in qualitative engineering education research Engineering education research is an inherently interdisciplinary endeavor [1-3] that is currently being undertaken by a community of engineers, social and educational researchers with diverse and contrasting disciplinary and epistemological perspectives [4]. An ongoing discourse in the community is consequently centered around appropriate research methods [5-9] and ways of conducting research of acceptable quality [4, 10, 11]. In this context, Borrego [12] asserts that “the field of engineering education has not yet developed its first paradigm” with the term paradigm being defined as “consensus with regard to [among other aspects] standards of rigor”(p. 6). Addressing this pre-paradigmatic nature of the field, this paper is concerned with questions of research quality in qualitative approaches to engineering education research. More specifically, we draw on an example study recently published in the Journal of Engineering Education [13] to present and explore a number of challenges to research quality in the context of concrete examples from the data collection and analysis. Based on these reflections, we present a processoriented framework of research quality that was developed while undertaking the example study and offer it here as a further step in the ongoing discussion of interpretive research quality in the engineering education research context. Example study: Interpretive investigation of Accidental Competency formation The interpretive study that provides the context for the illustrations used in this paper (see part 3) is an exploratory investigation of engineering students' competence formation from a broad, holistic perspective [13]. More specifically, the authors conceptualized the notion of Accidental Competencies as a lens through which to investigate how students' overall competence formation emerges from the complex interplay of explicit instruction and a wide range of influences from the learning environment [14]. Accidental Competencies were conceptualized as the unintended Page 25298.2 consequences, positive and negative, of students‟ overall experience of completing an engineering program. Data was collected in focus groups based on critical incident techniques [15-17] with 67 students in their transition from university studies into professional practice. The students were selected from a range of innovative placement programs (i.e. industry, co-op and service learning programs) from institutions in Australia, Germany, Thailand and the United States. This international selection ensured that a wide range of students' experiences could be captured and the focus on placement students meant that participants were able to recall detailed experiences from their education while having also had a significant exposure to engineering practice. The focus groups were digitally recorded and transcribed for the subsequent data analysis using the qualitative software NVivo7. The iterative analysis based on a grounded theory approach and constant comparative methods yielded clusters and subordinate categories of competencies that the students had developed. Similar codes described the educational influences and work experiences that contributed to these learning processes [for more information see: 13]. The illustrations of challenges to ensuring research quality are based on one dominant theme (role models) that emerged from the analysis. In the focus groups the students reported that teachers and industry engineers were role models who had a significant impact on the development of their professional self-perception. These development processes resulted from a complex interplay of the influence of teachers and engineers with other educational factors to significantly shape the students‟ “professional way of being” [18, p. 389]. Challenges to research quality: Socially constructed reality In investigating student learning that emerges from these complex interactions it became apparent that the „object‟ of our research interest was neither “out there” [19, p. 37] to be observed in a materialistic sense, nor was it is it solely „in the individual‟s head‟. Rather, it extended beyond the individual, in that it was constituted through, and emerged from, the shared lived experience ["Lebenswelt" in: 20] of groups of individuals [21]. Put another way, this meant that the reality we were interested in investigating was socially constructed [22-24], by the participants and the researcher [1] in the data gathering situation. Illustration: To clarify this point, this illustration considers an example from the above-described study that is concerned with the function of teachers as professional role models. Examining this result more closely illustrates the emergent and inter-subjective nature of the social reality under investigation. More specifically, the teacher‟s influence for the individual student was found to be constituted of concrete psychological realities – examples are feelings of consternation, or experiences of tensions between their own professional way of being and their perception of the workplace. The phenomenon of the teacher as a role model, however, emerged only from a sustained and complex interaction of the student with various teachers, other students and industry supervisors. More specifically, the influence of role models as the phenomenon under investigation consisted of, but at the same time, exceeded the individual Page 25298.3 student‟s psychological realities, emerging on a higher level from complex social interactions. This ontological assumption of a socially constructed reality poses the question as to whether the researcher can derive truthful knowledge claims about a social system as the research object. The traditional scientific paradigm assumes a transcendent, materialistic reality that can be known independent of context and time. In contrast, the above illustration identifies a constructed, or, inter-subjective, reality as the object of interpretive research. Equally, interpreting the data entails a subjective process of making sense of the participants‟ multiple perspectives. Considering the multitude of possible outcomes from such interpretations suggests that the social system under investigation does not “determine absolutely the one and only correct view that can be taken of it” [25, p. 14]. This, in turn, means that knowledge is socially constructed both in its production by the researcher and in its representation within the research community. In communicating knowledge claims to the research community, the constructivist nature of knowledge implies that representations must follow the meaning conventions of the research community by “calling things by the right names” [25, p. 23]. An example of how these aspects of qualitative research manifested in the example study is presented below. Illustration: Drawing on the prior illustrative example, the following illustrates the constructivist process of generating knowledge from interpretation by examining more closely how the final interpretation of the teachers‟ influence as role models was derived. During the data gathering, the students spoke of their individual experiences with academics, industry supervisors and of other educational influences. In this discussion, a shared multi-faceted view of the phenomenon emerged among the students. This understanding was constituted by their multiple perspectives but was, at this point, of a tacit nature, i.e. everyone knew what was being talked about and contributed their related stories. In the initial data analysis this shared understanding emerged across several transcripts and was, at this stage, captured in a preliminary node with the „invivo‟ description “academics vs. real engineers”. This first interpretation was socially constructed, in that it emerged from the students‟ shared understanding in several focus groups. Additionally, an expression taken directly from the respondents‟ own words was used to categorize this type of contribution. In terms of useful knowledge, however, this interpretation did not extend far beyond the context of the focus groups. The next step of representing this knowledge thus involved “calling things by the right names” [25, p. 23]. One term commonly used to describe the teachers‟ influence is that of the “role model”. However, this name for the category was one of several choices and was, as such, not directly “imposed by the structure of empirical reality” [25, p. 15]. More importantly, choosing this concept on the basis of my interpretive judgment also applied a range of P ge 25298.4 pre-existing conceptions and frameworks from the literature to this category. To provide a brief historical perspective, before becoming part of everyday language the notion of a role model was proposed as a specific sociologi