Peer Reviewing of OER in a Contested Domain--An Activity Theoretical Analysis.

Globally, we experience numerous initiatives to increase the adoption of open educational resources (OER), but quality concerns challenge the adoption. In this study we present an analysis of the peer review process of an OER. The OER under review is produced by the European Commission (EU). It has the goal to teach children about farm animal welfare. Following discussions with the EU about its quality, a panel review was conducted. The group of peers used a quality evaluation tool for initial individual assessment, which was then discussed in four consecutive meetings. Video recordings from the meetings and the final report constitute the research data. Cultural historical activity theory (CHAT) was used as the analytical framework. The results indicated that the main areas of negotiations were the content quality of the OER and the adoption of the OER in teaching practices. The examination of these concerns using CHAT indicated that the peer review process neither leads to accuracy nor legitimacy. In summary, OER and in particular the quality assessment of OER challenge the boundaries of higher education. A combination of peer review and crowd source review is suggested to underpin the openness and thus increase adoption of OER. The introduction of open learning is questioning the traditional view of knowledge as a commodity and learning as transmission of commodities. Open educational resources (OER) are part of a trend towards opening up higher education and seeing knowledge as a public good. However, the adoption of OER is still low and challenged by a number of obstacles, particularly quality concerns (Atenas et al., 2014; Ochoa & Duval, 2009). New practices were developed as a response to low adoption, and the concept of open educational practices (OEP) was coined as “...practices which support the (re)use and production of OER through institutional policies, promote innovative pedagogical models, and respect and empower learners as coproducers on their lifelong learning path...” (UNESCO, 2011). The changing focus from interactive online learning to knowledge creation through interactive online social negotiation, reflects a change of paradigm characterized by a shift in the use of Internet from the individual content provider, producing a product that can be distributed to others, which might be called the “Web 1.0”, to a network of content providers that not only are involved in interactive online learning but collectively develop, use and reuse OER in an iterative process of OEP (Auvinen & Ehlers, 2009), based on “Web 2.0” (O’Reilly, 2007). The benefits of OEP are understood in the academic community, but academics have not yet embraced and realised them (Atenas et al., 2014; Camilleri et al., 2014). One reason is its disruptive character, which requires expertise, time, commitment, and institutional support (Atenas et al., 2014). OEP has implications: 1) for the conceptions of teaching and learning since Journal of Interactive Online Learning Algers and Ljung 22 it respects and empowers learners as co-producers of learning (Ehlers, 2011); 2) for the teaching practices, since it explores the potential of using OER to transform educational practices (see e.g. Camilleri et al., 2014); and 3) for the quality assessment of OER. In line with quality assessment in all the sciences (Albert et al., 2012; Smith, 2006), peer review is the preferred quality assessment instrument for OER (Clements & Pawlowski, 2011). Some researchers claim that by adopting peer review, this view on learning as a commodity can be seen as being reinforced (Kanwar et al., 2010). However, a peer review process not only approves the quality of the disseminated knowledge, it can also be seen as bridging the gap between creators and users (Engeström & Sannino, 2010). Since OER based on web 2.0 are constantly in flux and have a multitude of versions and authors (Camilleri et al., 2014), peer review of OER is only a snapshot of its quality. Quality assessment in normative subject areas adds yet another level of complexity, because normative assumptions (on a scale from good to bad) can be debated. Individual reviewers bring with them underlying values which in normative subjects become their motives when they are involved in the negotiations with others. The aim of this article is to analyse a peer review process of an OER in animal welfare, which is a normative and contested area with many different interests and interest groups. This analysis focuses on what is negotiated in the review process, and is an attempt to understand underlying values of negotiations and contradictions. Through this analysis, we wish to contribute to the debate on accuracy and legitimacy related to the quality of OER and to present an alternative method. General characteristics of peer reviewing It is generally accepted that peer review has three aims: (1) to maintain a certain quality standard such as identifying defects of originality and accuracy, (2) to give the work a stamp of quality through summative assessment, and (3) to improve the work through formative assessment (Smith, 2006). However, despite the fact that peer review is used routinely for various academic processes such as for allocating grants, publishing papers, and promoting and rewarding academics, it is not perfect. Smith (2006, p. 178) claimed that “it is compared with democracy: A system full of problems but the least worst we have”. There are different views on how well this routine process is to be understood (Hylén, 2006; Kassier & Campion, 1994). Peer review in academic publishing can be described as a process when a third party, neither the author nor the editor scrutinizes the paper according to certain rules and makes a judgment whether the paper should be published or not (Smith, 2006). The process of peer review has been criticized for a number of reasons including that it is a) not standardized and objective, b) not reliable for detecting fraud, and c) time consuming and expensive (Smith, 1988; Smith, 2006). A strong evidence for bias against women when awarding grants has also been revealed (Wennerås & Wold, 1997), and reviewers have been found to steal ideas and block or slow down the publication of scientific results of competitors (Smith, 2006) and to supress conflicts of interest (King et al., 1997). In the recruitment of reviewers, there is a risk of conflict of interest when the reviewer is a competitor, conducting the same kind of research as the author (King et al., 1997), and therefore the editors seek peers in the same discipline, but with no or only vague connections (Smith, 2006). Authors and reviewers are encouraged to declare their conflicts of interest when the names of authors and sometimes reviewers are known to the other. Moreover, the Journal of Interactive Online Learning Algers and Ljung 23 recruitement should ideally lead to a group of reviewers that are relatively equal in terms of power (Auvien & Ehlers, 2009). Avoiding power asymmetry between reviewers is particularly important in interdisciplinary domains, where individual expertise required for competent evaluation across diverse fields is limited (Grainger, 2007), and in contested domains where the competences of the peers (King et al., 1997) and their values will affect the outcome. Thus peer reviewers should be people of equal social hierarchical level (Auvinen & Ehlers, 2009), yet as assessors they are claiming authority (Kelty et al., 2008). The peer review process has also been applied as an external quality assessment of OER, as a check-approval procedure of a specific version before publication (Auvinen & Ehlers, 2009; Camilleri et al., 2014). However, a check-edit procedure, where the OER is published but under edition is more commonly employed (Camilleri et al., 2014). This procedure is a kind of open peer review conducted after publication, which has also been introduced for some scientific journals (Smith, 2006). Panel review might be a dynamic approach for innovation and development. Theoretical framework The study of peer reviewing is a study of a complex social process. Cultural Historical Activity Theory (CHAT) is used as a lens for analyzing peer reviewing and to better understand underlying values for the assessors’ activities. CHAT, as conceptualized by Engeström (1987), studies both the individual and the social activities. The individual, the object, and the instruments, which constitute the upper triangle in the activity system are “the tip of the iceberg” as they represent the “visible instrumental actions” (Engeström, 1998, p. 79) (Figure 1). The lower part of the triangle illustrates that the activities are carried out within a social context, and the relationship between subject and the community is mediated by rules and the relationship between object and the community by the division of labour (Engeström, 1987). The rules, the community and the division of labour give structure to the activities at the systemic level (Engeström, 1987). However, these components in the activity system are not always visible at a systemic level but more tangible at a local level (Engeström, 1998). Sometimes the activities give rise to inner tensions and contradictions (Engeström, 1987). Engeström (2001, p. 134) has described a contradiction as “characterised by ambiguity, surprise, interpretation, sense-making, and potential for change”, and has proposed four levels of contradictions: Primary (within the nodes), secondary (between the nodes), tertiary (when more advanced object of activity is introduced), and quaternary (between the core activity system and outside activity systems) (Madyarov & Taef, 2012). The peer review process is seen as an activity system in which the assessment of the quality of OER is the object of the activity and the report to the EU produced during the process is the outcome. The subjects are the peer reviewers and the instruments are represented by the OER and the