A Guide to Conducting Systematic Reviews in Agri-Food Public Health

s should be independently assessed by at least 2 reviewers to increase the reliability of the inclusion and exclusion process. If both reviewers determine that the abstract does not meet inclusion criteria, it can be rejected. If the two reviewers disagree, the conflict should be resolved by consensus or, if this is not possible, by a third reviewer. Disagreements and their resolution should be documented. When a reference is passed as potentially relevant, then a full-text copy of the reference is obtained and the article moves on to the quality assessment stage. The references that do not meet the screening requirements are eliminated from the review. The references for which screening questions cannot be answered due to lack of information should remain in the study and efforts should be made to obtain further details of the study. The reproducibility of this process should be tested in the initial stages of the review (we evaluated agreement after all of the reviewers had screened the same 50 abstracts). If reproducibility is shown to be poor, rewording of the questions with more explicit criteria should be undertaken. The agreement between reviewers can be statistically evaluated using Cohen’s Kappa. If there are a large number of abstracts identified by the search strategies, the two reviewers may screen abstracts in parallel (i.e. the abstract passes relevance screening and moves on to the quality assessment stage as soon as one reviewer passes the abstract). This can greatly reduce the time necessary for the relevance screening stage. However, duplicate relevance screening should still be done initially to allow validation of the relevance screening questions, as described above. The ESR software allows not only the storage and distribution of references and full documents but also the design of electronic review forms, the assignment of reviewers to specific citation subsets or forms, the capture and reporting of reviewers input, automated inclusion and exclusion of references, and evaluation of reviewer agreement.