Robustness of journal rankings by network flows with different amounts of memory

As the number of scientific journals has multiplied, journal rankings have become increasingly important for scientific decisions. From submissions and subscriptions to grants and hirings, researchers, policy makers, and funding agencies make important decisions influenced by journal rankings such as the ISI journal impact factor. Typically, the rankings are derived from the citation network between a selection of journals and unavoidably depend on this selection. However, little is known about how robust rankings are to the selection of included journals. We compare the robustness of three journal rankings based on network flows induced on citation networks. They model pathways of researchers navigating the scholarly literature, stepping between journals and remembering their previous steps to different degrees: zero‐step memory as impact factor, one‐step memory as Eigenfactor, and two‐step memory, corresponding to zero‐, first‐, and second‐order Markov models of citation flow between journals. We conclude that higher‐order Markov models perform better and are more robust to the selection of journals. Whereas our analysis indicates that higher‐order models perform better, the performance gain for higher‐order Markov models comes at the cost of requiring more citation data over a longer time period.