Preliminary investigation on quantitative evaluation method of scientific papers based on text analysis

Recently, there are many resources of scientific research on the Web. Because the gap between the amount of academic information available on the Web and human processing abilities becomes large, several problems have arisen: (1) losing opportunities of research presentation, (2) loosing opportunities of gathering research information, (3) increasing burden of peer review, (4) difficulty in selecting papers to read. In order to solve these problems, quantitative evaluation index of a paper as a selection criterion is needed. This paper proposes quantitative evaluation methods of scientific papers on the basis of text analysis. The journal similarity of a target journal to an authoritative journal is defined with using distributed representations of papers. When the similarity of a target journal is high, its quality in terms of writing and organization is expected to be high. This paper also proposes an evaluation method using ROUGE (Recall-Oriented Understudy for Gisting Evaluation). Proposed evaluation methods are evaluated by experiments. Experiments results show that the journal similarity has rough correspondence to Scimago Journal Rank (SJR). The result also implies the possibility of evaluating journals that have not yet been indexed in some authoritative journal indices using the proposed methods. The evaluation method using ROUGE is shown to have the possibility of evaluating the consistency of papers.