Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service

Citebase is a new citation-ranked search and impact discovery service that measures citations of scholarly research papers which are openly accessible on the Web, i.e. papers that are assessable continuously online. Other services, such as ResearchIndex, have emerged in recent years to offer citation indexing of Web research papers. In the first detailed user evaluation of an open access Web citation indexing service, Citebase has been evaluated by nearly 200 users from different backgrounds. The paper details the procedures used in the evaluation, and analyses the results of this study, which took place between June and October 2002. It was found that within the scope of its primary components, the search interface and services available from its rich bibliographic records, Citebase can be used simply and reliably for the purpose intended, and that it compares favourably with other bibliographic services. It is shown tasks can be accomplished efficiently with Citebase regardless of the background of the user. More data need to be collected and the process refined before it is as reliable for measuring citation impact of indexed papers. Better explanations and guidance are required for first-time users. Coverage is seen as a limiting factor, even though Citebase indexes over 200,000 papers from arXiv. Non-physicists were frustrated at the lack of papers from other sciences. The principle of citation searching of open access archives has thus been demonstrated and need not be restricted to current users. Since the evaluation, Citebase has become a featured service of the ArXiv physics eprint archives.