Citations and performance evaluation

Gerry McKiernan has drawn attention on the BOAI discussion list to a special issue of Ethics in Science and Environmental Politics, devoted to 'The use and misuse of bibliometric indices in evaluating scholarly performance'. All of the papers are open access and the authors include Philip Campbell, Editor in Chief of Nature on escaping from the Impact Factor and concluding, Although the current system may be effective at measuring merit on national and institutional scales, the most effective and fair analysis of a person’s contribution derives from a direct assessment of individual papers, regardless of where they were published; Peter Lawrence on how measurement harms science; Anne-Wil K. Harzing and Ron van der Wal on Google Scholar as a new source for citation analysis; and Stevan Harnad on Validating research performance metrics against peer rankings. There are fourteen papers in all including the introduction to the issue and all are worth at least dipping into.

The Harzing and van der Wal paper is likely to be somewhat contentious, since other research into the use of Google Scholar has downplayed its value. I first raised the question of using Google in performance measurement in a message to the JESSE mailing list in 2002. I commented that:

My most cited paper is "On user studies and information needs" (1981) - a Web search (using Google) revealed 118 pages that listed the title. The pages were reading lists, free electronic journals, and documents that would never be covered by SSCI, such as reports from various agencies. SSCI revealed, if I recall aright, 79 citations of the paper. The question is: is the Web revealing impact more effectively than SSCI? Citation in scholarly papers takes a variety of forms and much citation is of a token variety - x is cited because x is always cited. On the other hand citation on reading lists implies some positive recommendation of the text, and mention in policy documents and the like, implies (at least in some cases) that some benefit has been found in the cited document.

This led to an extended and interesting discussion. It was followed by a paper on the subject in JASIST by Vaughan and Shaw (Volume 54, Issue 14, 1313-1322) and by a comparison of Google Scholar and Web of Science by Peter Jasco in Current Science, v.89, no. 9, 1537-1547 - incidentally, Harzing and van der Wal reply to some of Jasco's criticisms of Scholar.

All of this suggests that interest in the application of 'metrics' in performance measurement is growing and, perhaps, there is also a growing awareness that the impact factor provided by Web of Knowledge is not necessarily the only tool that can be used. We have also seen the emergence of the Scimago analysis of impact, which I have referred to earlier and these developments, as well as providing for a new academic industry of papers on Web citation, are likely to bring about a re-assessment of metrics for performance measurement and, perhaps, greater wariness about their use than is currently shown by some administrators.