Research assessment and bibliometrics

The Higher Education Funding Councils in the UK have issued an announcement on a pilot excercise (involving twenty-two UK universities) on the use of bibliometrics in the new "Research Excellence Framework", which will take over from the Research Assessment Excercise now underway.

[As an aside, it looks as though the marketing men have infiltrated the HEFC - "Research Assessment Exercise" was obviously far too explicit for them and so it has to be something new that completely hides what is actually going on - just as the "Committee of Vice Chancellors and Principals" became the totally fuzzy "Universities UK"! Makes one wonder about the intelligence of those at the top of the academic tree.]

However, back to the message. The announcement points to another document, Bibliometrics and the Research Excellence Framework. This tells us how the exercise will actually be carried out. Research output data will be collected from the participating institutions (why is this necessary, given that the HEFC already has such data for the current RAE?) and processed by Evidence Ltd., a data processing company based in Leeds.

Both documents express caution in using bibliometric indicators and the point is specifically made that journal impact factors will not be used. The bibliomtric indicators for each institution in each field will be 'normalised' by comparison with the "field norm", that is "the average number of citations for all papers published worldwide in the same field, over the same period". This is where Evidence Ltd. will need to be very careful indeed, since what constitutes the "same field" is open to wide interpretations. It will be especially risky to rely upon the journal groupings used by Web of Knowledge and SCOPUS to defined the "field". I referred in my earlier Weblog to this problem as far as defining the field of "Information Science & Library Science" is concerned, and I have no doubt that similar problems exist in other fields.

"Bibliometrics and the Research Excellence Framework" also notes that, because of the difficulty of using bibliometric indicators across all disciplines, "other indicators" will also be used. But we are not told what these "other indicators" might be - perhaps they don't actually know yet? The document also proposes the use of a "citation profile" which will show how the papers produced by a particular institution relate to "worldwide norms", so that papers are labelled, for example, "Below world average" or "Above world average". Quite what this means is difficult to understand - does HEFC seriously believe that this would be anything other than a completely arbitrary measure? Especially in social science fields, which are very much culture-bound, comparison of work done in the UK, with work carried out "worldwide" - which would actually mean (because of the volume of output) "carried out in the USA" - would simply result in nonsense.

Having retired from having anything to do with the administration of higher education I shall gaze on, fascinated, by what might emerge :-)