Wednesday, 15 January 2014

The problem of identifying research with impact

The deadline for submissions to the Research Excellence Framework (REF) from UK higher education institutions passed on 29 November 2013.  Academics around the country breathed a big sigh of relief and returned to normal life (i.e., preparing for the next exercise in research evaluation).

Research evaluations of this sort began in the UK in 1986 and play a major part in the allocation of research funds.  Up until the REF they relied very heavily on citations (eg, Norris & Oppenheim, 2003).  When the Research Assessment Exercise (RAE) was replaced with REF, the new evaluation attempted to shift the emphasis away from citations to other forms of impact.

The obvious advantage of citation-based measures of impact is that they are relatively simple to calculate and to organise, making comparisons possible.  Amongst the disadvantages is the fact that publications that are highly cited in the few years following their publication are often cited because they are convenient summaries of existing research and say little that is original.  Conversely, truly ground-breaking ideas often have no detectable impact in their first few years because their implications are not understood.

A classic example of such a case is that of Gregor Mendel's paper on hybridisation, which is generally regarded as laying the foundation for genetics.  It was published in 1865 but remained almost uncited throughout the remainder of the century ("about three times" according to Wikipedia 06/01/14, and twice according to Google Scholar).

Boris Belousov suffered similar lack of recognition but took it more to heart.  He tried for several years throughout the 1950s to publish a description of an interesting reaction he had observed, but reviewers considered it to be impossible.  It is now recognised as a discovery which forced "a change of perspective and emphasis". (Winfree, 1984)

Winfree, Arthur T. (1984), The Prehistory of the Belousov-Zhabotinsky Oscillator, Journalof Chemical Education 61(8) 661-663.

No comments:

Post a Comment