The problem of identifying research with impact

The deadline for submissions to the Research Excellence Framework (REF) from UK higher education institutions passed on 29 November 2013.  Academics around the country breathed a big sigh of relief and returned to normal life (i.e., preparing for the next exercise in research evaluation).

Research evaluations of this sort began in the UK in 1986 and play a major part in the allocation of research funds.  Up until the REF they relied very heavily on citations (eg, Norris & Oppenheim, 2003).  When the Research Assessment Exercise (RAE) was replaced with REF, the new evaluation attempted to shift the emphasis away from citations to other forms of impact.

The obvious advantage of citation-based measures of impact is that they are relatively simple to calculate and to organise, making comparisons possible.  Amongst the disadvantages is the fact that publications that are highly cited in the few years following their publication are often cited because they are convenient summaries of existing research and say little that is original.  Conversely, truly ground-breaking ideas often have no detectable impact in their first few years because their implications are not understood.

A classic example of such a case is that of Gregor Mendel's paper on hybridisation, which is generally regarded as laying the foundation for genetics.  It was published in 1865 but remained almost uncited throughout the remainder of the century ("about three times" according to Wikipedia 06/01/14, and twice according to Google Scholar).

Boris Belousov suffered similar lack of recognition but took it more to heart.  He tried for several years throughout the 1950s to publish a description of an interesting reaction he had observed, but reviewers considered it to be impossible.  It is now recognised as a discovery which forced "a change of perspective and emphasis". (Winfree, 1984)

Norris, M., & Oppenheim, C. (2003). Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 59(6), 709-730.

Winfree, Arthur T. (1984), The Prehistory of the Belousov-Zhabotinsky Oscillator, Journalof Chemical Education 61(8) 661-663.

Summoning of demons: an inappropriate response to criticism

Life sometimes gets in the way of blogging, so I haven't written anything here for a while.  The last entry (by Mary Anne Kennan) discussed the assumptions that underpin research methodologies.  The group discussion that followed its publication touched on some of the problems that arise when such assumptions are not properly examined.  One example that I proffered was the issue of research undertaken according to one methodology being assessed by reviewers following a different methodology.

A few months ago I published a blog entry in which I included some reviews of a paper that was subsequently published. The research being reported on was interpretivist, but it was reviewed as though it were positivist.  Consequently, the reviews were hostile.

Amongst the things that have got in the way of blogging recently has been Christmas, with its abundance of distractions.  One of the smaller distractions was Mark Gatiss' production of the M.R. James story "The Tractate Middoth".  Much of it took place in a library, and the vital clue to a missing will was an accession number.

As a child, I read quite a few M.R. James stories but did not remember this one.  I set out to discover and rediscover some.  One that I discovered was James' story, Casting of the Runes, which the inspiration for a 1950s horror film called Night of the Demon (renamed 'Curse of the Demon' in the USA).  In the film, the leader of a Satanic cult uses his knowledge of the occult to summon up demons to destroy his enemies.  I had seen the film but not read the story.  I did so and found that, in the original, the occult powers were used in response to bad reviews of the author's work.