The problem of identifying research with impact

The deadline for submissions to the Research Excellence Framework (REF) from UK higher education institutions passed on 29 November 2013.  Academics around the country breathed a big sigh of relief and returned to normal life (i.e., preparing for the next exercise in research evaluation).

Research evaluations of this sort began in the UK in 1986 and play a major part in the allocation of research funds.  Up until the REF they relied very heavily on citations (eg, Norris & Oppenheim, 2003).  When the Research Assessment Exercise (RAE) was replaced with REF, the new evaluation attempted to shift the emphasis away from citations to other forms of impact.

The obvious advantage of citation-based measures of impact is that they are relatively simple to calculate and to organise, making comparisons possible.  Amongst the disadvantages is the fact that publications that are highly cited in the few years following their publication are often cited because they are convenient summaries of existing research and say little that is original.  Conversely, truly ground-breaking ideas often have no detectable impact in their first few years because their implications are not understood.

A classic example of such a case is that of Gregor Mendel's paper on hybridisation, which is generally regarded as laying the foundation for genetics.  It was published in 1865 but remained almost uncited throughout the remainder of the century ("about three times" according to Wikipedia 06/01/14, and twice according to Google Scholar).

Boris Belousov suffered similar lack of recognition but took it more to heart.  He tried for several years throughout the 1950s to publish a description of an interesting reaction he had observed, but reviewers considered it to be impossible.  It is now recognised as a discovery which forced "a change of perspective and emphasis". (Winfree, 1984)

Norris, M., & Oppenheim, C. (2003). Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 59(6), 709-730.

Winfree, Arthur T. (1984), The Prehistory of the Belousov-Zhabotinsky Oscillator, Journalof Chemical Education 61(8) 661-663.

Summoning of demons: an inappropriate response to criticism

Life sometimes gets in the way of blogging, so I haven't written anything here for a while.  The last entry (by Mary Anne Kennan) discussed the assumptions that underpin research methodologies.  The group discussion that followed its publication touched on some of the problems that arise when such assumptions are not properly examined.  One example that I proffered was the issue of research undertaken according to one methodology being assessed by reviewers following a different methodology.

A few months ago I published a blog entry in which I included some reviews of a paper that was subsequently published. The research being reported on was interpretivist, but it was reviewed as though it were positivist.  Consequently, the reviews were hostile.

Amongst the things that have got in the way of blogging recently has been Christmas, with its abundance of distractions.  One of the smaller distractions was Mark Gatiss' production of the M.R. James story "The Tractate Middoth".  Much of it took place in a library, and the vital clue to a missing will was an accession number.

As a child, I read quite a few M.R. James stories but did not remember this one.  I set out to discover and rediscover some.  One that I discovered was James' story, Casting of the Runes, which the inspiration for a 1950s horror film called Night of the Demon (renamed 'Curse of the Demon' in the USA).  In the film, the leader of a Satanic cult uses his knowledge of the occult to summon up demons to destroy his enemies.  I had seen the film but not read the story.  I did so and found that, in the original, the occult powers were used in response to bad reviews of the author's work. 

(Re)opening a discussion about our methodological landscape (by Mary Anne Kennan, Charles Sturt University)

It is uncommon for published research to articulate the meta-theoretical assumptions which underpin different paradigms/approaches/traditions of research. Instead, methodological discussion tends to focus on research methods, techniques and tools. As researchers however, our methodological assumptions determine 

• our choice of research paradigm; 
• the formulation of, and the way in which we consider and address our research questions; 
• and also our selection of methods and techniques.

In this discussion we will consider the pros and cons of adopting a broader view of research methodology.  This includes being specific about meta-theoretical assumptions and their importance for achieving greater reflexive awareness of the “unconscious metaphysics” that underlie and influence how we see and research the world.  To do this we need to make a clear distinction between the concept of methodology as an overall logic of inquiry, and research method as a much narrower concept that defines processes, procedures and techniques that can be used to conduct empirical studies and collect and analyse data (Cecez-Kecmanovic, 2011).  To achieve this clear distinction, it is necessary to make explicit the assumptions and paradigms on which different methodologies are founded.

Methodological issues are discussed therefore, within a broader landscape which includes more than just research methods and techniques, but also addresses what lies behind them.  By discussing the issues in this context, we can ask questions such as:

How does our view of the world and meta-theoretical foundation influence the type of questions we will ask (and answers we will find) as researchers?

How does our world view open (and/or limit) the potential methodological paths that we will choose from?

How does being methodologically explicit help us to select and justify the research methods we choose to answer our questions?

References

Cecez-Kecmanovic, D. (2011). On methods, methodologies and how they matter. Paper presented at the European Conference on Information Systems, Helsinki, Finland, 9-11 June 2011. Retrieved from http://aisel.aisnet.org/ecis2011/233

Cecez-Kecmanovic, D. & Kennan, M.A. (2013) Chapter 5. The methodological landscape: Information systems and knowledge management in Research Methods: Information, Systems, and Contexts Williamson, K. & Johanson, G. (eds.). Tilde Publishing, Prahran, Victoria. pp113-138 ISBN 978-0-7346-1148-2

Giving an oral presentation (by J.E.A. Wallace)

All researchers need to learn to communicate details of their projects to a wider audience.  Obviously, different audiences require different approaches but to try to give some perspective, here are a few hints and tips gleaned from the experiences I’ve had with my own project thus far.

Firstly, and perhaps most importantly: know your audience.   This is not so much about the tone (although over-familiarity is a really bad idea in a formal setting!), but rather about the content of your talk. When presenting to your own group, there is perhaps a level of common knowledge that means that you can assume the basics and only give abbreviated reports on progress.   However, in any other context, it is best to assume that the audience has little knowledge of the specifics of your work. To that end, a brief introduction is useful to set the scene before getting into the details.

Secondly, pacing and timing are key.  A talk divided into clear sections holds the interest of an audience better than a long talk without breaks. It also makes it easier to rehearse timings and to be flexible.   Both are useful in preparing to talk in a formal setting such as a conference.  In such a setting, the Chair will offer some sort of signal near to the end of your slot, or a clock will be made available.  In either case, rehearsing timings beforehand is useful.  Do bear in mind though, that people tend to talk more quickly when nervous.

On the subject of nerves, remember that the audience is there to hear what you have to say. Everyone in the room is on your side so if there are any minor slip-ups, focus on your work and just keep going.  No-one will object if you need to take a moment to gather your thoughts mid-talk.

Lastly, don’t dread question sessions – they are usually a great opportunity to engage with the audience and possibly to establish some beneficial networking connections. From personal experience, these sessions have led to many worthwhile discussions after the fact, and to one worthwhile collaboration.

If anyone has any other experiences or additional suggestions  for presentation, please feel free to contribute in the comments.

Back to basics

Following last month's discussion, James Wallace made the useful observation that, since the bulk of people attending the researchers' group are PhD students, it would be helpful to have occasional talks on some of the basic skills needed for any researcher.  Two subjects he suggested were presenting and posters.

Not only was James forthcoming with his suggestions, he was also prepared to back them up by volunteering his services.  He kindly offered to lead this month's discussion on presenting.  He provided a useful blog entry in advance of the session which I took the liberty of editing a little.  James' original entry asserted that it is important for young researchers to develop the ability to communicate details of their projects.  I edited it to read "All researchers need to learn to communicate details of their projects to a wider audience."

Filtering the Web (by Simon Wakeling)

The emergence of the internet has allowed its users a previously unimagined access to information. A corollary effect of this information deluge has been the notion of information overload – the difficulty those users face finding, filtering and processing relevant information from amongst the sea of available content. The web services that have emerged to connect us with this information, from search engines to social networks, have in recent years increasingly turned to personalisation as a means of helping users successfully navigate the web. 

In many cases this personalisation takes the form of algorithmic content filtering, with implicit user feedback channels ranging from geography, to click history, to the activity of other users within social networks used to determine the information presented to the user (indeed it has been suggested that Google uses up to 57 different data sources to filter its content). An unintended consequence of this process, which it should be noted is rarely made explicit to the user, is to gradually and invisibly disconnect information seekers from contrary content. A much used example concerns the political liberal, whose regular consumption of left-wing sources and active online network of similarly minded friends eventually sees any conservative material excluded from his search results and news feeds. The result is a “filter bubble” – a term coined by Eli Pariser to describe the self-perpetuating results of the information intermediator’s algorithm. Pariser sees web users inhabiting “parallel but separate universes” in which the absence of the alien undermines objectivity.  

It is interesting to note that philosophers have already begun to engage with this issue. From an epistemological perspective, Thomas Simpson has noted that in an age when search engines are increasingly seen as “expert”, an inbuilt and invisible confirmation bias increases the potential for beliefs to be formed on the basis of “reasons unconnected with the truth”. Others have begun considering the ethical implications. Is the filter bubble a form of censorship? Do we as users have a right to expect objectivity from the systems we use?
 
These are interesting questions for Information Scientists to consider. First though, as scientists we might want to investigate if indeed the phenomena exists, and if it does how it can best be measured and its implications studied. Is it necessarily a bad thing? We might also begin imagining solutions. An example of work already undertaken in this area is Balancer, which presents a stick man leaning under the weight of the “liberal” or “conservative” news sources the user has accessed.  It’s certainly novel, despite its limitations and acknowledged imperfections. Perhaps though the significance of the Filter Bubble problem requires more revolutionary solutions. We just need to work out what they are…

Decision making for microbes

A long, long time ago, in a life far far away, I spent several years working as a researcher in the life sciences.  When I shifed into information sciences, I found myself looking at aspects of the subject from a biologist's perspective.  

One of the aspects that I found myself considering was that of decision making.  Living things benefit if they can make best use of the resources that surround them.  To do so, they need information.  Or so I suggested in a paper that I wrote ten years ago (Madden, 2004) in which I argued that when organisms evolved the ability to move (and possibly before), they evolved a need for information.  

As is always the case at the discussion meetings, the talk went in some interesting directions.  Unsurprisingly I was pulled up for my lax use of the term "World View".  I had quoted Checkland's assertion that "Judging from their behaviour, all beavers, all cuckoos, all bees have the same W[orld View], whereas man has available a range of W[orld View]s." (Checkland, 1984, p218).  It was, I conceded in the discussion, a questionable assertion that I had not questioned.

Checkland, P.B. (1984). Systems Thinking, Systems Practice 2nd Edn. Chichester:

Wiley.

Madden, A. D. (2004). Evolution and information. Journal of Documentation,60(1), 9-23.