It was nice to learn in last month's discussion of Amazon's Mechanical Turk. It seems that it's old knowledge amongst the programmers in the department, but its existence was news to me. Research ethics is always a good topic around which to generate heated discussion, and Mark succeeded - though his ethical dilemma (as expressed in his blog entry) was not one that others in the group shared. The discussion did however, take a (to me) more interesting angle when it switched to the influence of samples of convenience on research outcomes.
One of Mark's concerns was that participants were being paid significantly below the minimum wage. Given that responses were being elicited from around the world however, the question was asked: "Minimum wage for where?" In some parts of the world, the return for effort was (by the local standards) excellent. These parts of the world were, indeed, highly represented in the results.
Given the extent to which IR research findings have been based on western (usually English speaking) volunteers, studies such as Mark's may help to redress the balance and produce more robust findings.
I found myself guilty of drawing inappropriate conclusions based on an unrepresentative sample earlier this week, when I looked through some student evaluation forms. I collected the forms in after a seminar. It was the second of two that I had taken, and I was pleased to read on a number of the forms that the respondents enjoyed the seminars and felt that there should be more of them. What I failed to consider was that most of the students failed to turn up. The absent students had not completed forms, so the feedback reflected only the view of the minority.
One of Mark's concerns was that participants were being paid significantly below the minimum wage. Given that responses were being elicited from around the world however, the question was asked: "Minimum wage for where?" In some parts of the world, the return for effort was (by the local standards) excellent. These parts of the world were, indeed, highly represented in the results.
Given the extent to which IR research findings have been based on western (usually English speaking) volunteers, studies such as Mark's may help to redress the balance and produce more robust findings.
I found myself guilty of drawing inappropriate conclusions based on an unrepresentative sample earlier this week, when I looked through some student evaluation forms. I collected the forms in after a seminar. It was the second of two that I had taken, and I was pleased to read on a number of the forms that the respondents enjoyed the seminars and felt that there should be more of them. What I failed to consider was that most of the students failed to turn up. The absent students had not completed forms, so the feedback reflected only the view of the minority.
No comments:
Post a Comment