Following our last researchers' discussion, the pub conversation again turned to evaluation. We talked about the problems caused by applying simple measures to public bodies, and I rashly suggested a solution - that a second simple measure should be introduced, that is 'orthogonal' to the first.
For the purposes of analysis, I had recently been revisiting an interview with a school teacher. She (like many of my interviewees) was reflecting sadly on the extent to which educational values are skewed by the need to "teach to the exam". In other words, students are taught to anticipate exam questions and to frame their efforts with a view to producing a high scoring answer. Any school activity not related to exams is marginalized.
Other examples of evaluation-induced skewing were discussed (including the forthcoming Research Excellence Framework) and there was some head-shaking about the damage such an exercise will do to Higher Education.
The problem is common where simple measures are applied to complex systems. Simple measures are often simply met; and if meeting the measure is all that counts then other important criteria get neglected. I heard a macabre example of such skewness recently (27 August) on BBC Radio 4. Amanda Vickery, in a programme about sailors, interviewed Dr David Turner of Swansea University about disability in the British navy in the 18th Century. He observed that, back then, there were concerns "...that surgeons might be a little too willing to amputate limbs in cases where this wasn't altogether necessary. There are some complaints that naval and military surgeons cut off limbs because they are paid by the number of limbs that they amputated..." (12min 15sec).
Orthogonality
Any second measure should be as different as possible from the first, but should be given equal importance. In the case of schools for example, exam success is an obvious criterion for assessment; but a second criterion could be based on the response of school leavers to a survey designed to assess the school's success in promoting involvement and inclusion. The survey could include statements such as the following:
School funding and reputation are currently strongly linked to exam performance. If they were as strongly linked to positive responses to statements such as those above, decisions about a school's priorities would be based on a far more nuanced view of success.
For the purposes of analysis, I had recently been revisiting an interview with a school teacher. She (like many of my interviewees) was reflecting sadly on the extent to which educational values are skewed by the need to "teach to the exam". In other words, students are taught to anticipate exam questions and to frame their efforts with a view to producing a high scoring answer. Any school activity not related to exams is marginalized.
Other examples of evaluation-induced skewing were discussed (including the forthcoming Research Excellence Framework) and there was some head-shaking about the damage such an exercise will do to Higher Education.
The problem is common where simple measures are applied to complex systems. Simple measures are often simply met; and if meeting the measure is all that counts then other important criteria get neglected. I heard a macabre example of such skewness recently (27 August) on BBC Radio 4. Amanda Vickery, in a programme about sailors, interviewed Dr David Turner of Swansea University about disability in the British navy in the 18th Century. He observed that, back then, there were concerns "...that surgeons might be a little too willing to amputate limbs in cases where this wasn't altogether necessary. There are some complaints that naval and military surgeons cut off limbs because they are paid by the number of limbs that they amputated..." (12min 15sec).
Orthogonality
Any second measure should be as different as possible from the first, but should be given equal importance. In the case of schools for example, exam success is an obvious criterion for assessment; but a second criterion could be based on the response of school leavers to a survey designed to assess the school's success in promoting involvement and inclusion. The survey could include statements such as the following:
- If my school held a reunion next year, I would want to come back for it.
- My school allowed me to take part in at least one sport that I enjoyed.
- If something bad happened to me, there were people I could talk to at school.
- My teachers were interested in what I had to say.
- While I was at school, I took part in performances (eg, concerts or plays).
- I never felt threatened by the behaviour of other students.
School funding and reputation are currently strongly linked to exam performance. If they were as strongly linked to positive responses to statements such as those above, decisions about a school's priorities would be based on a far more nuanced view of success.
No comments:
Post a Comment