Evaluating evaluation
Created by Catherine Howell (University of Cambridge) on October 31, 2005
Our new Evaluation Group (are we a group? or a team? a division? -- that gives an idea of just how new this is) is launched as of this week. This is definitely a "soft launch", but it does represent a slight change of direction for CARET; or rather, less a change of direction than a more clearly defined focus for activities that we've been pursuing for some time, albeit under different banners.
The creation of the group represents an opportunity to examine alternative approaches to evaluation as a discipline, to reflect on our own practice, and to articulate our collective philosophy. Early days, but here's some Stuff We Like: action research, social network analysis, mixed qual/quant approaches, iterative design, teachers / students as designers.
My colleague Patrick Carmichael favours the strongly theory-driven approach of these Helsinki-based researchers. Those Finns are doing great things, bringing social network analysis together with management theory and activity theory... neat stuff, but complex (a bit like reading Anna Sfard). We're also looking at contrasting approaches taken by the MIT folks, ILRT Bristol, and Erica McAteer and colleagues up in Scotland.
I'd like to be a little provocative now, and cite five things we're NOT interested in:
1. Promoting particular products. Just because a product occupies a large market niche among students does not make it the best fit for teaching and learning. Duke University, ahoy!
2. Basing our student teaching and learning support on our perceptions of what students do in their spare time.
3. Summarising qualitative data from small user samples ... and calling that an evaluation.
4. Basing data solely on exam results.
5. Listing deliverables for a specific project, describing processes taken in order to achieve deliverables, and calling that an evaluation. It's not an evaluation: it says nothing about fitness for purpose, the user experience, or documented changes in user behaviour / institutional culture / attitudes and expectations.
The creation of the group represents an opportunity to examine alternative approaches to evaluation as a discipline, to reflect on our own practice, and to articulate our collective philosophy. Early days, but here's some Stuff We Like: action research, social network analysis, mixed qual/quant approaches, iterative design, teachers / students as designers.
My colleague Patrick Carmichael favours the strongly theory-driven approach of these Helsinki-based researchers. Those Finns are doing great things, bringing social network analysis together with management theory and activity theory... neat stuff, but complex (a bit like reading Anna Sfard). We're also looking at contrasting approaches taken by the MIT folks, ILRT Bristol, and Erica McAteer and colleagues up in Scotland.
I'd like to be a little provocative now, and cite five things we're NOT interested in:
1. Promoting particular products. Just because a product occupies a large market niche among students does not make it the best fit for teaching and learning. Duke University, ahoy!
2. Basing our student teaching and learning support on our perceptions of what students do in their spare time.
3. Summarising qualitative data from small user samples ... and calling that an evaluation.
4. Basing data solely on exam results.
5. Listing deliverables for a specific project, describing processes taken in order to achieve deliverables, and calling that an evaluation. It's not an evaluation: it says nothing about fitness for purpose, the user experience, or documented changes in user behaviour / institutional culture / attitudes and expectations.
Bookmark/Search this page with:
