This article was published on this day 3 years ago. I’ve had to clean up a few links and delete some out-of-date references, but otherwise the article still makes some useful points about assessing Computing and related subjects using rubrics.

Some years ago I was approached by Julie Lindsay and Vicki Davis with an invitation to be a "meta-judge" on the Horizon 2008 Project. It was a great honour to be asked, and I hope my judgements are received in a positive way.

But, as usually the case with this sort of thing, it did raise doubts in my mind about the value of rubrics for this type of activity. They are useful, but they are also limited, and not nearly as objective as one might think.

The Horizon Project involved students from several countries collaborating with each other to do research into how modern technology is affecting various aspects of modern life (government, education, health and others). The end product, besides the wiki itself, was a video submitted by each student. These have been judged by a number of educationalists, who decided on the winner in each of the 13 categories. My role as "meta-judge" was to decide which of these 13 finalists was the ultimate winner.

I have to say that this was not an easy task despite having the rubric to guide me. It isn't easy on a human level, if I can put it that way. The trouble with identifying one winner is that by doing so you automatically identify 12 "losers"! I would hope that those 12 don't see it that way. The quality of all the videos was extremely high, and there are even one or two that didn't come out on top that I will have no hesitation in using in my own work (with full credit and citation given, of course). To end up as one of just 13 finalists is good going, and all of the students should feel proud of themselves.

Indeed, even those students not in the final line-up did a fantastic job. The wiki (now sadly no longer extant) boasted a cornucopia of ideas and resources, almost all of which were put together by the students.

The rubric I used was called Rubric 1, Multimedia Artifact. As rubrics go, it wasn’t bad at all. It was shorter than many, which is good, because the longer, ie more detailed, they are, the more easy they are to apply, but the less meaningful they become. The reason is that once you start breaking things down into their component parts, you end up with a tick list of competencies which, taken together, may not mean very much at all. That is because the whole is nearly always greater than the sum of its parts, so even if someone has all of the individual skills required or, as in this case, has carried out all of the tasks required, the end result may still not be very good. So you end up having to use your own judgement about how to grade something, which is exactly what a rubric is meant to avoid in the first place. Let me give you a concrete example.

One of the sentences in the rubric read:

"Content is constructed from a superficial synthesis of information on the wiki."

That seems straightforward enough, until you come across a case where the content on the wiki page is itself superficial -- in which case the right thing for the student to have done would have been to ignore the wiki page all together and put in some fresh insights. But if they had done that, they wouldn't get credit for using the information on the wiki page. In other words, it's a no-win situation which actually penalises the student who exercises her own judgement.

I think the main problems with rubrics in general can be summarised as follows:

1. Do the individual criteria reflect what it is we are trying to measure? This is (broadly speaking) the problem of validity.

2. Are the criteria "locked down" sufficiently to ensure that the rubric yields consistent results between different students and between different assessors (judges)? This is known as the problem of reliability.

3. Are the criteria too "locked down", which could lead to an incorrect overall assessment being made (the validity problem) or assessors introducing their own interpretations to aid the process of coming to a "correct" conclusion (the reliability problem)?

4. Does the rubric emphasise process at the expense of product? It is often said that in educational ICT, it's the process that's important. Well actually, that is not entirely true, and we do young people a grave disservice if we fail to tell them so. If you don't agree with me, that's fine, but I invite you to consider two scenarios, and reflect which one is the most likely to happen in real life:

Imagine: Your Headteacher or Principal asks you to write a report on whether there is a gender bias in the examination results for your subject, in time for a review meeting next Wednesday. You can't find the information you need, so you write a report on the benefits of blogging instead. You desktop publish it so it looks great, and even burn it onto a CD for good measure. To add the icing on the cake, you even make a 5 minute video introducing the topic in order to get the meeting off to a flying start.

Scenario 1:

The boss says:

"Wow, that is fantastic. It's not what I asked for at all, but let's face it, it's the process that's important. Let me raise your salary."

Scenario 2:

The boss says:

"What is this? I asked you to produce a report on gender issues. If you can't follow a simple instruction like that, do you really think you're cut out for this job?"

OK, I know that both responses are slightly far-fetched, but hopefully I've made my point.

Which also leads me on to another thing. I think some of my judgements may have come across as a bit uncompromising. But I really do not see the point of saying something like "Great video", or even "Poor video", without adding enough information for the student to get a good idea of why it was good or poor, and how to improve their work and take it to the next level in the rubric.

Getting back to the issue of interpretation, I am afraid that, in the interests of better accuracy and of giving the students useful feedback, I introduced some of my own criteria. Well, I was the sole meta-judge, a title so grand that I felt it gave me carte blanche to interpret the rubric as I saw fit. Lord Acton was right: absolute power really does corrupt absolutely .

The extra criteria I applied were as follows:

1. Did the medium reflect the message?

To explain what I mean by this, let me give you an example of where it didn't. In one of the videos, the viewer was shown some text which said that businesses can now make predictions. This was then followed by a photograph of chips used in casinos. So, unless the video was intended to convey the idea that predictions can now be made which are subject to pure chance, which I somehow doubt, that was a completely inappropriate message.

2. Could I learn what I needed to know about the topic without having to read the wiki? If not, then I would be at a loss to explain the point of having the video, unless question #3 applied. This includes the question: is the information given actually meaningful? Look at that point about businesses can now make predictions. Businesses have always made predictions, so that statement tells me nothing. What I want to know is, how does communications technology aid forecasting, and does it make the process more accurate?

3. Did the video inspire me to want to find out more, or to do something, even though there wasn't much substance to it? If so, and if that was at least partly the aim, maybe that would be perfectly OK. I'd take some convincing though.

4. Did the video only synthesise the information on the wiki, or did it do more? The word "synthesise" implies adding value in some way: it's more than merely "summarise". But if if the information was of a poor quality, did the student deal with the matter effectively or merely accept the situation?

5. In every case I watched the video first, and then read the wiki, because I wanted to come to it with as few preconceived ideas as possible, to see if the video was able to stand on its own. I then read the wiki and then re-watched the video (sometimes more than once), looking for specific things.

If you have any views on using rubrics, I'd love to hear them -- especially if you completely disagree with anything I've said in this post!

* The Computers in Classrooms newsletter was launched in the year 2000, and is now called Digital Education. It has loads of subscribers, who say things like:

“Many thanks Terry,  for your contributions. I am pointing my students to your site.”

“WOW! I don't know what possessed me to link to your newsletter, Terry, but you have captured, in the sample issue sent to me, what seems to be the "top ten" list of significant educational issues and boiled them down into - "manageable undertakings?!" Don't let the teaching colleges find out about this, or they will run you out of town.”

“I greatly respect the ethics you bring to your work!!!!”

It’s free to subscribe, and there’s more information here: Newsletters.

This is a modified version of an article that was originally published on 9th June 2008.