These three blogs (part I, part II, part III here) are based on a talk I gave at Headteachers’ Roundtable Summit in March 2019. My thoughts on this topic have been extensively shaped by conversations with Ben White, a psychology teacher in Kent. Neither of us yet know what we think!

Our beliefs about our academic ability are often so tightly intertwined with our sense of self that we must take care in how we talk about it. It is only with a clear mental model of how feedback might alter a parent or student’s beliefs and goals that we can understand how to manage risks and enhance potential gains involved in communicating attainment. Inducing competitive behaviour can be enormously helpful in encouraging student effort in these situations where many of the benefits of learning are long-term and poorly appreciated by the learner. People tend to be highly motivated by facing up to social comparisons, and students will make these comparisons whether you give them ranking information or not. However, the mental model I’ve described suggests that pushing the competitive focus too far risks lowering effort or pushing students off the game, especially where they feel there is no prospect of doing well.

Risks in giving clear, cohort-referenced feedback

The model implies there are three situations where giving parents or students clear cohort or nationally-referenced feedback can lower future effort. Firstly, if they receive unexpectedly positive feedback, it could lead to complacency about effort required in the future. In fact, the simple act of learning your place on the curve with greater certainty than you had before could even be unhelpful if it gives you greater comfort that you are where you want to be. Secondly, there is a risk of demoralisation if you come to believe that the effort needed for a small improvement in ranking isn’t worth it. Thirdly, game-switching is a risk if you decide you can achieve better returns by working at something else. I think all these risks, but particularly the third, are deeply culturally situated and framed by how you communicate the value of achievement and hard work. Only you can know whether you have created a school climate where you can keep all your students playing the grading game you create for them.

Risks in giving kind and fuzzy feedback

So what are the risks around the alternative – the fuzzy language we use to talk about attainment? Teachers often suffer from rater leniency (being a little too generous with student grading) when there is room for subjectivity. The mental model shows why ‘kind’ feedback might – or might not – be so helpful. On the one hand, the model highlights why we might want to be lenient. By saying “you’ve learnt a lot”, we are hoping students feel more confident in their ability to learn in the future. This is unambiguously a good thing. Also, by saying “you’ve learnt a lot”, we are hoping to keep vulnerable students engaged in our grading game. However, leniency in the rating of a skill level can equally reduce motivation as it may signal the student has already done enough to get to the position they’d like to be in. Hence, while raising confidence in the ability to acquire a certain skill or achieve an outcome can be beneficial, raising confidence in the skill itself or the level of past achievements can be detrimental.

Withholding clear attainment information from parents and students can also be damaging if it de-prioritises YOUR game in their minds and fails to give them the information they need to ensure they maintain the position on the bell curve that they would like to achieve. Primary schools are the masters of ‘fuzzy’ feedback. I suspect a majority of primary parents are told their child is ‘as expected’ in most schools, yet these parents would respond quite differently to discovering their child was ranked 7/30 versus 23/30 in class. What mental model of the beliefs, capabilities and desires of that child and their parents leads primary schools to believe it is in the family’s interest to withhold clear attainment information? There is nice evidence from elsewhere in the world that shows powerful effects of communicating frequent and transparent attainment grades with parents of younger children. It would be great to have a trial in this country to learn which of our families respond to this information.

Downplaying the importance of prior attainment in the game

One unambiguous finding from this literature is the importance of trying to maintain strong student beliefs in their ability to climb up the rankings through making an effort. The teacher’s dilemma is how to maintain strong beliefs that learning is productive (i.e. you can do this) without telling students they’ve already done well enough (i.e. you’re already there). One implication is that you need to construct a game where feedback scores are truly responsive to changes in effort.

The problem we face is that performance in a test is frequently more strongly determined by prior knowledge/IQ than by recent student effort. Students are frequently rational in appreciating that effort yields few rewards. One approach to lessening the anchoring effect of prior knowledge is to encourage comparisons between students with similar prior attainment. For example, if a school has subject ability-setting, then within-class comparisons promote a competition where effort is more strongly rewarded than do whole-school comparisons. This approach will only be effective though if students ‘buy into’ the within-class games you’ve constructed, of course.

I am frequently asked why we need make comparisons with other students at all. Asking a student to compete with their own past performance avoids many of the problems I have discussed, though we tend to be less motivated by competition with ourselves! Ipsative feedback compares performance in the same assessment of the same domain over time and we frequently use it outside school settings (e.g. my 5km speed this week compared to last). It would be great to see these ipsative comparisons encouraged more in schools, but there are good reasons why their application is limited. When we teach we tend to continuously expand the knowledge domain we wish to assess, making ipsative comparisons less straightforward (except in restricted domains such as times tables). (And since everyone in the class must plough on with learning the curriculum at the same speed, regardless of whether they are ready, mastery assessment approaches where we accumulate a list of competencies as they are reached also aren’t very practical). I think creating strange ‘progress’ measures, fudging within-student comparisons between non-standardised tests from one term to the next, are attempts to encourage students to make these comparisons with themselves. They can certainly be justified by the mental model described in the previous post. (For what it’s worth, though, I am not really convinced they are credible metrics in a game that students actually care about.)

A meaningful game where there are more winners than losers over time

If you want to construct a game where making an effort typically yields a decent return, why on earth would you make it a zero-sum game – as ranking does – where half your class will necessarily be losers despite making an effort? One initial step to avoid this is to create benchmarks that are external to the school, i.e. national reference points (invented, if necessarily), to remove the requirement for there to be losers.

National-benchmarking, such as using standardised scores, still doesn’t ensure effort is typically rewarded though, unless your school happens to be able to outpace the national benchmark. To do this, your system for describing attainment could invent typical grades or levels that rise a little each year as students move through the school, generating a sense that pupils are getting better at your game. And so, our mental model starts to explain why schools invent and re-invent arbitrary levels systems!

But our mental model also asserts that we need our game to feel meaningful to students, with rewards that they value (otherwise they won’t feel inclined to work hard at it). Achieving a ‘Level Triangle’ (or whatever you choose to call it) might not feel meaningful enough to some. Is this how you settle on the idea of using GCSE grades as your levels system, since we know they are a grade which has motivational currency to students? Why not invent fictional internal scales and tell students they are at a GCSE grade 2 in Year 7, grade 3 in Year 8, and so on? Of course, technically this is a nonsense – a 12 year old who hasn’t studied the GCSE specification cannot possibly be a GCSE grade anything.

We find ourselves creating meaningless games, but ones that might be worthwhile because they have better motivational properties than any other game we could invent for our students to play! I hate meaningless data, but I’d find it hard to argue that schools shouldn’t use it if they could demonstrate it increased student effort.

The curious flightpath games we play

This gives us a new perspective on trying to make sense of the flightpath, a 5-year board game where individual students are asked to keep on, or beat, the path we’ve set up for them. It is easy to be dismissive of this game on grounds of its validity, especially when we know how little longitudinal attainment data conforms to the paths we create. But we should also ask whether it is more or less motivating than the termly ranking game or any other grading game we could give them as an alternative.

I’m pretty sure the standard, fixed flightpath that maps student attainment from Key Stage Two data, and is impervious to effort or new information, has poor motivational properties. The motivational properties are poor for those students who discover they are on track for a Grade 3, before they’ve had a chance to work hard in secondary school. They might also be poor for those who are told that, in all likelihood, they’ll attain a Grade 8 or 9. The game card prioritises the signal of prior attainment (bad for motivation) and underplays the importance of effort in reaching any desired goal.

But what about the schools who use dynamic flightpaths, updating each students game card each term or year in light of new effort and attainment information? Suppose that, at all times, the game card also shows the non-zero probability of any grade being attained in the future to signal the importance of effort in achieving goals. Is it possible that this type of dynamic grading can help students create a game it is possible to do well at, preserving useful beliefs about effort being productive, whilst also signalling that more effort is needed to get to the next position they’d like to attain?

This is all speculation – there isn’t any research out there that can tell you the impact of using target grades, predictions or flightpaths in different types of schools. All we can do is to invoke mental models to think through how they might affect motivation (one nice speculation about target grades is by James Theo).

The ethics of telling un-truths

When we construct grading games that prioritise manipulating behavioural responses over the whole-truth about attainment, we have to face up to tricky ethical dilemmas. We face these all the time in schools when we tell the half-truths we do to parents and students about attainment; the exploration of mental models simply makes it more explicit why we do it, and who we might help or damage in the process.

Mental models also make it explicit that one grading system will not suit all students equally well. Slightly over-confident, competitively minded students who are able to figure out how to translate effort into learning would do well in a pure rankings system. They will have classmates who find competition stressful and, even with considerable effort, risk slipping behind each year for reasons entirely outside their control. Those researchers who showed that cohort-referenced grades can improve school exam results also showed they increased inequality in happiness amongst students overall. If there are trade-offs, whose welfare do we prioritise?

Conclusion

Choosing how to give attainment feedback to students and their parents is a minefield, but I hope by now you appreciate that choosing NOT to give clear, interpretable (i.e. often norm-referenced) feedback on how a student is doing is not a neutral position to take. It can be damaging to the motivation of certain students under certain circumstances, and you need a clear mental framework to understand why this happens.

Equally, validity of inference should not be the only concern in working out how you are going to report attainment at school. Systems that look bizarre on the face of it, such as flightpaths, might have an intelligible approach to motivating and managing students’ complex belief systems.

If, on getting to the end of these posts, you feel utterly confused about what it is right to do, I think that’s OK. We can be pretty sure that choosing your grading system isn’t the most important decision a leadership team makes. It is true that many of these studies identify a costless and significantly positive effect of giving attainment feedback, particularly at points in time where the stakes are high or where attainment is not yet well known. However, the overall impact of a change in attainment reporting on end-of-school outcomes will typically be quite small, on average.

Nobody can tell you how you should construct your own grading game. The findings of the literature are inconsistent because the mental models of how we are trying to change student beliefs are very complex. How your students will respond to your grading system through your manipulation of their belief systems strongly depends on your school culture and on localised social norms amongst peers. The best you can do is take the time to learn what students believe to be true about themselves – both in their current attainment and in their capability to learn and progress. It is these existing beliefs that students hold about themselves that give you a clue as to how they might respond to your grading game.

Good luck with writing the rules of your grading game (it’s not easy)!

These three blogs (part I, part II here, part III) are based on a talk I gave at Headteachers’ Roundtable Summit in March 2019. My thoughts on this topic have been extensively shaped by conversations with Ben White, a psychology teacher in Kent. Neither of us yet know what we think!

The two fundamental jobs that children need to do are to feel successful and to have friends – every day. Sure, they could hire school to get these jobs done. Some achieve success and friends in the classroom, the band, the math club, or the basketball team. But to feel successful and have friends, they could also drop out of school and join a gang, or buy a car and cruise the streets. Viewed from the perspective of jobs, it becomes very clear that schools don’t often do these jobs well at all – in fact, all too often, schools are structured to help most students feel like failures.

Clayton M. Christensen, James Allworth and Karen Dillon
How Will You Measure Your Life?

Where do we believe we are on the grading curve?

Whether we like it or not, from a very young age students start to develop an idea of where they sit on a bell curve of attainment, relative to their peers. All that cohort-referenced feedback does is to give students new information about how well they are doing in the ‘game’ of trying to climb up the bell curve. If only it were so simple that teachers could encourage students to make an effort in this ‘game’ by handing out regular cohort-referenced feedback! Though the examples in my first post showed that it’s a good bet that introducing cohort-referencing will raise effort, on average, other studies show it is risky strategy because it can alter beliefs in quite unhelpful ways.

How can this happen? Surely everyone wants to work hard to get as far up the ladder as they can? This blog post sets out the mental model that explains three dimensions of each of your student’s belief systems that you must consider when thinking through how they respond to learning their attainment.

Beliefs about where we sit on the curve

There is pretty convincing evidence that, rather than constantly striving to be the best, we instead tend to prioritise self-esteem maintenance or keeping ourselves at the place in the hierarchy to which we have become accustom! (Remember we are having to juggle multiple games in life so choice about where to direct effort is sometimes necessary.) This means that the effect of attainment feedback rather depends on how it allows us to update our prior ideas about position on the bell curve (or at least lessen the fuzziness we have about how well we are doing).

Where a mark or grade received simply confirms our prior view of how well we were doing, the act of receiving the feedback might have no impact because we have no need to adjust behaviours to close the gap between our current performance and the internal standard we had in mind for ourselves.

This was demonstrated in a nice experiment, albeit one on undergraduates, which provided students with their position in the grade distribution every six months. The researchers found that giving this grading feedback actually LOWERED academic outcomes more frequently than it raised them! How can this be? Well, it turns out that, in the absence of knowing their rank, many of these students had actually underestimated how well they were doing. When they received the good news they were doing better than they thought, their (self-reported) satisfaction increased and they ramped down the effort they were putting into their studies. There was a smaller group for whom the reverse was true – they had overestimated their position in the year and so receiving the bad news of their true ranking caused them to increase their effort.

England’s long experiment with giving Year 12 students additional nationally-benchmarked feedback in the form of AS exam results was mirrored by a nicely-evaluated Greek policy experiment that introduced a nationally-benchmarked penultimate year exam. The strongest predictor of your response to this type of additional penultimate year feedback was whether it gave you a positive or negative surprise.

• Negative surprise: Learning that your 11^th grade performance was worse than for other students with a similar 10^th grade scores to you generally led to greater effort for final year exams, regardless of your prior attainment (shown in green).
• Positive surprise: Learning you did better in 11^th grade than would be suggested by your 10^th grade position led to less effort for final year exams (red on the chart).

This finding that prior over or under confidence in attainment is central to how we respond to feedback is an incredibly important consideration when trying to second-guess how your students might respond to the feedback you give them. If they find out they are doing worse than they thought, they tend to pull your finger out! If they find out they are doing better than they thought, they tend to step off the gas! It is prior beliefs in attainment, relative to actual attainment, that frequently predict a student’s response (and this finding is replicated in many studies).

Now, it’s worth saying that, although this wasn’t true for the university grading study described above, people tend to be a little over-optimistic about how well they doing [£] so giving clearer feedback tends to be more helpful than not in pushing people to make an effort. However, this doesn’t mean that getting bad news is ALWAYS useful. To understand why, we need to look at how feedback shapes other beliefs we hold about ourselves.

Beliefs about productivity of effort

So far, we’ve learnt how feedback shapes the beliefs we hold about our current performance. However, it also shapes our beliefs about our ability to learn or climb up the rankings. Receiving a grade is small part of a feedback loop, all the pieces of which must be in place for it to produce a useful behavioural response. The student needs to believe it is possible to play the game, i.e. that their effort can productively translate into learning that in turn translates into performance in the next test. (Where students do not feel they have the knowledge and capacity to translate effort into learning then decisions about how to convey attainment are irrelevant.)

All teachers know how important it is for students to maintain self-efficacy in order to persevere over extended periods of time at school, often without short-term rewards. Teachers will also know that, whilst we can be sure that grit and growth mindset are malleable characteristics, particularly in young children, it has proved difficult to construct reliable interventions to improve mindset. Unfortunately for us in designing grading feedback, it can often simultaneously influence beliefs in performance and beliefs in ability to learn such that they work against each other. Learning you’ve been unexpectedly successful in a test enforces a belief in your ability to learn, yet dampens your desire to do so!

Feedback can, correctly or incorrectly, seriously impair our belief in the productivity of our effort – what Dweck (1986) calls ‘learned helplessness’. So, you’ve found out you are towards the bottom of the bell curve. You’ve come to believe that even a huge amount of effort can’t get you to a much better position. Then what? Well, you might be OK with reducing your own internal standard of how good you want to be and continue to take part in the grading game…. but it’s understandable that it doesn’t appeal to everyone.

Choosing to play your grading game

Young dropouts are often behaving quite as rationally as their more successful peers. If you are fourteen or fifteen and well behind academically, your chances of catching up start to look very very slim. And if you are not going to improve your relative position much, even if you change your whole lifestyle, then why not give up on the academic competition completely? It is likely to be better for your self-respect and, at least in the short term, not obviously worse for your other prospects: time spent in school when it isn’t going to improve your job opportunities, and you don’t enjoy the work, is just time wasted.

Alison Wolf
Does Education Matter?

Nobody likes to compete in a game they know they will do badly in, regardless of how hard they work. This is particularly true in education, where our ability can be tightly intertwined with our sense of self and how other people perceive us. Empirical research shows that in addition to liking to be at the top of the pile, we are also strongly “last-place” averse. If avoiding the bottom of the ladder is a powerful motivator in humans, how are your students going to try to avoid it? By trying to climb up? Or by trying to climb off?

Even where performance in the game is kept entirely private, there is good evidence that students develop strategies to avoid learning their ranking. In experiments, those who suspect they are lower performers are more likely to avoid learning their rank in class (e.g. here), thus refusing to take part in the grading game. Even worse is the phenomenon of ‘self-handicapping’ – deliberately withdrawing effort where there is potential for self-image-damaging feedback. For example, staying up late gaming with friends before an important exam allows us to attribute poor exam performance to the tiredness rather than low ability.

Where rankings are made public, our responses to them will be strongly shaped by adherence to social norms and the preservation of our social reputation. I don’t want to get into reviewing the literature on award-giving (here is a nice summary), but it makes it clear that responses are heavily framed by peer context. Awards can be motivating if handed out with care in the right circumstances (i.e. sparingly, in just-circumstances, exploiting the bond of loyalty between giver and receiver, but with consideration of effects on non-recipients). However, if we choose to make academic effort observable to peers by publishing grades or awards, students may act to avoid social penalties or gain social favour by conforming to prevailing norms.

In the US, academics Bursztyn and Jensen have run school experiments highlighting how these risks play out with students. They introduced a performance-based leaderboard for an online learning platform which announced the top three performers in the classroom, in the school, and overall on the platform. This publication of the leaderboard led to a decrease in student performance on the platform, which was primarily driven by a decline in effort provided by students who were top performers prior to the introduction of the leaderboard. This suggests it created a fear of peer social sanction for high performers (by contrast, if anything, the performance of those at the bottom of the distribution slightly improved).

In another experiment, they showed that whether or not you sign-up to a voluntary test preparation course is affected by whether your sign-up is made public. In classrooms with high attaining students taking advanced classes, sign-up rates were unaffected by whether the enrolment decision was public. It was in the classrooms with less high attaining students that enrolment rates were much lower with a public sign-up system. Social norms of the classroom matter. (A nice feature of the study was their ability to observe the same student sitting alongside high achieving peers in one classroom and alongside more mixed peers in another – think about a scenario where a student is in ‘top set’ for one subject and mixed ability class for another. These same students were negatively affected by the sign-up sheets being made public in the more mixed class but were not in the high-attaining class.)

The mental model of responses to grades for your students

Handing out clear, cohort or nationally-referenced feedback to students or their parents on how they are performing has the potential to be a powerful motivator, but comes with risks. Knowing a student’s ability or attainment cannot help you predict how they will respond. What matters is how the grading information:

• Changes their beliefs about their attainment
• Changes their beliefs about their ability to learn and get better
• Changes their desire to keep playing the competition of trying to be the best, or maintain their position, or avoid the bottom rung

Every teacher has to give some feedback on attainment and there is no risk-free or value-neutral approach to doing it. I hope you can think about the mental model I’ve described in relation to students you know. Have you ever asked them, for example, how well they think they are doing before you hand out their grades? Who is suffering from ‘learned helplessness‘ and is there anything you can do to counter that? Which five students you teach are most likely to respond negatively to learning their attainment in a more transparent way?

Schools do quite bizarre things to avoid telling students their position on the attainment bell curve. Bizarre, yes, but not necessarily unhelpful. In the final blog post, I will use this mental model to describe why schools end up creating these strange grading games!

You can find part III here.

These three blogs (part I here, part II, part III) are based on a talk I gave at Headteachers’ Roundtable Summit in March 2019. My thoughts on this topic have been extensively shaped by conversations with Ben White, a psychology teacher in Kent. Neither of us yet know what we think!

Teachers are rarely trained in how to give formal, summative feedback to students and their parents – following a test, in a school report, or in a conversation at parents’ evening. We instinctively form views of how a student is doing in relation to others we teach, yet when we report attainment we frequently translate these relative perspectives into some sort of strange scale that it is hard to interpret.

“Stuart is performing at a Level 2W”
“Mark achieved a Grade B+”
“James scored 68% in the class test”
“Laura has been award a Grade Triangle”

Grades, levels, percentages, arbitrary notions of expected, targets, progress measures, GCSE grades given in year 7, flightpaths… Why do you make it so complicated when the truth about attainment is really quite simple?

I want to explore the arguments of the ‘truth-advocates’ out there. There are those who think we should simply tell students what we can validly infer from the end-of-term test we write and administer in our school. Teacher-bloggers such as Mark Enser and Matthew Benyohai rightly point out that when we provide marks without norm-referencing information about how others have performed, our so-called grade actually provides them with little information (of course, students scramble around class to figure out this cohort-referenced information themselves). A ‘Grade B’ isn’t inherently a phrase that contains meaning; its meaning for students arises through knowledge of the distribution of grades awarded. The ‘cleanest’ version of cohort-referencing – just handing out class or school rankings – seems to be quite commonplace in certain secondary school subject departments today, according to this Teacher Tapp sample. However, are there considerations beyond validity of inference we should consider before handing out rankings or grades?

The second set of ‘truth-advocates’ who like school-rankings prioritise attainment feedback’s role in an educational process where they are hoping to change student behaviours. You can read Deborah Hawkins’ blog about how termly rank order assessments are used at Glenmoor and Winton Academies to induce student effort. By creating a termly ranking game, they recognise the challenges schools face in trying to persuade students to spend time on their game – the game of getting good at maths or French – rather than the other games of life – being popular, playing sport, pursuing new relationships, and so on. Creating a game where students are induced to work harder to climb up the bell curve of achievement is potentially so powerful because we care a great deal about our place in social hierarchies. The economist Adam Smith (1759) once said, “rank among our equals, is, perhaps, the strongest of all our desires”. Biological research has shown that high rank is often associated with high concentrations of serotonin, a neurotransmitter in the brain that enhances feelings of well-being. What’s more, social comparisons are an indispensable part of bonding among adolescents.

It is easy to find academic studies that back up the grading policies of schools that report rankings or cohort-referenced grades. For example, a school in Spain experimented with giving students ‘grade-curving’ information, alongside the grades they had always received (e.g. supplementing the news of receiving a Grade C with information that the class average was a Grade B+). The provision of this cohort-referenced information led to an increase of 5% in students’ grades and the effect was significant for both low and high attainers. When the information was removed the following year, the effect disappeared. Similarly, a randomised trial on more than 1,000 sixth graders in Swedish primary schools found student performance was significantly higher with relative grading than with standard absolute grading. These positive effects of cohort-referencing are mirrored in numerous university field and lab experiments (e.g. here and here).

So, why don’t we all follow the ‘truth-advocates’ and give students clear, cohort or nationally-referenced feedback on how they are doing, allowing them to compete with their peers? We avoid this feedback, of course, because we are nervous about how our students will respond to it. Whether we are conscious of it or not, we all have a mental model of how we hope reporting attainment might change student behaviours. It is these implicit, mental models that explain why we might tell a half-truth to a student or parent, assuring them they are doing fine when the opposite is true. Mental models explain what meaning we hope to convey when we tell a student they have 68% in a test or why we allow students a few minutes to compare their marked papers with others in class. They explain why we send quite odd ‘tracking’ data home to parents, and are privately quite content that it doesn’t allow them to infer whether their child is doing better or worse than average.

In the blog posts that follow, I hope to persuade you of the importance of developing a clear mental model of how your students might respond to learning their attainment. I am collating together a messy empirical literature from education, behavioural psychology and economics, which lacks a consistent theoretical footing from which to build generalisable findings. Having read these studies, I think it is most useful for teachers to develop mental models that emphasise changes to the students beliefs about themselves. Beliefs often fulfil important psychological and functional needs of the individual [£]. This literature emphasises three dimensions to describe how grading feedback affects student behaviour:[i]

1. The effect on student beliefs about their attainment
2. The effect on student beliefs about their ability to learn
3. The effect on their willingness to play the game you want them to play

Talking about attainment is something that no teacher can avoid, and choosing to use fuzzy and ambiguous language with parents and students is not a value-neutral approach (for reasons that will become clear by blog three). Neither is telling children the whole truth about how they are performing. For so much of our time as teachers we talk about how the child can change the grade they receive. In these posts, we will be talking about how the grade can change the child!

You can find part II here.

---

[i] Note – this differs somewhat from the favoured feedback model of educationalists – Kluger and DeNisi’s (1996) Feedback Intervention Theory – which seems particularly pertinent to predicting mechanistic responses to task-based feedback but isn’t well-aligned with the disciplinary traditions of the empirical research I am reviewing here.