When Joe Kirby and I presented data on whether Teach First is working at ResearchED2013 earlier this month, the best audience question came from Arthur Baker who wanted to know whether the improvements in average school attainment following Teach First participation meant that the most disadvantaged students were benefiting, or not.

This question struck a particular chord with me because I had recently read Machin and Silva’s chapter in The Tail, an excellent collection of essays about underachieving pupils in English schools. They showed that, although the early sponsored academies had indeed improved average GCSE outcomes (see Machin and Vernoit for evidence on this), there was no improvement in GCSE attainment for those starting secondary school at the bottom of the Key Stage two distribution. So, a policy designed to improve life chances for those at risk of being left behind in under-performing schools seemingly did not impact on those within the schools who were most educationally disadvantaged.

To return to Arthur’s question: what sort of pupils benefit most from a school’s participation in Teach First? We didn’t answer this in my original paper with Jay Allnutt, but I’ve re-opened up our datasets to have a look. In the paper we compare the performance of a particular pupil across their core subjects of GCSE English, maths and science to see how well the pupil performs in the department(s) who chose to take Teach First participants, compared to those who did not. We showed that departments taking on Teach First participants were more ineffective than others within the same school before joining Teach First (scoring about 10% of a GCSE grade worse), but that by years two and three of participation they were outperforming their neighbouring departments who did not join Teach First (by about 15% of a GCSE grade).

To mirror the analysis of Machin and Silva on academies, I looked at how lower and higher ability children responded to the presence of Teach First participants in the department. The chart below shows that the overall pattern of impact of Teach First is similar for children across the ability distribution. Those scoring in either the top 10% or bottom 10% at Key Stage two do not appear to be particularly suffering in their attainment within the department who will soon join Teach First. (We can only speculate as to why – e.g. very high achieving pupils have strategies and other support systems for overcoming any teacher quality issues; very low prior attainment pupils have specific circumstances that dominate GCSE grades across all subjects.) In years two and three after joining Teach First, it appears that improvements in attainment are shared across the ability spectrum. It is hard to make claims as to why this should be so, not least because Teach First themselves have not systematically collected information on what year groups and which ability sets within year groups their participants are teaching.

Finally, to answer the question that Arthur Baker was most interested in, this chart shows that the responsiveness of free school meals pupils to the placement of Teach First participants in departments is very similar to the response of other pupils. If anything, the free school meals pupils appear to benefit a little more (and this difference is statistically significant in year two).

Last week I published a paper I wrote with Jay Allnutt about the impact of Teach First on GCSE attainment. We received a large amount of feedback on the paper, via a seminar presentation at BERA conference, comments on a blog I wrote, twitter and email. Rather than simply present these research findings at researched2013, I showed the audience some of feedback we received on the paper from the education community to reflect on how research can move forward understanding of major education policies. This blog gives an approximate overview of my presentation at researched2013, following on from Joe Kirby’s @joe__kirby blog of his talk.

The paper I published with Jay Allnutt showed that schools participating in Teach First improved their GCSE results. In their second year of participation, the improvements were in the order of one grade in one of a pupil’s best eight subjects (5% of a pupil s.d.) or a two percentage point gain in the % achieving five or more GCSEs at A*-C (incl. Eng and maths).

Making these sorts of claims was not straightforward because…

…Schools were not selected at random to join Teach First

I think all new education policies should be randomised in their implementation, providing it is possible. Teach First received substantial government funds so has a duty to taxpayers to demonstrate its effectiveness. This was a major failing on the part of Civil Servants who agreed to fund it (Sam Freedman @samfr spoke eloquently about this issue). Rather than recruit school’s from 20 local authorities in London in the first year of the programme, recruitment should have taken place across a randomly drawn lot of deprived schools or alternatively local authorities should have been randomised into the programme.

This did not happen, producing a very serious identification problem because Teach First schools look very different to others – they are concentrated in London, are relatively deprived, were likely to have had particularly severe teacher recruitment problems and may have had headteachers who were particularly dynamic or risk-taking.

We try to deal with non-random selection in our estimation by finding a set of schools that look identical to the Teach First schools, except that they were not participating in the early years:

And rather than simply comparing GCSE performance between our Teach First schools and a matched control group of schools, we run regressions that model changes taking place year-by-year at every school:

The first set of responses we received to our papers asked…

Are there confounding factors?

“it seems possible that managerial teams that are ‘early adopters’ [of Teach First] are a different calibre to ‘followers’ who catch up with trends after it starts to look cool … [so it might be] … higher managerial chutzpah of the early adopters doing others things nothing to do with TF that cause results to rise”

[blog comment]

For this type of criticism to be valid, the timing of the adoption of ‘higher managerial chutzpah’ must have to exactly coincide with the participation in Teach First. (Why? Well, we match on change in GCSE scores so their superior performance cannot have preceded the programme and we perform falsification tests, which demonstrate that Teach First did not have an impact in the years before the school joined the programme.) But more importantly, if our findings are entirely due to ‘higher managerial chutzpah then we would not witness positive effects of Teach First for those departments who received TeachFirsters, compared to those in the same school who did not.

The next set of feedback we received claimed that…

The effects are too large small

It seems most unlikely four [Teach First] teachers can raise the attainment of a school or department single handed by teaching their pupils better, not least as they simply do not teach enough pupils.

[blog comment]

Does ‘5% of a standard deviation’ represent significant and positive impact?

[@David_Cameron76 on twitter and in the audience at researched!]

This poster asked the far more important question…

The effects are too large small, given costs

Are these estimated marginal benefits worth the massive public subsidy that @TeachFirst receives? I don’t think so

[@jpjsavage on twitter]

We don’t answer this question in our research paper, but I hope we’ll be able to in a new Nuffield-funded project I’m working on which is led by Ellen Greaves at IFS. We’ve been asked to look at the relative costs and benefits of all the different teacher-training routes. There are two major impediments we face: no national database of school and departmental participation in PGCE programmes exists and SCHOOL DIRECT WAS NOT RANDOMISED (@samfr – this happened your watch did it not? Why no randomisation to stagger roll-out?).

Some commenters on our research did not believe our findings because…

Your research is clearly biased

…Clearly you have identified what you see as a benefit of this approach (albeit based on the work of someone who clearly has such a positive view of what Teach First are doing that he has gone to work for them)…

[Jonathan Savage as blog comment]

“You’ve seen improvements in these schools and have just gone looking for explanations…”

[seminar audience member]

I have no interest in Teach First in the sense that I wasn’t paid by them to do the research, I trained as a teachers via a traditional PGCE and the wider ‘narrative’ of the research I do isn’t underpinned by a theoretical view that programmes such as Teach First are likely to be successful. My co-author was trained by Teach First, worked in a secondary school whilst he conducted this analysis, and now works for Teach First. If people believe our research is biased it is hard for us to persuade them otherwise. It isn’t enough to claim that scientific and quantitative methods are less susceptible to bias, but this isn’t true (see this article on how doing maths is influenced by political beliefs). We should have been required to declare our methods – our outcomes variables and regressions we planned to run – before we conducted the research, as EEF evaluators are required to, but we didn’t.

We also had the usual comments from people who ask…

Can we ever ‘know’ anything?

You identify so many potential caveats, influencing factors, limitations to the validity of the methodology and other potential problems it is hard to take your findings seriously.

[Jonathan Savage as blog comment]

Paper ‘estimates’ benefits. Others claim them as facts.

[‏@jpjsavage on twitter]

Can you really claim direct causal links? Your paper is rightly tentative the headlines are not.

[@egwilson on twitter]

There are philosophical questions about what knowledge and causality are; I’m not the right person to blog on these things. But my problem with the commenters above is that by invoking the ‘it is not causality’ argument, they are taking a value position that there is still no evidence that Teach First works. For me, research on policies should simply aim to shift the balance of probabilities on whether they work or not. Does our research alter the balance of probabilities that Teach First works? I think it does, though we can argue about how much. Am I convinced enough of my findings that I’d be happy to suggest to a headteacher that they join the programme? I think I would (though if they were really reluctant I definitely wouldn’t push it!).

Some commenters rightly ask what the ‘thing’ is that we’ve identified, or…

How should we interpret the effect?

Could you account for the vacancy-filling nature of the programme? That is, a mathematics teacher is better than no mathematics teacher?

[comment by email]

(My response: This is just part of the Teach First effect so we don’t want to ‘account’ for it in our estimation!)

I don’t think you can compare to other ITE methods, as not part of study?

[@BeckyFrancis75 on twitter]

(My response: Correct)

What if the ’cause’ is involvement in “ITE” for a school/department & not “TF“?

[@JohnClarke1960 on twitter]

(My response: Great question! And we’ll be able to say more about this once we’ve completed our Nuffield project)

Have you proved that Teach First impacts on poor kids within these schools, as opposed to other kids?

[a question at researched2013]

(My response: Great question! Let me crunch the data and get back to you)

Now we start getting onto the really important questions, as raised by @samfr in a blogpost…

How might Teach First plausibly change the pupil experience?

And we had a few more great questions from the audience at researched2013:

Have PGCE courses seen a deterioration in applicants due to Teach First?

 

Why is Teach First more appealing than a PGCE and what role does the salary versus student loans play in this?

 

How can we know if Teach First or PGCE applicants have the qualitatively superior characteristics?

 

Is a major advantage of Teach First the two year length of the programme, compared to just one year for a PGCE?