Five observations on the TouchPaper problems party

Last weekend, Laura McInerney and I hosted a rather experimental TouchPaper problems party. Her blog here tells you what happened on the day and a few party goers have started writing up their own thoughts on the day (here, here and here). Here are five observations from the perspective of a rather-out-of-touch-with-the-classroom academic:

1. I have never done so little preparation for event I have hosted. Laura was in the wrong part of the world; I was busy shelving research projects for my impending maternity leave. So, we could either host a day with close to zero preparation or not at all. I asked the Director of IOE, Chris Husbands, to give us a room, drinks and wifi while Laura wrote a set of blog posts to light the TouchPaper for each group. We outsourced all intellectual preparation to our fantastic group leaders in return for a couple of drinks and dinner. Without their help we would have been sunk. Thank you.

2. Smart, motivated adults don’t need as much event structure as young, easily-distracted children. Giving the groups complete discretion as to how they spent four hours produced an amazing diversity of activities. Some of the approaches reflected the size of the group (consensus is that 3 is too small and 8 is too large). One group spent an hour largely silent writing and reading pink post-it notes:

2014-01-18 11.58.13

And while a couple of party-goers commented that they would have liked more structure in the day, more appeared to value the flexibility afforded to groups to do exactly as they pleased.

3. To answer the TouchPaper problems, group work needs to be complemented by intense individual endeavour. Group work enables sophisticated discussions about the interpretation of the TouchPaper problem and how it can be broken down into necessary sub-questions. It also works well for research design. But somewhere between conceptualisation and research design somebody needs to review relevant literature.

4.  Research summaries that instruct teachers what to do in the classroom are a poor substitute for intense engagement in a research question. Educational research naturally wants to get teachers to use their studies to inform classroom practice (e.g. see current BERA and EEF efforts). Teachers are busy people and it is easy to infer that ‘digested’ research summaries and toolkits are the only practical approaches to achieving this. But these summaries are necessarily reductionist (e.g. you should give your pupils’ feedback on their work) and I wonder whether they alienate teachers who appreciate the complexity of deciding best practice across a variety of settings. Teachers live an incredibly hectic and stressful life during term-time, yet more than one party-goer said that engaging in the TouchPaper problems on a Saturday was exactly what they needed to re-fuel and reaffirm their love of teaching.

5. My dream school would employ these 40-odd party-goers, from the PGCE student through to the headteachers and ex-teaching drifters like this and thisI was blown away by some of the conversations I eavesdropped on (and occasionally baffled by our complexity group):

2014-01-18 11.43.56

Admittedly we made no effort to recruit a representative sample of teachers (see point 1). Our party was made up of a self-selected group of the most engaged of the edu-twitterati and bloggers. Any future TouchPaper problems party might have to be organised differently to reflect diversity of engagement in educational policy-making and research amongst more typical teachers.


Announcing: The 1st TouchPaper Problem Solving Party

An invitation from Laura McInerney to our TouchPaper Problem Solving Party…

Laura McInerney


Back in September, the ResearchEd conference hosted a vast range of speakers suggesting how research might be more effectively used in education. My own contribution was a presentation of 7 problems which, if answered, would help teachers understand important things about their job. (See the full talk here)

The list was called the “TouchPaper Problems” – a reference to the blue paper with which one lights fireworks. I created each problem because I felt it would give information useful in classrooms. They are difficult questions though. Each one will require several layers of theory-testing and consideration before they can be considered ‘solved’. This sort of public problem-solving approach in the past motivated mathematicians and engineers to solve some of the most fundamental problems in their sector. My theory is simple: we should do the same in education.

To move things forward the brilliant Becky Allen suggested it…

View original post 455 more words

Averages and tails: What sort of pupils benefit most from Teach First participants?

When Joe Kirby and I presented data on whether Teach First is working at ResearchED2013 earlier this month, the best audience question came from Arthur Baker who wanted to know whether the improvements in average school attainment following Teach First participation meant that the most disadvantaged students were benefiting, or not.

This question struck a particular chord with me because I had recently read Machin and Silva’s chapter in The Tail, an excellent collection of essays about underachieving pupils in English schools. They showed that, although the early sponsored academies had indeed improved average GCSE outcomes (see Machin and Vernoit for evidence on this), there was no improvement in GCSE attainment for those starting secondary school at the bottom of the Key Stage two distribution. So, a policy designed to improve life chances for those at risk of being left behind in under-performing schools seemingly did not impact on those within the schools who were most educationally disadvantaged.

To return to Arthur’s question: what sort of pupils benefit most from a school’s participation in Teach First? We didn’t answer this in my original paper with Jay Allnutt, but I’ve re-opened up our datasets to have a look. In the paper we compare the performance of a particular pupil across their core subjects of GCSE English, maths and science to see how well the pupil performs in the department(s) who chose to take Teach First participants, compared to those who did not. We showed that departments taking on Teach First participants were more ineffective than others within the same school before joining Teach First (scoring about 10% of a GCSE grade worse), but that by years two and three of participation they were outperforming their neighbouring departments who did not join Teach First (by about 15% of a GCSE grade).


To mirror the analysis of Machin and Silva on academies, I looked at how lower and higher ability children responded to the presence of Teach First participants in the department. The chart below shows that the overall pattern of impact of Teach First is similar for children across the ability distribution. Those scoring in either the top 10% or bottom 10% at Key Stage two do not appear to be particularly suffering in their attainment within the department who will soon join Teach First. (We can only speculate as to why – e.g. very high achieving pupils have strategies and other support systems for overcoming any teacher quality issues; very low prior attainment pupils have specific circumstances that dominate GCSE grades across all subjects.) In years two and three after joining Teach First, it appears that improvements in attainment are shared across the ability spectrum. It is hard to make claims as to why this should be so, not least because Teach First themselves have not systematically collected information on what year groups and which ability sets within year groups their participants are teaching.


Finally, to answer the question that Arthur Baker was most interested in, this chart shows that the responsiveness of free school meals pupils to the placement of Teach First participants in departments is very similar to the response of other pupils. If anything, the free school meals pupils appear to benefit a little more (and this difference is statistically significant in year two).


Do we really have to wait a whole year for researched2014?

Others have blogged about researched2013 and how great it was. I want researched2014 to be just like researched2013 because it was perfect. But just in case Tom Bennett and Helene decide to tamper with the current model, here are seven (it’s always 3 or 7) minor modifications I’d vote for:

  1. Many participants were tweeters, so wifi is pretty important. But if we get wifi next year our phone batteries will run down, so we’ll need a phone charging station too.
  2. Let’s have a consistent hashtag next time. How about #resed2014 compromise, or am I just encouraging further hashtag proliferation? (And what about a screen somewhere showing the tweets as they appear?)
  3. Participant lists are really useful for people who like stalking at or after the conference
  4. If we’d had twitter avatars and handles on our name badges I would have been able to spot Andrew Old!
  5. It was fine having no lunch break. Lunch breaks just produce huge queues. How about charging everyone for a lunch bag in the ticket price that they can pick up and eat anytime?
  6. Dulwich College worked pretty well but it was so hard to get to! Some of us visited a Dulwich village pub afterwards. I couldn’t stay long, but met even more new people and learnt about Croydon schools and English GCSE controlled assessment! Can we have a venue nearer a nice pub please?
  7. Laura McInerney‘s (say “mac-in-errr-knee”) session on #touchpaper problems is the one I’m still thinking about because it was participatory and left us something very specific to think about. I thought about running a more interactive session myself, but chickened out. Conventional talks are great, but perhaps a little more audience participation (or homework!) might be nice for next year?

I hope we don’t have to wait until September 2014 to engage with researched stuff again. It is probably unreasonable to expect Tom Bennett to do everything for us (please Tom?), so I guess we each have to make some sort of tiny contribution. What is yours going to be?

How can we learn if Teach First is working?

Last week I published a paper I wrote with Jay Allnutt about the impact of Teach First on GCSE attainment. We received a large amount of feedback on the paper, via a seminar presentation at BERA conference, comments on a blog I wrote, twitter and email. Rather than simply present these research findings at researched2013, I showed the audience some of feedback we received on the paper from the education community to reflect on how research can move forward understanding of major education policies. This blog gives an approximate overview of my presentation at researched2013, following on from Joe Kirby’s @joe__kirby blog of his talk.

The paper I published with Jay Allnutt showed that schools participating in Teach First improved their GCSE results. In their second year of participation, the improvements were in the order of one grade in one of a pupil’s best eight subjects (5% of a pupil s.d.) or a two percentage point gain in the % achieving five or more GCSEs at A*-C (incl. Eng and maths).

Making these sorts of claims was not straightforward because…

…Schools were not selected at random to join Teach First

I think all new education policies should be randomised in their implementation, providing it is possible. Teach First received substantial government funds so has a duty to taxpayers to demonstrate its effectiveness. This was a major failing on the part of Civil Servants who agreed to fund it (Sam Freedman @samfr spoke eloquently about this issue). Rather than recruit school’s from 20 local authorities in London in the first year of the programme, recruitment should have taken place across a randomly drawn lot of deprived schools or alternatively local authorities should have been randomised into the programme.

This did not happen, producing a very serious identification problem because Teach First schools look very different to others – they are concentrated in London, are relatively deprived, were likely to have had particularly severe teacher recruitment problems and may have had headteachers who were particularly dynamic or risk-taking.

We try to deal with non-random selection in our estimation by finding a set of schools that look identical to the Teach First schools, except that they were not participating in the early years:

Is Teach First working

And rather than simply comparing GCSE performance between our Teach First schools and a matched control group of schools, we run regressions that model changes taking place year-by-year at every school:

Is Teach First workingDID

The first set of responses we received to our papers asked…

Are there confounding factors?

“it seems possible that managerial teams that are ‘early adopters’ [of Teach First] are a different calibre to ‘followers’ who catch up with trends after it starts to look cool … [so it might be] … higher managerial chutzpah of the early adopters doing others things nothing to do with TF that cause results to rise”

[blog comment]

For this type of criticism to be valid, the timing of the adoption of ‘higher managerial chutzpah’ must have to exactly coincide with the participation in Teach First. (Why? Well, we match on change in GCSE scores so their superior performance cannot have preceded the programme and we perform falsification tests, which demonstrate that Teach First did not have an impact in the years before the school joined the programme.) But more importantly, if our findings are entirely due to ‘higher managerial chutzpah then we would not witness positive effects of Teach First for those departments who received TeachFirsters, compared to those in the same school who did not.

Is Teach First workingDEPT

The next set of feedback we received claimed that…

The effects are too large small

It seems most unlikely four [Teach First] teachers can raise the attainment of a school or department single handed by teaching their pupils better, not least as they simply do not teach enough pupils.

[blog comment]

Does ‘5% of a standard deviation’ represent significant and positive impact?

[@David_Cameron76 on twitter and in the audience at researched!]

This poster asked the far more important question…

The effects are too large small, given costs

Are these estimated marginal benefits worth the massive public subsidy that @TeachFirst receives? I don’t think so

[@jpjsavage on twitter]

We don’t answer this question in our research paper, but I hope we’ll be able to in a new Nuffield-funded project I’m working on which is led by Ellen Greaves at IFS. We’ve been asked to look at the relative costs and benefits of all the different teacher-training routes. There are two major impediments we face: no national database of school and departmental participation in PGCE programmes exists and SCHOOL DIRECT WAS NOT RANDOMISED (@samfr – this happened your watch did it not? Why no randomisation to stagger roll-out?).

Some commenters on our research did not believe our findings because…

Your research is clearly biased

…Clearly you have identified what you see as a benefit of this approach (albeit based on the work of someone who clearly has such a positive view of what Teach First are doing that he has gone to work for them)…

[Jonathan Savage as blog comment]

“You’ve seen improvements in these schools and have just gone looking for explanations…”

[seminar audience member]

I have no interest in Teach First in the sense that I wasn’t paid by them to do the research, I trained as a teachers via a traditional PGCE and the wider ‘narrative’ of the research I do isn’t underpinned by a theoretical view that programmes such as Teach First are likely to be successful. My co-author was trained by Teach First, worked in a secondary school whilst he conducted this analysis, and now works for Teach First. If people believe our research is biased it is hard for us to persuade them otherwise. It isn’t enough to claim that scientific and quantitative methods are less susceptible to bias, but this isn’t true (see this article on how doing maths is influenced by political beliefs). We should have been required to declare our methods – our outcomes variables and regressions we planned to run – before we conducted the research, as EEF evaluators are required to, but we didn’t.

We also had the usual comments from people who ask…

Can we ever ‘know’ anything?

You identify so many potential caveats, influencing factors, limitations to the validity of the methodology and other potential problems it is hard to take your findings seriously.

[Jonathan Savage as blog comment]

Paper ‘estimates’ benefits. Others claim them as facts.

[‏@jpjsavage on twitter]

Can you really claim direct causal links? Your paper is rightly tentative the headlines are not.

[@egwilson on twitter]

There are philosophical questions about what knowledge and causality are; I’m not the right person to blog on these things. But my problem with the commenters above is that by invoking the ‘it is not causality’ argument, they are taking a value position that there is still no evidence that Teach First works. For me, research on policies should simply aim to shift the balance of probabilities on whether they work or not. Does our research alter the balance of probabilities that Teach First works? I think it does, though we can argue about how much. Am I convinced enough of my findings that I’d be happy to suggest to a headteacher that they join the programme? I think I would (though if they were really reluctant I definitely wouldn’t push it!).

Some commenters rightly ask what the ‘thing’ is that we’ve identified, or…

How should we interpret the effect?

Could you account for the vacancy-filling nature of the programme? That is, a mathematics teacher is better than no mathematics teacher?

[comment by email]

(My response: This is just part of the Teach First effect so we don’t want to ‘account’ for it in our estimation!)

I don’t think you can compare to other ITE methods, as not part of study?

[@BeckyFrancis75 on twitter]

(My response: Correct)

What if the ’cause’ is involvement in “ITE” for a school/department & not “TF“?

[@JohnClarke1960 on twitter]

(My response: Great question! And we’ll be able to say more about this once we’ve completed our Nuffield project)

Have you proved that Teach First impacts on poor kids within these schools, as opposed to other kids?

[a question at researched2013]

(My response: Great question! Let me crunch the data and get back to you)

Now we start getting onto the really important questions, as raised by @samfr in a blogpost…

How might Teach First plausibly change the pupil experience?

Is Teach First workingHOW

Is Teach First workingHOW2

And we had a few more great questions from the audience at researched2013:

Have PGCE courses seen a deterioration in applicants due to Teach First?
Why is Teach First more appealing than a PGCE and what role does the salary versus student loans play in this?
How can we know if Teach First or PGCE applicants have the qualitatively superior characteristics?
Is a major advantage of Teach First the two year length of the programme, compared to just one year for a PGCE?

Yes, they’re young and inexperienced. But Teach First participants have the right stuff


Rebecca Allen

Today, Jay Allnutt and I published a new piece of analysis (PDF) showing that schools taking on Teach First participants have achieved gains in their GCSE results as a result of the programme. Our analysis tracks the performance of these schools in the first three years after they join the programme and compares them to changes in progress at a set of schools that look identical, except for their Teach First participation in that year.

We make sure this comparison set of schools have the same pupil demographic profile, the same prior levels and trends in GCSE performance, are in the same region of England and are all schools who will choose to join Teach First at some point in the future (formally this is known as a matched difference-in-differences panel estimation). Overall, school-wide gains in GCSE results are in the order of an improvement of about…

View original post 629 more words

Looking for research curious teachers

Are you a teacher who is interested in how we should train our new teachers?

Would you like to observe how (fairly) large scale research is carried out?

I am looking for some research-curious teachers and headteachers who might be willing to sit on the advisory board for one of the research projects I am currently involved with. The purpose of the advisory boards is to help inform the research questions and methods, and to help interpret the findings of large projects.

The projects are:

  • A Nuffield-funded investigation into initial teacher training (this is a joint Institute for Fiscal Studies, National Foundation for Educational Research and Institute of Education project). This project investigates the costs and benefits of different teacher training routes. In particular, we would like to discover whether certain routes are more effective than others, in terms of recruitment, the costs (particularly time costs) and benefits to schools associated with training, and their subsequent retention in state schools in England. It would be particularly useful to include teachers on our advisory board who have experience of mentoring PGCE students, GTP/SCITT/School Direct teachers or who have recently trained themselves. The project is led by Ellen Greaves at IFS.
  • An ESRC-funded investigation of the early careers of teachers. This project investigate how the early experiences of teachers in training placement schools and first posts affects their subsequent likelihood to move into particular types of schools or exit the profession altogether. Understanding how to create a schooling environment that retains the best teachers within the state maintained system is critical because we know that relatively large numbers of high quality teachers leave the profession every year and this turnover is damaging to pupil achievement. The project will conduct a large survey of PGCE students and will match this data to institutional records and the School Workforce Census to track the careers of these teachers. It would be particularly useful to have early career teachers on our advisory board for this project.

The advisory board for each of these projects meets about twice a year for the two-year life of the project. The advisory boards would be held at teacher-friendly times, e.g. after 4:30pm during term time or possibly at half term holidays. Obviously 100% attendance of advisory board members is difficult to achieve, so we will recruit more teachers than we need in the hope of achieving representation at each meeting. The meetings will held at IOE or IFS (i.e. in Bloomsbury, close to Euston station) so teachers will need to think about whether it is feasible to travel into this part of London.

If you would like to get involved or would like to know more, please do contact me at In your email it would be helpful if you could tell me how many years you have been teaching, what training route you originally took and what school you now teach in.

EDIT: Thanks for all the interest in these projects. We’ve filled the spaces on our Advisory Boards, but do still send an email if you’d like to give feedback on our proposed surveys.