Sigh! New year, new grammar school paper. This time a HEPI paper by an ex-Civil Servant, Iain Mansfield, who has turned his hand to quantitative social research, starting with one of the most complex questions it is possible to devise. Thankfully I missed most of the commentary on it (bout of tonsillitis). If you didn’t, Lindsey Macmillan et al. have written one response. However, tonight I am better and from my quick read of the HEPI report it is immediately clear how poor much of the analysis is. Sigh again! I am determined that we don’t have to go through this annual charade ad-infinitum.
So I’ve got two rules that I think would help calm the debate and raise the quality of the argument that we are having about what age we should allow academic selection.
Rule 1: You can’t publish research yourself on the question of the causal impact of academic selection until you have passed a test to show you understand why the existing literature is so complex. Seriously, there is a reason why academics are forced to summarise the existing literature before they are allowed to publish their own findings! You need to be able to answer questions like:
- From the following set of papers, which explicitly (a) acknowledge and (b) attempt to deal with the fact that over 20% of students at grammar schools live in a different local authority?
- What are the consequences of ignoring the fact that 12% of students at grammar schools transferred from private primaries? Name five challenges that researchers face in incorporating private schools into analysis.
- Contrast at least two different approaches to constructing the set of pseudo secondary-moderns that have been used in the literature to-date. What are the pros and cons of these approaches?
- Manning and Pischke argued the Fernando Galindo-Rueda and Vignoles paper was invalid by invoking what seemed to be a neat falsification test. What was the test and under what sorts of assumptions would it have been valid?
Rule 2: You cannot be a public commentator on a piece of ‘research’ about the causal impact of grammar schools unless you can first answer the following questions about the research you plan to comment on.
- Have the authors acknowledged that large numbers in grammar schools live in different, and usually non-selective, local authorities and do you understand how they have dealt with this problem?
- Have the authors acknowledged that the presence of grammar schools distorts the nature of local private schools? Have they dealt with this (e.g. how are private school students included in their analysis groups)?
- When considering the impact of selective areas as a whole, how do they define non-selective schools in selective areas, i.e. the group of schools that students are going to who fail the 11+? (Top tip – most are not categorised as secondary moderns in DfE databases.)
- What is their counterfactual to living in a selective local authority? How have they ensured the types of families and students who are living in the counterfactual areas are similar?
There are plenty of public commentators who are capable to reading research carefully enough that they can meet Rule 2. You can’t screen them by their job title or fancy letters before or after their name. That’s why we need new rules. And these rules only apply to impact analysis of the sort that HEPI published today. Very happy to have people writing and talking qualitatively about the system. These perspectives are also important. Moreover, there are many, many questions where basic exploratory analysis is really interesting. It’s what I like to do most days. The causal impact of academic selection at the age of 11 isn’t one of them.
Sorry if this all sounds exclusive but… well… grammar schools are exclusive and they get great results. That’s the point! So let’s make the conversation about their impact on the system a little more sophisticated and maybe we’ll get a better result too.