What Works for What?

“If you’re not using evidence, you must be using prejudice …”
Kevan Collins, EEF

In our brave new world of research based this and evidence informed that, I am very much swimming against the tide, because I don’t think we should try to scientise education. While almost everyone else is focused on neuroscience, randomised controlled trials and the EEF, I’ve just written a book about education as an art form. Yes, educational research can be interesting and valuable, and it certainly adds to the store of knowledge we have on the subject of how we might educate our children, but there are a significant number of problems with basing our practice on what evidence ‘says’, and some of those advocating evidence based practice seem happy to gloss over these. Commentators will often talk (at great length) about the kinds of bias to which other human beings are subject, but they completely fail to see how biased they are themselves. I sometimes feel like I’m shouting “but the emperor has no clothes!” while people tell me that he is, in fact, wearing Armani.

The first problem with evidence based anything, is that in education we don’t have a common set of values to which we want to work – we haven’t even agreed what education is for. The Education Select Committee did hold an inquiry to try and figure this out, but they didn’t seem to come up with much in the way of an answer, and it’s all gone very quiet on that front now. Meanwhile, on social media, one group of people believes that the main goal of schools is to get as many facts into our kids as possible so they get ‘cleverer’. Another group of people thinks that the main purpose of school is to make kids ‘socially mobile’ and to end inequality. (There is quite a bit of cross over between these first two groups.) However, another group of people believes that school is about preparing children for the future, equipping them with skills that will help them do jobs, some of which don’t even exist yet. And yet another group thinks school should be about agency and oracy for young people. (Again, there is lots of crossover between these two.) Then there is a group of people (often politicians) who think that school is mainly there to create a well trained and compliant future workforce. And finally, if we’re totally honest with ourselves, some parents see school as a free source of childcare that allows them to go out to work. But if we can’t even agree what we’re doing school for, how on earth can we agree about what works for doing it?

Building on the problem of what education is for, it’s also crucial to consider what it is not for, and to take into account some pretty serious questions about ethics. If we could prove that Intervention X would lead to better outcomes, at least in the short term, should we always use it? Let’s say Intervention X involved hitting children with a stick. Would we still be happy to do the intervention? Clearly not, since that would be illegal, although it is not so very long ago that schools were perfectly happy to use this method. Now let’s say Intervention X involved the public ranking of children according to their test results. Would we still be happy to do the intervention? You might or might not think that this is a perfectly reasonable thing to do. But put it this way: if you thought it was okay to use this method on my children, I would not be happy to send them to your school.

The next problem with basing education on research results is that schools tend to be messy places, where all kinds of big and small people meet to work and slosh around together. There’s nothing at all wrong with this – it’s part of the reason why I love schools so much. However, this is nothing like the controlled, clinical environment where medical randomised controlled trials take place. As it happens, my best mate is the manager of a Clinical Research Facility. The CRF she runs cost millions of pounds to set up. She has access to cutting edge resources, top end equipment, highly qualified staff and a clinically controlled atmosphere in which trials can be run at great expense. Whereas in education we have Class 3E in a mobile classroom on a wet Friday afternoon doing a new scheme with Mrs Jones who has been teaching for twenty five years and is, to be frank, fed up of new schemes. You might be able to spot the slight flaw in this bit of the plan.

Yet another problem with research in education is that we have to find a way to measure the outcomes of the research we do, and education being education, we like nothing better than a nice test to sum up how we’re doing. The EEF are currently funding a trial into Philosophy for Children. Nothing wrong with that, surely? A bit of philosophy sounds like a lovely idea. In fact, I don’t know why I didn’t think of it before! But if you read about the trial you eventually get to the bit on the website that tells you that “The primary outcome measure will be attainment measured by combined Key Stage 2 maths and English scores.” We are measuring the value of philosophy for children by seeing what it does to high stakes test results. Clearly it’s not the children who need the philosophy lessons.

The next thing to bear in mind is that all these trials cost money to fund (the EEF was set up with a DfE grant of £125 million). At a time when budgets are being slashed, and school staff are being cut, we are still spending millions of pounds trialing new interventions.  The EEF was set up with the aim of doing research to narrow the gap between outcomes for children from advantaged and disadvantaged homes. While this is an admirable aim, despite all the talk about how we can do this, the gap stays persistently wide and more children are living in poverty, not less. And at the same time as the EEF are funding research into things that might close the gap, we have a government who have decided to ignore what research says and fund things that we are told are likely to widen it. How can anyone stomach the DfE calling for evidence based education when they see fit to fund new grammar schools?

Another very interesting problem about education research, which hadn’t occurred to me until I read an article about it recently, is that we seem to have completely overlooked the possibility that interventions can have negative side effects. When the doctor gives you a prescription for some new pills, she will warn you that there might be side effects. The leaflet inside the box of pills outlines exactly what the potential side effects are. But in education, we have become so focused on what research tells us about the benefits of certain approaches, that we have completely forgotten to think about the downsides. I’ve mostly given up commenting on phonics, because every time I do, I get angry messages from people who produce SSP schemes. But a few years ago I wrote this blog, listing many of my concerns. When people are deeply invested in a piece of research being proved ‘correct’, there can be a distinct reticence to discuss the potential downsides. And history tells us a story about the issues this has caused in the field of medicine.

And finally, call me a picky, middle class, sharp elbowed parent if you like, but those in favour of research in education seem to be building the system on the assumption that it is okay to use school children as guinea pigs. And this might happen to include my children. When my friend recruits for her trials, she has to follow very careful procedures to select suitable test subjects, and the people involved in the trials get paid. While I’m sure there are careful procedures in education, you can’t exactly remove your child from school if you don’t want them taking part in a trial. I’m fine for my children’s class teachers to try out new approaches that they think will suit my child; I’m not fine for my children to be used as guinea pigs for trials that could make money for someone’s company. I’m happy for my children’s teachers to make decisions based on experience and intuition; I’m not happy for my children to be involved in someone else’s expensive experiment. No, I’m not using evidence, but I’m not using prejudice either. I am exercising my professional judgement. And we used to set great store in that.

This entry was posted in Education, Evidence, Research. Bookmark the permalink.

10 Responses to What Works for What?

  1. Tim A says:

    Well said Sue.

    Alfie Kohn writes brilliantly on the issue in his book “What does in mean to be well-educated”. His general point is that we don’t have a definition of well-educated that we all buy in to. For everyone, like me, who likes to know a little bit about a wide range of topics there are others who have huge depth of knowledge about specific topics. We need a diversity of talents for society to flourish.

    I read blogs telling me it is VITAL that children are exposed to a canon of literature “the best ever thought or said”. That they are able to master mathematics and complex grammar whilst at Primary school. That evidence shows they will do better in exams if taught in complete silence, tested regularly and are never told they are doing well. Even that schools should dictate what time children go to bed, wake up and what they should eat for breakfast.

    School is not just preparation for the rest of life. It is life. I want my children to be exposed to a wide variety of experiences at school so they can find out what they enjoy. I also want them to be exposed to a variety of teachers, teaching styles, personalities and challenges.

    Liked by 2 people

  2. carmelohagan says:

    I agree with every word. So glad you have said it. This “research” is leading to all sorts of cults and factions. Money being used for the wrong reasons and quite frankly in some cases to further profiles and careers. In my experience much of it happens onTwitter and the vast majority of my educations friends including brilliant and well loved and well respected HTs have not heard of most of it. They are too busy caring about their colleagues, pupils and the community. So much reinventing of wheels. Thanks as always Sue for putting it so eloquently.

    Liked by 1 person

  3. Pingback: Research evidence or Professional Judgement?

  4. I agree that there are significant difficulties with respect to research in education. However, the same has been true with respect to research in most other disciplines, and only perseverence and much careful thought has resolved these such that robust evidence has been arrived at in, say, medicine. The issue becomes about how important the area to be investigated is. I wonder if the root problem is that education isn’t really important enough to invest the time, effort and money that would be required to robustly determine what can be determined, and then determine what are actually effective teaching methodologies?

    But, attempting to apply the principles of sound scientific research in education doesn’t constitute scientism, and until we arrive at the knowledge needed to distinguish appropriate pedagogy, teaching will remain susceptible to fatuous ideas.


    • suecowley says:

      Thanks for your comment. I think it’s the search for ‘an answer’ that leaves us open to fatuous ideas and I worry that ‘evidence based’ is all about that – just look at how the DfE/Nash etc. are behaving around it, while cherry picking what they use.


      • All approaches can be misused (either through ignorance or ideology), but the principles are still the principles. Attempting to find out if a practice is effective is, in principle, the right way to proceed, I think. If the approach adopted is to not seek improvements because this might lead to adopting fatuous ideas, the result will be no improvements. The solution would be to determine if the ideas are fatuous before implementing them. Unless you think that no improvements are needed?

        Liked by 1 person

        • suecowley says:

          I think consolidation of experience and more funding is needed mainly, but I have no problem with research in principle, just mainly in current practice.


  5. Excellent analysis Sue, another issue is why does the ‘evidence’ of some of the so-called experts differ so much? Here’s some examples – http://visablelearning.blogspot.com.au/p/other-researchers.html


  6. Nick McIvor says:

    I think you’re setting up a few straw men here: You make the point that schools are:
    “nothing like the controlled, clinical environment where medical randomised controlled trials take place.”
    You’re absolutely right. Everyone involved in educational research knows this and that’s one of the strongest arguments agains RCTs in education. In fact it can be extended to RCTs in all forms of social science. I don’t think that makes them all irrelevant. One of the things that encorages me about reports from the EEF is the fact that most interventions they investigate report zero impact – just like most drug trials.

    You criticise the EEF P4C trial on the grounds that:
    ‘We are measuring the value of philosophy for children by seeing what it does to high stakes test results.’
    That’s what EEF does – they look at the impact of interventions on the basis of the quantitative measures available, which at KS2 are their year 6 tests. That is not equivalent to saying ‘the only reason for teaching philosophy is to improve test scores in English and Maths’. One of the ‘intuitive’ arguments for teaching philosophy at primary level is that it will improve ‘higher level thinking’ and therefore impact on measurable academic progress. The EEF investigation is to get some insight into the validity of that claim. Even if they find zero impact (and experience would suggest that this is quite a likely outcome) that is not necessarily an argument for abandoning P4C, it would just encourage us to justify the practice in terms other than ‘it will help them to do better in their exams’.

    You make the statement:
    “I’m happy for my children’s teachers to make decisions based on experience and intuition; I’m not happy for my children to be involved in someone else’s expensive experiment”
    What are ‘ experience and intuition’ are if not a store of informal evidence possessed by an individual teacher? The trouble is, that store of information, without any external challenge orsystematic exploration can end up looking a lot like prejuedice. In fact all good teachers reflect on their practice continually and make small changes every way. In that sense, the whole education system is gigantic collection of ongoing experiments, it’s just that lots of people make all kinds of claims for cause and effect that remain unproven. All Kevan Collins and others are arguing for is a bit more caution about our claims for impact and a more organised attempt to investigate the assumptions we all make every day.

    Having said all of this, I think the usefulness of RCTs is limited and that any ‘what works’ style claims need to be looked at quite carefully before individual schools make dramatic changes. I should point out that I’m a teacher with an interest in educational research, not a spokesperson for the EEF. I think one of the most useful approaches for teacher development is to find ways of practically assessing the evidence for their intuitive practices.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.