Monday 23 April 2012

Research Versus "The Anecdote"

This is a post about the challenges of communicating research findings which have to go up against compelling but unrepresentative anecdotal stories, taking as an example the anecdotal attack on the family courts.

My colleague Peter Clarke used to joke that his students would "never let the facts get in the way of a good argument". In this post, I offer some thoughts on the difficulty that researchers often have in dispelling myths when they have to challenge an established public or media narrative that is supported by the all-powerful anecdote. Just as facts didn't stop Peter's students' arguments, research can struggle to overcome the rhetorical appeal of the anecdote. To paraphrase Peter, the media and the public "never let the research get in the way of a good anecdote".

Why anecdotes are compelling and research is not

Anecdotes have a number of qualities which make them great stories. First, they have human interest – we get to learn about a real person and to pry into their personal experience in some way. Second, they are usually quite simple – events unfold in an understandable way, and the consequences are easily explained. Third, they are believable because they (at least supposedly) actually happened. And finally, anecdotes are usually dramatic and extreme – after all, that’s why the story is being told. It’s like the news – only unusual or dramatic events are worthy of being reported, so you never hear:

“This morning, a woman got into her car, drove to work, and arrived there 20 minutes later without incident. On arrival, she picked up the post, made a coffee and started work. In other news, James Smith aged 8 has successfully learnt the six times table.”

Those aren't news stories, and they wouldn't be good anecdotes, because they are normal events. On the other hand, car accidents or mail bombs are news and would make great anecdotes - they are dramatic and extreme.

Research struggles to compete with any of this, no matter how interesting it is. But if the research is relevant and well done, why should this be?  Part of the reason might be that research findings rarely have any of the qualities that anecdotes display.

Most research does, in fact, have a huge amount of human interest, which seems like a good starting point. But the problem for researchers is that we struggle to tell people about the human interest in our findings because, for ethical reasons, we anonymise our work and try to make sure that the people who help us with our work are not identifiable in what we say about it – so we don't talk about individual people and we don't reveal personal experiences.

Then there’s the problem that research findings are rarely simple. We find complicated things, and as researchers we are nervous about over-simplifying what our evidence shows. When asked a question about our findings, often the instinctive (but perhaps unhelpful) response for many of us is: “well, it depends”. We find it almost painful to have to give straightforward answers, in case we look like we are misrepresenting the evidence.

Research ought to be believable, though – but perhaps the problem is that the way we try to convince people that our findings can be trusted is to tell them about our methodology – and let’s face it, that’s boring. Anecdotes don’t have to convince you of their veracity, because they come pre-stamped as being “actual events”, whereas research is an overview which can be harder to understand if you aren’t a professional researcher.

And then finally, dramatic and extreme? No, research findings are rarely dramatic or extreme. In fact, a lot of the time what we see is the exact opposite – we’re finding empirical confirmation of normal and everyday things. If you did research on people’s morning routines, most people’s would be as unexciting as my non-news story of a woman’s drive to work.

So all of this means that researchers can find it hard to give their findings traction in the public consciousness, particularly when there’s a good anecdote or two which contradict what the researcher is trying to say. Sometimes these mismatches between empirical reality and anecdotal rhetoric are annoying but insignificant - but at other times, they are dangerously misleading and potentially very damaging.

I'm going to use the second half of this post to illustrate what can happen when anecdotes trump research by taking an example from my area of research, the family justice system.

A practical example - the family courts 

The thing that has got me thinking about this at the moment is the on-going attack made by Camilla Cavendish and The Times on the family courts of England and Wales, the latest instalment of which appeared last week. This has been going on for some years now, with Cavendish writing about what she calls the "scandal" of the family courts of England and Wales. The cause of this problem, Cavendish thinks, is that the family court cases are "secret", because they are heard in private and the media are not normally in attendance. The answer, therefore, is to “open them up” to allow the light to shine in by allowing the media (and perhaps the general public) to go to family court hearings and then tell people about what they see.

I think that Cavendish is probably well-meaning, and perhaps genuinely sees herself as some crusader for justice, but her ill-informed criticisms of the family justice system based on anecdotal stories which ignore consistent research findings are wrong-headed and dangerous. Her ideas might sound sensible, but I think that the idea of "opening up the family courts" is truly misguided.

The reason that the media and the public are kept out of family court hearings is not because they are “secret” but because they are private. We are talking about incredibly sensitive, personal events in people’s lives – divorces and separations, allegations (which might or might not be true) about parents neglecting or abusing their children, questions about whether a child should have a dangerous medical operation or not, women seeking protection from physical, mental or emotional abuse by their partners or other family members, and so on. These are things that people are entitled to keep quiet, and they should be able to get help from the court without the rest of us standing on the sidelines gawping.

Leaving the merits of the debate aside, though, the real problem here is the way that the rhetoric has been taken hostage by anecdotes and stories. Cavendish and others have constructed this debate as being about "secret justice", frequently using individual cases to make vast sweeping statements about the family courts of England and Wales. Cavendish’s most recent rant damned the entire system based on one extreme case (which, incidentally, was largely taking place in the public gaze of the criminal courts, not the private sphere of the family courts) and, for great comment on that I'd recommend Lucy Reed on the Pink Tape blog. As Reed says, one complicated case where it turned out that the medical evidence that the court had to rely on was wrong does not indicate that there is a systemic problem with the family courts. There are thousands upon thousands of family court cases every year, and the vast majority proceed carefully and correctly.

Obviously sometimes people aren’t happy with the outcome of their case, but that seems unremarkable. When people go to court, sometimes they lose and sometimes they are unhappy about that, especially when the case was about something that was personally important to them (their children, for example). But just because people aren't happy doesn't mean that the system is broken. In cases where something has actually gone wrong, we have appeal courts – and, if you look at those cases, appeal judges are not shy of criticising their junior colleagues when things go wrong. (Most Court of Appeal cases are available free of charge on the British and Irish Legal Information Institute’s website.)

The reason that this matters is that the anti-family court narrative that is currently in vogue is undermining public confidence in an important institution. The family justice system isn't perfect, but in the vast majority of cases it does a good job in difficult circumstances. The independent Family Justice Review in 2011 made minor suggestions for changes to the system, but on the whole was positive in its conclusions about the workings of the system. The people who work in family justice - lawyers, judges, social workers, mediators, financial advisers, guardians and others - are mostly dedicated and hard-working, and they are doing their best to find reasonable solutions to complicated and highly conflicted problems. We shouldn't allow unrepresentative anecdotes to lead us to think that there is some scandal in the family courts, no matter how good the story sounds.

Author's note: I wrote a briefing paper on media access to family
court cases in 2009, funded by the Nuffield Foundation. While I
conduct research into family law and the family courts, I do not
work in the family justice system and have no personal interest in it.