Tracey Brown asks what constitutes evidence

Never mind how the government uses evidence, let’s wind back a stage and ask, exactly what is scientific evidence in the policy world? Because, in practice, we seem to be confronted by a wide range of interpretations of that authority-bestowing word.

MRI case study

In July, the Science and Technology Select Committee released the first case study in its forthcoming report on scientific advice. It looked at how we have nearly reached the point of introducing EC occupational health rules on exposure to magnetic resonance imaging, without evidence to support them. Such data as does exist has been described by every scientific body as outdated. The protestations of the scientists and clinicians were just brushed off.

This report is among the most critical ever produced by the Committee, taking in the failings of Commission officials, advisory boards and the UK’s Health and Safety Executive, all of whom seemed to say different things about the evidence.

Other issues

Aside from MRI restrictions, the Select Committee has looked at drugs policy and identity cards. It might equally have considered regulation of transport safety, guidelines on sun exposure, the Human Tissue Bill or banning full fat milk in schools. The treatment of evidence varies from policy initiative to regulation, from department to agency.

The Food Standards Agency, set up amid calls for independent scientific evidence, is now citing a consumer survey as ‘evidence’ in a row about the health benefits of sticking coloured ‘traffic light’ labels on foods. The Highways Agency will doubtless be pleased to learn that people find traffic lights ‘a simple system’. Whether sticking them on food will reflect nutritional needs or change behaviour... well, scientists are interested but the agency doesn’t appear to want this kind of evidence.

Despite the Office for Science and Innovation’s introduction of departmental chief scientists, the problems are wide-ranging and defy easy translation into rules that Sir Humphrey can police. But the Select Committee’s report might prompt a start. Here are some suggestions:

Rules of engagement

Rule One: the status of scientific evidence is as important as its conclusions. This does not mean the unusual or untested is ignored – as the experience of badgers and bovine TB shows, sometimes you have to look beyond what exists and seek new evidence. But regard for status does mean that you should know what you are looking at. Is it peer reviewed and published in a journal recognised in the field? Have findings been cited? Repeated? How stable is the science? What do citations say? Is it a big study? A case report? Is it a chap reporting earth-shattering results from experiments conducted alone, in his garage, ten years ago and still not published? I might be sending it up there, but leaf through the referenced submissions of recent enquiries and you’ll see why more of these questions would be worthwhile.

Rule Two: there are no rules except Rule One. That is, there is no box to tick, no one-size-fits-all ‘good evidence’ form to fill. It’s a question of judgement and, ultimately, responsibility. This is difficult for government and it is what will make it so hard to respond effectively to the Select Committee report. Few are anxious to be in that accountable seat. Too frequently we have policy initiatives dressed as evidence, so responsibility lies everywhere and nowhere.

Evidence to fit policy

Government is also not going to be able to respond to the report’s conclusions while evidence is treated as that thing you find to fit the policy, sometimes when the policy announcement is emerging from the printer. The now Lord Waldegrave, whose BSE experience informed  the first Chief Scientist’s guidelines, warned against the pressure for quick answers. But a pragmatic approach to the facts is being encouraged by a speed of policy development that precludes appreciation of evidence.

Science, by contrast, makes a pretty good job of dispassionate review, through critical exposure to peers, self-questioning and a humility before evidence that doesn’t suit the policy world. Nor should it, but that doesn’t remove the need to know which questions to answer scientifically, when officials would do well to mimic the critical scrutiny of science.

Scrutiny

As a matter of accountable judgement, we can’t have a standards committee for evidence; we can only examine the way that questions are pursued in particular cases.

The Select Committee has promised to revisit some of its cases next year, to review progress. Perhaps this kind of challenging scrutiny of judgement is the best way forward where rules won’t do.

Tracey Brown is the Director of Sense About Science