In the last few years, there has been a huge upsurge in referencing source material when writing a newspaper/magazine/blog article. As a scientist myself, I think it’s a brilliant thing – giving everyone the opportunity to see the research the story is based on and judge for themselves whether it has been discussed in a non-biased way. Of course, there are still issues of free access to studies, but here I’m going to give a quick-fire summary of how to judge a paper efficiently, once you have it on your screen (or in your hands…).
When I was at uni, I was taught a number of strategies designed to help you quickly read and extract the important information – as a student, you have to read a LOT of papers… some are a nice, manageable 3-7pg in length, others can clock in at over 20pg and take hours to read and understand, so it’s good to know some shortcuts.
1) Go straight to the Methods
This is the single most important part of a scientific paper- if the methods don’t make sense, no amount of beautiful prose in the abstract and conclusions will change the fact that the data is not sound.
If they don’t mention inclusion/exclusion criteria, how they decided on their sample size and whether they lost any subjects along the way, how can you trust that the data they so proudly present is actually representative of the experiment they did? Without complete transparency on what DIDN’T make it into the study, how can you trust what did?
Especially if it is a clinical trial, the protocol (i.e. how they did it, their experimental plan) must be pre-registered, stopping them from changing their minds about what they are really looking for half-way through a study because the primary outcome is rubbish – calculations of how many subjects are required are based on outcome measure, so changing the outcome measure half-way through invalidates the group sizes, so you have no way of knowing if any failure to detect a response is due to there not being any effect or because there just weren’t enough individuals tested.
It also lets you use a bit of common sense – I recently read a paper which claimed to be a double blind, randomised control trial… which used blinding codes of “MG” and “CG” on pill bottles given to patients. This was a study looking at the effects of magnesium. The chemical formula of magnesium is… Mg. Admittedly not everyone might know this, but if you are told you are enrolled on a trial looking at effects of magnesium supplements, and are handed a bottled marked “MG”, it’s not that hard to figure out which group you are in, unblinding yourself, and making the study invalid. Just as an aside, if you ever see a study involving animal which claims to be “double blinded”, chuck it. Throw it away. That statement suggests the animals were unaware of which treatment they received… and shows up the authors as not understanding the concept of blinding (or masking as it is sometimes referred to).
2) Abstracts & Conclusion
This is a common way of reading papers, I’m sure some of you reading this do the same… but by relying on the 2 parts of the paper which are most open to interpretive bias, you don’t see the big picture. Authors want to draw readers in with a jazzy abstract that highlights what they think is important, and skims over anything considered weird, unexpected, or plain dodgy, and then in the conclusion, they want to leave the reader feeling that their work is super-exciting, super-relevant, and going to lead to innumerable life-improving breakthroughs, so by concentrating on these areas of a paper, you are basically allowing yourself to be spoonfed with exactly what the authors project as important, and that isn’t the same as actually being important and useful.
I was taught this while studying my PhD by another supervisor – his way of reading papers was to go straight to the last figure, the “Moneyshot” as it were. This will usually be the figure with the most impact, showing off the peak of the experimental findings – if that “looks good”, then read the rest… ok? NO. Not ok. By doing this, you are again falling in line with how the authors want their work to be seen. By all means judge the graphs and presentation of data, but don’t base your entire opinion of a paper on the pretty pictures – sometimes it just won’t add up. All sorts of dirty tricks can be used to make data look more sexy than it actually is – messing about with the axes (not starting at ‘0’ starts alarms bells immediately), showing raw data on some and ‘% response’ on others (makes me very suspicious – if the effect is there, you should be able to see it without resorting to data transformations of this type), and of course, if the error bars overlap, thre probably isn’t an effect – statistically significant doesn’t always mean biologically relevent,
So there we are – 3 different ways I have read papers in the past… number 1 is how I currently read papers, and although it might make you a bit angry and frustrated at first, being able to critically evaluate a scientific article will improve both your understanding, and if you work in the field, your own reporting as well.