Three rules for critiquing studies

Modern evidence-based medicine fundamentally requires a continuous flow of new evidence. As obvious statements go, it’s still a reflection on our healthcare practice today. But with thousands of new articles published on a monthly basis, the task of simply keeping up on the literature can quickly become overwhelming. But it doesn’t have to.

Although not specifically about how to read a scientific paper, “How To Read A Book” can teach us principles that hold true across any form of written media. We’ve already examined the four key aspects of reading in previous articles. But to remind ourselves of them those four elements are:

  1. Elementary reading (Beginning, middle, end)
  2. Inspectional reading (skimming systematically – get the most out of a book within a given relatively short time yet too short of a time to get everything that could be gotten)
  3. Analytical reading (Organized, thorough, complete – the best reading you can do)
  4. Syntopical reading (Comparative reading – reading many papers/books at once and places them in relation to one another and to a subject about which they all revolve)

Analytical reading is the stage in which the most is gained from a given manuscript. That is, of course, after systematically skimming through the less important or relevant literature. What’s left is likely to be impactful to your practice, and thus worthy of the time invested into understanding its content. Aside from reading, re-reading and even more re-reading, there are tools we can employ to derive the most out of each manuscript by applying tested techniques and methods. 

There are three stages of analytical reading. The first stage of analytical reading is to determine what is the article about as a whole. The second stage of analytical reading asks what is being said in detail, and how? Lastly, the third stage of analytical reading outlines the rules for criticizing an article or manuscript as a communication of knowledge.

This last stage is less about actually reading and more about challenging the content of the paper itself. This third stage of analytical reading where the rules for criticizing an article or manuscript as a communication of knowledge are outlined consists of the following elements.

  1. Before beginning any critique you must first complete your outline and your interpretation of the article (Do not say you agree, disagree, or suspend judgment, until you can say “I understand”)
    • Somewhat intuitive, but many arrive at a manuscript with prior biases and attitudes towards a given drug, intervention or procedure. For those in the FOAMed world, consider any article related to rt-PA for stroke. It’s difficult to put aside these feelings while being objective in current and newly gained knowledge.
  2. Demonstrate that you recognize the difference between knowledge and mere personal opinion by presenting good reasons for any critical judgment you make
    • Many papers may conclude something different from what the data would suggest, or irrelevant to the pre-defined objectives of the study. Be critical of these conclusions as an opinion and not new knowledge.
  3. Support these critiques by identifying specific elements within the methods, results, or discussion as to where the author appears to be uninformed, misinformed, illogical, or the analysis is incomplete.
    • Drawing specific passages from the discussion and relating them to the manuscript is a key element to this. Furthermore, if there have been “letters to the editor/authors” submitted related to this manuscript, scrutinize the authors’ response to such letters. Do they acknowledge new limitations not included in the manuscript?

These critiques and analyses should be done in good faith, however. It is easy to become cynical and confrontational towards any paper when critiquing it in this manner. As an author of many studies myself, we have to acknowledge that even in the best-designed studies, there are unintentional omissions, or considerations not fully accounted for. As such, fully removing yourself emotionally is a skill that should be developed alongside this final step in analytical reading.

Three rules for critiquing studies
Three rules for critiquing studies
Three rules for critiquing studies