Introduction

A peer reviewer plays a vital role in improving the quality of the manuscript reviewed. In a larger context, scientific peer review also protects the quality and integrity of scientific research. Your peers rely on this work to understand what research to trust and build on, leading to better, faster science.

Your manuscripts will also improve because, over time, you’ll learn how to use your knowledge in peer review to fine-tune your own papers.

Peer reviewing will help you evaluate the importance and accuracy of research questions; the appropriateness of methodological and statistical approaches; and build up a set of best-practice tips to prepare and organize your project. And finally, by learning common errors to watch out for when peer reviewing, you’ll inevitably learn to avoid the same mistakes in your own work.

Recommended Steps

  1. Skim the paper very quickly to get a general sense of the article. Underline key words and arguments, and summarize key points. This will help you quickly “tune in” to the paper during the next read.
  2. Sit in a quiet place and read the manuscript critically. Make sure you have the tables, figures and references visible. Ask yourself key questions, including: Does it have a relevant title and valuable research question or hypothesis? Are appropriate papers referenced? What’s the author’s motivation for the study and the idea behind it? Are the data and tools suitable? What’s new about it? Why does it matter? Are there other considerations? In particular, pay attention to the following:
    1. Do the title and abstract cover the main aspects of the work, and would it spark interest in the right audience?
    2. Is the Introduction easy to follow for most readers of this particular level (CS seniors)? Does it provide a hypothesis or aim of the study? Does it explain the background of the project? Does it cite the appropriate sources (it is very rare that an introductory section doesn't need citations)?
    3. Does the Discussion/Related Works address the main findings, and does it give proper recognition to similar work in this field? Does the Related Works section clearly convey an overview of the chosen area and explain the relationship of the current paper to previous work?
    4. Does the Methods/Our Approach section provide enough details for the general reader to understand and/or repeat the experiments?
    5. Does the Results section give the right amount of detail to understand the basic outcomes of the experiments?
    6. Do the Results refer to the figures in a logical order? Do the numbers in the tables add up correctly? Are any figures/tables mislabeled or unclear?
    7. Given the data that was used in this study, did the authors perform all the logical analyses? Did they include the proper controls/comparisons?
    8. In general, is the paper easy to follow and does it have a logical flow? Are there any language issues?
    9. Did the authors make all their data (e.g. sequence reads, code, questionnaires used) available for the readers?
    10. Is this paper novel and an advancement of the field, or have other people done very similar work?
  3. Take notes about the major, moderate and minor revisions that need to be made. You need to make sure you can put the paper down and come back to it with fresh eyes later on. Note-taking is essential for this.
  4. Are there any areas of ambiguity or vagueness?
  5. Create a list of things to check. For example, does the referenced study actually show what is claimed in the paper?
  6. Assess language and grammar. Does the paper flow? Does it have connectivity? Does it have clarity? Are the words and structure concise and effective?
  7. Summarize your notes. This can include overview, contribution, strengths and weaknesses. You can also include the manuscript’s contribution/context for the authors (really just to clarify whether you view it similarly, or not), then prioritize and collate the major revisions and minor/specific revisions into feedback. Try to compile this in a logical way, grouping similar things under a common heading where possible, and numbering them for ease of reference.
  8. Use the paper's section numbers, page numbers, etc. to specifically identify problem areas and your suggestions for improvement so that the author knows which parts of the paper you're referring to.
  9. In addition to describing ways in which the paper can be improved, discuss parts of the paper that you thought were particularly strong, e.g. things that were explained in a lot of detail, arguments that were made very persuasively, thorough investigation of a particular aspect of the solution, etc. This helps the authors know that those parts can be left as-is in future versions of their documents and don't need further revision.

Common Flaws to Watch out for

Try to look beyond the inevitable (but minute) grammar and wording problems and focus on identifying more structural problems, such as:
Over-interpretation of results
Over-interpretation has no place in scientific work. Ensure the conclusions drawn in the paper are based on the data presented and are not extrapolated beyond that (to a larger population or setting, for example). You should also watch out for studies that focus on seemingly important differences where none exist.
Lack of evidence to support conclusions
A paper's opening and concluding statements must be justified and evidence-based. If you're not convinced of the results, it could mean the researchers need to clarify aspects of their methodological procedure, add more references to support their claims, or include additional data or further analysis.
Not addressing threats to validity.
Almost every study or experiment makes some sort of assumption about the applicability of the results to a different or broader situation. The paper should acknowledge these threats to the validity of the experimental results and explain how the researchers mitigated or tried to minimize these threats.
Too many words.
A common pain point in manuscripts is that it’s too wordy. It’s important to keep check on this scenario and encourage clear, concise and effective text where possible. Too many words can be distracting for the reader, which at best could cause them to lose interest and at worst could lead to them misinterpreting the work.
Too few words.
Asking for more details to support claims is also common. Watch for vague statements that come out of nowhere and if you have problems understanding anything or concern on completeness and/or concreteness, you should ask for more clarifications. It is entirely possible for a paper to have both too many and too few words

Citation

Much of this text was adapted from D Xu (2021).