Writing reviews for a journal is one of those things which most scientists never actively learn. For laypeople, this may be surprising, given how often the scientific method with its rigorous peer review procedure is being mentioned in the news nowadays. How can it be, one may ask oneself, that this procedure that is usually presented as the core principle of scientific reasoning, is never really actively taught? If the review by experts is the core of the scientific method and what decides about the acceptance of an article, how can it be that scientists do never take a course on article reviewing, and how can it be that reviewers are (as I have previously discussed in a German blogpost) themselves never reviewed or graded?

I have no real idea why this is the case. Maybe, it reflects just the way humans behave normally? We are all a bit conservative and reluctant to change. The moment we write our first review, we may shout and complain that there’s no real instruction of how to do it. But at the same time, we feel so proud that we were finally asked to write one, that we forget all the problems we had in the moment we submitted our first review. When we then meet younger colleagues who are in the same situation in which we once were a long time ago, we just tend to think, as again many people do: don’t complain, young person, I also had to go through this, why should you have it any easier than me?

This situation of not having clear review instructions and not having any clear process by which scholars sit down and evaluate our reviews is very unfortunate, specifically also because writing reviews is a complex business. I realized this again when I sat down to prepare this blogpost, since I could not find a way to prepare a straightforward how-to-guide for writing a review in linguistics, since I do not even know if people would consider my own reviews as useful. So what I will provide in this post is less a full guide of writing a review, but more an at times loose collection of ideas and thoughts on the topic of review writing that have been accompanying me during the past years.

What is an initial review for a journal?

Before we start with some concrete suggestions and thoughts, I have to clarify what I mean by an “initial review for a journal”, since in science, there are quite a few different kinds of reviews one can write. In contrast to a review article, which summarizes the state of the art in a given research field, a book review, which critically evaluates scientific monographs, and a review for a grant proposal, where one decides whether a scholar should receive funding for a project or not, a peer review for a journal refers to the report a scientist writes on a journal article that was submitted to a journal for publication and which is supposed to help the editors of the journal to assess whether the article merits publication in their journal or not.

I distinguish the initial review from the follow-up review. The initial review is the report one writes upon the first submission of an article. Since articles are often not directly accepted, but rather sent back to the authors with a list of change requests, typically labelled as “minor modifications” (almost accepted, no follow-up review required) and “major modifications” (article has to be reviewed another time), there is a considerable difference between the review one writes upon initial submission and the follow-up review. While the first needs sufficient detail to summarize and assess the paper, the follow-up review may be quite short, even just a sentence at times, provided the authors have convinced the reviewer with their modified manuscript.

In this post, I will share my thoughts regarding initial reviews for a journal. When receiving manuscripts for publication, the editors of a journal select reviewers who can help them to assess the quality of the work, and then invite them via email to share their thoughts. In the following, I will try to run quickly through the major stages of writing an initial review for a journal, thereby discussing (1) what one should consider before accepting to review a paper, (2) initial quality checks of the study, (3) the preparation of the review report, (4) the active writing process, and (5) how to decide on a recommendation.

1 What one should consider before accepting a review invitation

When being asked to review, one should first make some background check on the journal. This is not only helpful in order to find out if the journal is not a predatory publisher, or one of those journals which try to maximize profit by bothering potential reviewers with tons of automated emails (I had the most annoying experience with the Frontiers journals so far, but I am sure there are journals out there who do worse), but also to understand the journal’s basic review procedure. Some journals have set up very fancy review practices and guidelines, which can require a lot of work (Frontiers journals are again an example), some publish all reviews online (MDPI journals are an example here), some require non-anonymous reviews, and some have very explicit forms one needs to fill out. Although there are not many reasons to not make a review for a given journal, it is useful to keep these questions in mind, since it will also influence how much time one will have to devote to the review.

As a second step, one should read the abstract, and make sure that one feels equipped to evaluate the research. If this is not the case, one should politely refuse to review the study and contact the editors, recommending colleagues who could do the review instead.

2 Initial quality checks

After having accepted to review a paper, one needs to organize oneself and decide on which day one wants to write the review (reviews should not take longer than one day, although younger scholars may feel they need a bit more time for this, and it may be helpful to do a review in smaller pieces over several days, in order to avoid that one writes it in a biased mood).

At the same time, when having received the electronic version of the paper, one should make sure that the basic conditions of the review have been met. This means that one should check (1) if data and code are required, they are offered in editable form, so that one can test them on one’s own computer, and that one should make sure that (2) there are no conflicts of interest with respect to the study.

If one detects conflicts of interest (e.g. that one works on the same topic and is about to submit a similar paper to another journal), one needs to withdraw the review. To help the editors in finding a good reviewer as replacement, it is always useful to recommend a colleague who could do it. If data and code are not submitted, one should inform the editors and ask them to contact the authors so that they can (1) submit data and code and (2) confirm that data and code will be shared upon publication as well. In order to make sure editors understand the urgency of this claim, it is useful to cite recent studies on open research, such as Nature’s (2018) editorial that emphasizes reviewers rights to request data and code during the review process). It is also useful to point editors to the principles of open science and FAIR management of data and code (Wilkinson et al. 2016. Last not least, it helps to point to one’s institute’s policy of not supporting irreproducible research.

Referring to one’s institute’s review policy is especially helpful to emphasize the importance of the claim, while at the same time making sure the editors do not blame one for being overly pedantic. The past years have seen an abundance of studies in which data and code were not submitted and in many cases, authors have even refused to share them upon request. One may argue that our research is less harmful when not being substantiated with data and code, since there are no consequences of our findings (contrary to the missing data in medical research). But if we want to stand up for the principles of scientific research, we need to stand up for the basic principles of transparency in our research. If I start trusting somebody who told me a dated phylogeny for some language family yields 6000 years as divergence time without supplying data and code, what is the difference in trusting somebody who claims to have turned water into wine without providing the chemical formula?

When authors point out that data sharing would expose problems of privacy concerns or similar, it is always possible to find appropriate solutions. Copy-righting issues, for example, can be creatively handled in many cases, by sharing parts of the data. If parts of the data consist in the form that it would impact on the privacy of individuals, scholars can share the data in a maximally anonymized form (see List 2020a for an example on sharing copyrighted rhyme data). If the data are not available completely, due to unclear licenses, one can submit code that allows the readers to crawl the data directly (see the approach in Tjuka et al. 2020 as an example).

All in all, it is important to emphasize that there is no reason to submit research that is irreproducible, and one should refuse to review a study in case editors do not help in this regard or authors do not comply with it. In this way, one will not prevent the study from being published, but one can raise concerns and does not contribute to endorse bad scientific practice.

3 Preparing the review report

There are different strategies one can follow when it comes to prepare the review report. It may, for example, be useful to first read the paper and make an excerpt, in which one copy-pastes quotes along with short comments (similar to the EvoBib collection, cv. List 2020b). But this always depends on the initial impression that a study makes on the reviewer. If the study seems to be good but with a few things that could be enhanced, it is probably best to read it and make some notes to oneself. However, if the study is significantly flawed, and this is already clear from the beginning, it may not be worth to read it in all its details, but rather pick out the most important points to confront the authors.

It may happen to reviewers, especially when they are younger, that they feel intimidated by the overly complex style in which a study is written. As a result, reviewers may endorse the study in order to avoid that they have to admit that they did not really understand it (as it apparently happened as part of the Grievance Studies Affair).

As a rule, reviewers should be honest about their confidence when criticizing a study. This means also, that, when being confronted with a huge bunch of mathematical formulas which one does not understand, one has the right to emphasize that one cannot judge this oneself but recommends the editor to look for a statistician. The same should apply to text that is hardly readable due to an exaggerated use of non-standard terminology. Additionally, one also has the right to criticize overcomplexity. Scholars should be able to adapt their articles to their readership. At times, one has the impression that formulas and exaggerated terminology are used to intimidate readers rather than to inform them. Good reviewers and editors are needed in order to spot these cases.

While I usually try in my reviews to understand in full what methods and arguments authors have been using, I also often encounter situations where I simply have to admit that a given study goes beyond my expertise. Since I know that editors tend to have huge troubles in finding enough reviewers for their submissions, I try to still write a review in these cases, but I always make it clear to the authors that I only comment on the points where I feel competent, and I repeat this also in a personal letter to the editor, to make clear that my review should not weigh as much as an alternative review by somebody who understands the study in all detail.

4 Writing the review report

Before writing the review report, one should check the journal’s online system, as some journals have very specific questions. If this is not the case, one will have to write a report in free form, here it is recommended to follow some basic structure, even if there are no clear guidelines for free reviews.

For an initial review one should always provide a summary of the paper. This helps to illustrate that one has understood the study properly. This summary should then be accompanied by a short recommendation and assessment. Here, most journals distinguish the “magic ABCD”, namely (A) accept without modifications, (B) accept with minor modifications, (C) revise and resubmit, and (D) reject. Unless one accepts a study without modifications (which rarely happens anyway),
more detailed comments on major issues should follow in a second section, and minor comments (spelling, layout, etc.) should be summarized in an additional section. References should be provided in a final section.

While this structure is by no means required and only stems from my own experience, it is useful for authors to revise their studies afterwards, as it clearly states what is considered as important major revisions, and what can be done quickly in the form of minor fixes.

Not all reviewers provide detailed references for literature they mention, but I consider this as an extremely bad style. Just mentioning that some Shannon said something in 1993 about juggling and science, for example, is bad style, as the authors have no real way to verify it, especially if the names are common. Even if there is only one Shannon who wrote an article on juggling and science in 1993 (it happens to be Claude Shannon, 1916-2001, the father of information theory), the rigor we demand as reviewers from the authors should be the same we demand from ourselves as reviewers writing a review.

It is not always easy to guarantee one’s anonymity, as it may be obvious from comments of a reviewer who one is, especially in areas where there are only a few experts. Therefore, one should always write as if the review was openly accessible to anybody, assuming that the authors know one’s identity and that colleagues can see one’s review along with the article. While this seems to go against the original idea of blind peer review (which is supposed to shelter younger scholars from the revenge of senior scholars when submitting demanding reviews), it has helped me a lot to refrain from being polemic and trying to be constructive instead. While anonymity may protect reviewers, it is not a wildcard to be offensive. Unfortunately, not all colleagues understand this.

5 Deciding on a recommendation

Although the “magic ABCD” of recommendations looks rather straightforward and simple, it may be difficult to decide on a good recommendation. Since there is less of a competition for the best journals in the humanities and editors are usually glad if they can fill their next volume with enough articles of an acceptable quality, it does not happen too often that papers are directly rejected if they fulfill general standards of scientific quality. Similarly, given that “accept without modification” means that the article is judged to be in a state where it could be published tomorrow, it is also unlikely that any study will fulfill these demands upon first submission, and my experience is that all articles which I have submitted in the past have greatly profited from critical peer review. As a result, reviewers will usually pick one of the two lighter options and recommend a resubmission with major revisions or to accept the paper with minor modifications.

Reasons for a full rejection are: (1) extreme hypotheses in a paper that are falsely confirmed (e.g., by wrong methods), (2) claims for new methods that have been developed although these methods already exist, (3) scientific misconduct (e.g., plagiarism), (4) the study appears to be out of topic for the journal.

Papers need a major revision if one feels that (1) whole passages should be rewritten and restated, or (2) that data and code have not been submitted and need to be inspected in a second review round. For minor revisions, where it is not guaranteed that one will see the paper again as a reviewer in revised form, these recommendations can be given when only few things need to be added, like footnotes and references, and the majority of the study seems to be convincing enough.

No matter what recommendation one gives to the editor: unless the paper is a complete disaster, one should always try to encourage the authors, even if one feels that there is still a lot to be done. Even if I recommend to reject a paper, I try to give the authors some suggestions on how they could turn their work into a successful study. A good review, even if it is a rejection, helps the authors to improve.

Final remarks

This collection of ideas on how to write an initial review for a journal has taken me more time than I originally expected. It is probably also less organized than I initially hoped, and I even do not know if it is exhaustive enough to serve as a useful guideline for those who have yet to write their first reviews. However, despite the fact that I am not necessarily content with this contribution, I hope that it could help to contribute to a future debate about reviewing in linguistics in specific and in science in general. The fact that one of the most important aspects of scientific practice is barely discussed, taught, and evaluated should remind us all that the scientific method is a fluid collection of best practices in scientific research, it is not an approach that has reached perfection and does no longer need to be questioned.

References

• Johann-Mattis List (2020-08-24): General remarks on rhyming (From rhymes to networks 2). The Genealogical World of Phylogenetic Networks 9.8. .
• Nature, Editorial Board (2018): Referees ́rights. Nature 560. 409.
• Shannon, Claude E. (1993): Some. In: : Claude Elwood Shannon. Collected Papers. New York:IEEE. 850-864.
• Tjuka, Annika and Forkel, Robert and List, Johann-Mattis (2020): Linking norms, ratings, and relations of words and concepts across multiple language varieties. PsyArXiv 0.0. 1-24. [Preprint, under review, not peer-reviewed]
• Wilkinson, Mark D. and Dumontier, Michel and Aalbersberg, Ilsbrand J. and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz B. and Bourne, Philip E. and others (2016): The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3.