I outlined the style of good academic reviews to Jonathan in light of our impending OSDC review responsibilities, and it’s worth noting here too.
For information’s sake, my authority, such as it is, on reviewing comes from being the editorial assistant of Computational Linguistics, which is a journal with a hardworking editor and conscientious reviewers. Not all academic reviews are of the quality I discuss below. They should be.
Begin with stating the title of the paper you are reviewing. Then spend one to three paragraphs summarising its content, particularly what you perceive as its major findings and conclusions.
This has a couple of purposes. The first is that if the reviews have got mixed up in the system the author finds out as soon as possible and doesn’t have to slog through a review that (perhaps) is a partial match for their paper and (especially in academic circles) a privacy problem to boot. The second is so that they know in what light to read the rest of the review. If they see that you have understood its fundamentals they will be inclined to take the entire review seriously. If they see you have misunderstood it, they can do one of two things. One is to realise that their paper is confusing, and to make its focus clearer. The other is to discount your review. The decision here may be affected by the following section.
The main body of the review is a discussion of how to improve the paper. Both the tone and discussion will vary considerably depending on certain factors:
- is the paper already accepted?
- is this the only reviewing round or will you or another reviewer be checking the changes?
For OSDC, both factors hold. For almost all conferences, there is only (at most) one reviewing round for full papers. This makes reviews more limited in scope than journal reviews, where substantial changes are often recommended even (or perhaps especially) to articles the reviewer fundamentally likes. Journal reviewers can have a role which is not far from being anonymous co-authors. (If a colleague did as much re-reading and suggestions of additional work and additional reading as Computational Linguistics reviewers do, many people would consider adding them to the authors list.)
In the event that the article has been accepted, or that this is the single reviewing round, you should limit the scope of your suggestions to much more cosmetic things. Someone who has had an article accepted is just going to be annoyed that you want it to have a whole new body of work incorporated, and they will ignore you. (And if it’s rejected after a single reviewing round, they are probably ill-placed to revise much!) In the OSDC scenario, reviewers are going to be mostly limited to suggestions as to how to structure the argument and the paper better, and not really able to productively suggest changes to the argument or the work described in the paper.
As you write your review and this section in particular, keep in mind the key factor of providing useful critiques: how could this work be better on its own terms?
That is, don’t provide a review that is, fundamentally, about how the paper would have been better if you’d written it… about your pet topic. This is a subtle, tempting and common mistake, and if you have never caught yourself in it, you are likely to be the worst affected. Remember: What is the paper trying to do? How can it do it better? Avoid the temptation to suggest that it would be a better paper if it was doing something different from its current aim. (There is a little more leeway for this in journal reviews, but even in that case, generally what happens if a reviewer thinks this is that they review the article on its current form and recommend a fate suited to its current aims, and additionally comment that they would be interested in seeing further work in the additional direction should the authors choose.)
As a recipient of reviews, I do have a couple of things to add. One is to respect page limits. If you are reviewing for a work with a page limit, especially a conference, and you do really want to see a longer discussion of foo, please suggest which bar could be shortened or cut. Otherwise it is close to impossible for an author to consider your suggestion. Also, if you are making suggestions for future work that you think the authors should consider but which you do not actually want to see in the article, make this clear in the text of your review. I would probably recommend a whole separate section for this if you’re going to do it.
A review may conclude with a list of typos, spelling mistakes, suggested rephrasings, etc. Mistakes that affect the reading of the paper (eg mislabeled figures and sections) go right at the start of this list. A sufficiently ill-proofread paper may go back with a suggestion that the authors find the mistakes themselves.