I outlined the style of good academic reviews to Jonathan in light of our impending OSDC review
responsibilities, and it's worth noting here too.
For information's sake, my authority, such as it is, on reviewing comes from
being the editorial assistant of Computational Linguistics, which is a
journal with a hardworking editor and conscientious reviewers. Not all academic
reviews are of the quality I discuss below. They should be.
Begin with stating the title of the paper you are reviewing. Then spend one
to three paragraphs summarising its content, particularly what you perceive as
its major findings and conclusions.
This has a couple of purposes. The first is that if the reviews have got
mixed up in the system the author finds out as soon as possible and doesn't
have to slog through a review that (perhaps) is a partial match for their paper
and (especially in academic circles) a privacy problem to boot. The second is
so that they know in what light to read the rest of the review. If they see
that you have understood its fundamentals they will be inclined to take the
entire review seriously. If they see you have misunderstood it, they can do one
of two things. One is to realise that their paper is confusing, and to make its
focus clearer. The other is to discount your review. The decision here may be
affected by the following section.
The main body of the review is a discussion of how to improve the paper.
Both the tone and discussion will vary considerably depending on certain
factors:
- is the paper already accepted?
- is this the only reviewing round or will you or another reviewer be
checking the changes?
For OSDC, both factors hold. For almost all conferences, there is only (at
most) one reviewing round for full papers. This makes reviews more limited in
scope than journal reviews, where substantial changes are often recommended
even (or perhaps especially) to articles the reviewer fundamentally likes.
Journal reviewers can have a role which is not far from being anonymous
co-authors. (If a colleague did as much re-reading and suggestions of
additional work and additional reading as Computational Linguistics
reviewers do, many people would consider adding them to the authors list.)
In the event that the article has been accepted, or that this is the single
reviewing round, you should limit the scope of your suggestions to much more
cosmetic things. Someone who has had an article accepted is just going to be
annoyed that you want it to have a whole new body of work incorporated, and
they will ignore you. (And if it's rejected after a single reviewing round,
they are probably ill-placed to revise much!) In the OSDC scenario, reviewers
are going to be mostly limited to suggestions as to how to structure the
argument and the paper better, and not really able to productively suggest
changes to the argument or the work described in the paper.
As you write your review and this section in particular, keep in mind the
key factor of providing useful critiques: how could this work be better on
its own terms?
That is, don't provide a review that is, fundamentally,
about how the paper would have been better if you'd written it... about your
pet topic. This is a subtle, tempting and common mistake, and if you have never
caught yourself in it, you are likely to be the worst affected. Remember: What
is the paper trying to do? How can it do it better? Avoid the temptation to
suggest that it would be a better paper if it was doing something different
from its current aim. (There is a little more leeway for this in journal
reviews, but even in that case, generally what happens if a reviewer thinks
this is that they review the article on its current form and recommend a fate
suited to its current aims, and additionally comment that they would be
interested in seeing further work in the additional direction should the
authors choose.)
As a recipient of reviews, I do have a couple of things to add. One is to
respect page limits. If you are reviewing for a work with a page limit,
especially a conference, and you do really want to see a longer
discussion of foo, please suggest which bar could be shortened or cut.
Otherwise it is close to impossible for an author to consider your suggestion.
Also, if you are making suggestions for future work that you think the authors
should consider but which you do not actually want to see in the article, make
this clear in the text of your review. I would probably recommend a whole
separate section for this if you're going to do it.
A review may conclude with a list of typos, spelling mistakes, suggested
rephrasings, etc. Mistakes that affect the reading of the paper (eg mislabeled
figures and sections) go right at the start of this list. A sufficiently
ill-proofread paper may go back with a suggestion that the authors find the
mistakes themselves.