How to review a technical paper

This page contains some notes on reviewing technical papers, intended as a guide for students who are reviewing papers for the first time. If you are not one of my students and you are looking at this page, please note that the guidelines here are shaped by my experience in reviewing mathematically-oriented papers in the control systems community. Standards and approaches are somewhat different in other disciplines.

General comments

There are three general features that need to be present for a paper to be accepted for publication

• Correct - The paper needs to be technically correct and provide sufficient detail for correctness to be evaluated.
• Novel - The results should be something that has not been previously published and is more than just an obvious application of previous results
• Interesting - The paper should be of interest to the community represented by the journal or conference.

Most review forms will have some set of questions that try to get at these three issues, as well as other factors such as clear writing, appropriateness for the journal, etc.

It is important to keep in mind that your review is advice to the editor making the final decision, as well as feedback to the authors. The part of the review that gets sent to the authors should not explicitly recommend for or against publication, but rather give the assessment (against the criteria above) that lead to your recommendation (which typically goes on a separate part of the form). I usually find the following rough format to be useful for structuring my review:

• First paragraph: summary of the main contributions of the paper, as I understood them
• Next paragraph or two: high level assessment of the paper, including an assessment of whether the paper is correct, novel and interesting. If I didn't have time to check all of the technical details, I might say so here (or in the confidential comments to the editor)
• List of major comments: if there are major issues with the paper, I will usually include a bulleted list of things that I think should be fixed. You can anticipate that authors who resubmit their paper will tell you what they did to fix each of these, so make sure that the issues are as clear as possible.
• List of minor comments: if I found other minor things along the way that should be fixed but are not major issues, I list them at the end

In writing reviews, it is important to maintain the anonymity of the process: the authors don't know who the reviewers are and you should seek to write your review so that this is not apparent. This is trickiest when there is some of your own work that the authors didn't know about, but if you try to be fairly broad in pointing out other work that should be cited, this can help.

Conference papers

Conference papers in the controls community are "non-archival", so the level of contribution can be a bit lower. In particular, I usually concentrate on correctness and novelty for conference papers, figuring that whether it is interesting or not will be conveyed by the number of people who show up for the talk.

The main difference in reviewing conference papers is that the paper will be accepted or rejected based on the reviews, and then the process is done. So it is particularly important that the comments are clear and constructive, so that if the paper is accepted the authors can make revisions that improve the quality of the paper.

Sample review

Here's a sample review that I did for an article in PLoS One. This online journal has a policy of publishing anything that is correct and original, then letting readers determine whether it was interesting or not (by rating it). I told PLoS One that they could publish my review with my name attached, so I feel OK posting it here.

This paper analyzes a model of chemotaxis to explore how the frequency response of the sensing pathway affects the dynamics of the closed loop system. The model of the sensing pathway is based on a set of ordinary differential equations (derived by others) that capture the phosphorylation of CheY based on binding of a receptor protein and subsequent signal transduction. This model is linearized about a relevant operating point to determine the steady state frequency response (transfer function) and the properties of that transfer function (low pass versus band pass) are explored in the context of the behavior of the full model.

The authors explore two different classes of pathways: those with adaptation and those without. The pathways without adaptation correspond to a sensing pathway that behaves roughly as a low pass filter. The pathways with adaptation correspond to a bandpass filter, with no response at zero frequency, which provides a mechanism for insensitivity to constant inputs (ligand concentration).

The modeling and analysis approaches used by the authors are fairly standard ones in the control community, with some attempts to provide an understanding of the role of various features in the frequency response in terms of their biological relevance. The insights provided were a bit heuristic in nature and no experimental results were included or cited to validate the explanations. In particular, there are statements regarding the "disadvantages" or non-adaptive response and the "improvements" due to certain parameter choices that are not well justified.

Using the modeling and analysis of the natural system as a starting point, the authors also provide a "redesign" strategy for changing the response of the system. This is done by using a simple pathway whose behavior can provide both positive and negative chemotaxis (moving up or down the attractant concentration gradient). This "design of dynamics" was completely model-based, with no supporting experimental data. It was not clear whether the differential equations that were used to model the redesigned pathway are realizable.

Overall, the paper uses a correct application of tools from control theory (the term cybernetics is used in the paper) to explore the dynamics of a model for chemotaxis. Based on this model, a design approach is proposed that can be used to modify the dynamics of the simulation model.

In addition to the general comments above, there are several specific areas that should be improved:

• In describing the frequency response (Fig 2), it would be useful to be more clear about whether the response is the open loop response of the "chemotaxis pathway" (from Fig 1) or the closed loop response. I assume the open loop response

• Figure 2 and its relationship with equations [1] and [2] (which are supposed to be based on this, I think) are a bit confusing. In the left Bode plot in Fig 2, the phase at low frequency is zero, whereas in equation [1] there is a minus sign in the transfer function (so low frequency phase would be -180). The figure also shows a second order low pass filter, whereas equation [1] is a first order low pass filter. In the right Bode plot in Fig 2, on the other hand, the low frequency phase *agrees* with that of equation [2]. It seems like one should be more consistent in the choice of signs. The bandpass filter show in Fig 2 is also one order higher than the equation (since there is a 270 degree change in phase instead of 180).

• In the section on pathway design, the authors says that the pathway is "proved" to be the smallest structure that works as a band-pass filter. This is a bit overstated since it is not clear what "smallest" means and the statement is made in the context of a specific choice of mechanism (a simple model of phosphorylation).

• The discussion of nonlinear effects leading to "robustness" of perfect adaptation could be clarified: what does "robust" mean in this context. Since the authors are using tools form control theory, where robustness has a formal definition, a more rigorous statement might be justified.

• After equation [10], the variable y_ti should probably be y_it to match equation [10].