EMNLP 2018 Review Form Commentary

Introduction

This year, EMNLP is using a new review form, which we hope will prompt reviewers to give useful and constructive feedback to authors, as well as help the area chairs and program chairs make informed decisions about which papers to accept.

This commentary on the review form is intended to provide guidance for reviewers about what criteria we should be using. We also hope that it will help authors as they put they finishing touches on their submissions.

The review form is divided into three parts:

  1. In-Depth Review
  2. Questions and Feedback for the Author(s)
  3. Confidential Information

Part 1 is visible to both the authors and to the program committee. Part 2 is also visible to both the authors and committee, but is primarily intended for authors. Part 3 is visible to the program committee only.

In-Depth Review

Your answers to these questions will be shared with both the authors and the committee. They will help the committee to decide the outcome of the paper, as well as assure the authors that you have read and understood their paper.

What is this paper about, and what contributions does it make?
Please describe what problem or question this paper addresses, and the main contributions that it makes towards a solution or answer.

The first question asks you for a summary of the paper and its contributions to the field. We understand contributions very broadly to include anything that furthers knowledge of natural language processing or its methodologies or its applications. A contribution could be: a new method, model, task, software, or dataset; analysis or a negative result about an existing method, model, task, software or dataset; methodological, conceptual, experimental, theoretical, or analytical advancement; and this list is not exhaustive. Remember that short papers can be opinion pieces or interesting application notes, or describe work in progress or negative results.

What strengths does this paper have?
Please describe the main strengths you see in the paper or the work it describes, regardless of whether you recommend this paper be accepted or not.
What weaknesses does this paper have?
Please describe any weaknesses you see in the paper or the work it describes, regardless of whether you recommend this paper be accepted or not.

The second and third questions ask you about the strengths and weaknesses of this submission. They are intended to help you and us to look beyond an ostensibly invincible strength or fatal weakness to gain a fuller view of both positive and negative aspects that we can weigh against each other. Please consider all aspects of the paper. Is it appropriate for EMNLP? Is it interesting, innovative, insightful? How original or creative is it? How impactful will it be? Is it well written? Is it clear what was done, and how it relates to or differs from existing work? How feasible would it be to replicate? Again, this is not an exhaustive list of potentially positive or negative aspects.

Overall recommendation
Do you think this paper should be accepted to EMNLP 2018?
In making your overall recommendation, please take into account all of the paper's strengths and weaknesses. Please rank short papers relative to other short papers, and long papers relative to other long papers. Acceptable short submissions include: small, focused contributions; works in progress; negative results and opinion pieces; and interesting application notes.
  • 5 = Exciting: I would fight for this paper to be accepted.
  • 4 = Strong: I would like to see it accepted.
  • 3 = Borderline: It has some merits but also some serious problems. I'm ambivalent about this one.
  • 2 = Mediocre: I would rather not see it in the conference.
  • 1 = Poor: I would fight to have it rejected.

This year, we are using the 5-point scale that most people in the ACL community are accustomed to, with scores greater than 3 indicating a recommendation to accept and scores less than 3 indicating a recommendation to reject. We are also including half-point scores, in the hopes that you can use 2.5 and 3.5 when you are ambivalent but lean slightly towards rejecting or accepting, and that you can use 1.5 and 4.5 when you feel strongly but don’t want to go to the extreme.

Although your review will not be reduced to a single numerical score when we decide which papers to accept, please make sure that the scores you assign to the papers you review reflect how you would rank them. Remember that we want a conference full of interesting, diverse, timely and original work. On the other hand, remember that no paper is perfect, and that authors have about 3 weeks to address reviewer comments before the camera-ready deadline.

Questions and Feedback for the Author(s)

This part will be shared with both the authors and the committee.

Questions for the Author(s)
Please write any questions you have for the author(s) that you would like answers for in the author response (which you should take into account in your final review).

Please remember that authors are not allowed to present new results in their response, so please don’t ask for new experiments here.

Missing References
Please list any references that should be included in the bibliography or need to be discussed in more depth.
Presentation Improvements
If there is anything in the paper that you found difficult to follow, please suggest how it could be better organized, motivated, or explained.
Typos, Grammar, and Style
Please list any typographical or grammatical errors, as well as any stylistic issues that should be improved.

The feedback section is intended to help the authors improve their submission (whether for presentation at this conference or for submission to another venue). Discussion of the paper’s strengths or weaknesses belongs in the first section (“In-Depth Review”). If the paper is accepted, the comments should guide the authors in making revisions for a final manuscript. If the paper is rejected, your comments may still help the authors improve their submission for another venue. Hence, the more detailed and constructive you make your comments, the more useful your review will be - both for the committee and for the authors.

Confidential Information

Your answers to these questions will be shared with the committee only, not the authors.

Recommendation for Oral Presentation
We have fewer slots for oral presentations (talks) than for posters, and want to make sure that the most appropriate papers get selected for talks. Note that the published proceedings will make no distinction between papers presented orally and those presented as posters.

Would this paper make for a good talk (rather than a poster)?

Although there is expected to be greater competition for oral presentations than poster presentations, it is not necessarily the case that papers selected for oral presentations are better than those selected for poster presentations. For example, a good oral presentation may be of interest to a broad audience or tell an interesting story, whereas a good poster may be more specialized or involve complex ideas that require a longer, deeper conversation.

Recommendation for Best Paper Award
There will be separate Best Paper Awards for long and for short papers.
Recommendation for Best Data Set or Resource Paper Award
The data set or resource must be publicly available in order for the paper to be eligible. It does not need to be submitted with the paper.
Justification for your recommendations
Please describe briefly why you think this paper should receive an award.

If you answer “yes” to either of these questions, the paper will be considered for the award. Because a “yes” answer does not in itself constitute a nomination, we are keeping the answers to these questions hidden from authors.

Reviewer Confidence
How confident are you in your assessment of this paper?
  • 5 = Positive that my evaluation is correct. I read the paper very carefully and I am very familiar with related work.
  • 4 = Quite sure. I tried to check the important points carefully. It's unlikely, though conceivable, that I missed something that should affect my ratings.
  • 3 = Pretty sure, but there's a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper's details, e.g., the math, experimental design, or novelty.
  • 2 = Willing to defend my evaluation, but it is fairly likely that I missed some details, didn't understand some central points, or can't be sure about the novelty of the work.
  • 1 = Not my area, or paper was hard for me to understand. My evaluation is just an educated guess.

It’s helpful to the committee to know your level of confidence in your review. Because we keep this information hidden from authors, please don’t be reluctant to select 1 or 5 if your confidence truly is very low or high.

Confidential Comments to the Area Chairs/PC chairs
Is there anything you want to say solely to the committee? For example, a very strong (negative) opinion on the paper, which might offend the authors in some way, or something which would expose your identity to the authors.