Reviewer Guidelines

Guidelines for the Review Process

Overall Process

Each submitted paper will be reviewed by at least three experts in the field. Submissions will be judged on the following criteria.

• Suitability: Does the contribution fit to the scope of QoMEX and in particular QoE, UX, immersive experiences, as well as to the list of topics of interest?

• Originality: What new ideas, approaches and/or results are introduced?

• Significance/contribution: Is the paper relevance for QoE and UX research? What is the benefit that others can gain from the contribution?

• Validity of the work: How confidently can researchers and practitioners use the results? Is the paper technically/methodologically sound? E.g. measurement setup, design of studies, conduction of simulation studies, statistical analysis, system modelling

• Related work: Is state-of-the-art relevant to the described work sufficiently addressed? Does the paper refer to peer-reviewed publications?

• Thoroughness: Is the research carried out in a thorough way? Are the discussions and conclusions detailed enough?

• Replicability: Are the results and conclusions clearly described and reproducible? Are public data sets or open source code available?

• Presentation clarity: Is the structure of the text and the presentation of the results clear?

Submitted papers of the general and special session tracks will undergo a double-blind review process. Only the dataset & tools track will follow a single-blind review process as authors need to include a link to the dataset or tool which will be evaluated by the reviewers.

Each paper is assigned a Meta-Reviewer who is responsible for coordinating the reviews of the paper and helping the TPC chairs in the decision process . The Meta-Reviewer is selected by the TPC chairs, and part of the program committee. The responsibility of the Meta-Reviewer, assigned 3-4 full papers, is as follows:

• Assign one extra reviewer per paper, in addition to the ones assigned by the TPC chairs.

• Double check the reviews, ensuring high-quality standards.

• Write a meta-review, summarizing the strengths and weaknesses; inform the authors if a rebuttal is expected or not, and clearly indicate the topics that such a rebuttal should cover.

• After the rebuttal, encourage discussion between the reviewers, update the meta-review if needed and make a recommendation to the TPC chairs about the paper.

• Participate in the (hybrid) TPC meeting on April 18th- 19th 2023 via Teams or in person in Ghent, Belgium (IMEC-Ghent University).

For short and full papers, there will be a paper bidding phase in order to ensure a good match between an individual reviewer’s expertise/interests and her/his review assignments. The online TPC meeting for full papers will take place on April 18th-19th, 2023, and is organized by the TPC Chairs. Details will be provided later. All TPC Members are invited to participate in the TPC meeting.

For short papers, an online TPC meeting will be hosted on May 16, 2023. All TPC Members are invited to participate in that TPC meeting.

The TPC members are invited by the TPC chairs to cover all topics as mentioned in the Call for Papers. In particular, the domains of QoE and UX are reflected in the technical program committee.

For TPC Members and Reviewers

Important information related to the paper assignment and review process:

• Reviews are assigned to the reviewers, manually, taking into account claimed papers, topic preferences and conflicts of interest.

• Submissions will be judged on suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity.

• In the TPC meeting and during the discussion phase, a review with a high score but with a short unsubstantiated review will be discounted.

High quality reviews are of utmost importance to QoMEX 2023 and critical for the advancement of the field. For this reason, please keep the following review guidelines in mind:

• Avoid being overly negative: try to be critical yet constructive in your reviews, as any research paper has its advantages and drawbacks.

• Please write a review with the quality that you would expect for your own papers. Do not submit very short, unsubstantiated reviews.

• Keep a fair and open-minded approach, favoring selection of breakthrough papers that can open new areas and new research directions. 

• Be particularly open-minded when reviewing Special Session papers as they might address different scientific communities.

• If any of the papers are reviewed with the help of graduate students, we expect a TPC member to provide sufficient guidance and oversight, and personally submit the reviews to EDAS.

• Use the “comments to the TPC chairs” field to share confidential remarks or concerns you may have regarding a submission.

• Finally, we recognize the efforts and time that TPC members and reviewers contribute to reviewing assigned papers. A best reviewer recognition award will be handed out during the conference.

Full papers

Submitted full papers of the general and special session tracks will undergo a double-blind review process. Submitted full papers must not exceed six (6) pages, including figures, tables and references. Please see the paper submission instructions.

Reviewers should consider the following in their evaluation: suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. Please note that we also invite to submit position papers as full papers which describe a novel or bold idea; in that case experimental results may not be existing, but the paper should be nevertheless scientifically valid and convincing, and contribute to QoMEX overall.

• Full Paper Registration: Feb 14, 2023

• Paper Bidding: Feb 17 –  Feb 20, 2023

• Full paper Submission: Feb 19, 2023

• Review phase: Feb 19 - March 19, 2023

• Rebuttal phase: March 29 - April 6, 2023

• Updated meta-reviews (post-rebuttal) due: April 16, 2023

• Reviewer discussion phase: April 7- April 16, 2023

• TPC Meeting: April 18 - April 19, 2023 (hybrid)

• Author notification: April 21, 2023

• Camera-ready: TBD

Short papers

Submitted short papers of the general and special session tracks will undergo a double-blind review process. Submitted short papers must not exceed three (3) pages, including figures, tables and references. Short paper submissions must describe original and unpublished work. Please see the paper submission instructions.

Reviewers should consider the following in their evaluation: suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. While a short paper is not a shortened long paper, the characteristics of short papers may include one or more of the following:

• Small, but focused contribution

• Work-in-progress or late-breaking results

• A position paper or contribution that is pushing/challenging existing boundaries or paradigms

• A counter-example to an existing technique that helps to understand its limitations

• Extensions or evaluations of existing methods or explorations of new methods

• An interesting application nugget

The timeline for the short papers is as follows.

• Short paper submission: April 28, 2023

• Paper Bidding: April 28 - April 30, 2023

• Review phase: April 30 - May 12, 2023

• Reviewer discussion phase: May 12 - May 14, 2023

• Short paper TPC meeting:  May 16, 2023

• Short paper notification: May 17, 2023

• Short paper camera-ready: TBD

Dataset & Tools track

Full and short paper submissions to the dataset & tools track are single-blind. The authors need to provide a link to the dataset or tool which will be elaborated by the reviewers as part of the submission. The data and tool must be made available to the reviewers upon submission and will have to be published upon acceptance.

Reviewers should consider the specific features of papers submitted to the dataset & tools track and evaluate the submissions as follows:

• The dataset or tool must be available online (e.g. on Github, zenodo, etc.).

• Dataset papers must contain all relevant information and describe its content and origin, such that readers can fully understand the experimental environment and execution. Additional information may be provided online in the description of the dataset.

  • its content and origin;

  • the methodology and tools used to obtain it;

  • the schema, structure, or layout to store it;

  • usage scenarios or open analysis challenges for it;

  • relationships to other existing datasets;

  • its current limitations.

• Tool papers must present either a new tool, a new tool component, or novel extensions to an existing tool. Tool papers should provide a short description of the theoretical foundations, after which emphasis should be on the design and implementation concerns (incl. software architecture and core data structures), closed by the description of experience with realistic case studies.

• Dataset and Tool papers must clearly describe the novelty and significance and how this dataset/tool advances QoE or UX research.

References

Papers submitted to QoMEX 2023 should refer to peer-reviewed publications. References to non-peer-reviewed publications (including public repositories such as arXiv, Preprints, software, and personal communications) should only be made if there is no peer-reviewed publication available, should be kept to a minimum, and should appear as footnotes in the text (i.e., not in the references list). Reviewers should comment on the suitability of references.