BELIV2012

BELIV 2012: Beyond Time and Errors - Novel Evaluation Methods for Visualization

A workshop at the VisWeek 2012 Conference on October 14(/15), 2012 in Seattle, WA, USA.

BELIV 2012 NOTES

Day 1 Wrap-up - Click this link to see our notes from Day 1, to be used for discussion on Day 2

DESCRIPTION

Visualization has recently gained much relevance for its ability to cope with complex data analysis tasks and communication. While the overall use visualizations is accelerating, the growth of techniques for the evaluation of these systems has been slow. To understand these complex behaviors, evaluation efforts should be targeted at the component level, the system level, and the work environment level. The commonly used evaluation metrics such as task time completion and number of errors appear insufficient to quantify the quality of a visualization system; thus the name of the workshop: "beyond time and errors ...".

The BELIV workshop series is a bi-annual event focusing on the challenges of evaluation in visualization. While it has been focused on information visualization in the past, BELIV 2012 aims at gathering researchers in all fields of visualization to continue the exploration of novel evaluation methods, and to structure the knowledge on evaluation in visualization around a schema, where researchers can easily identify unsolved problems and research gaps.

This is the fourth edition of the BELIV workshop series and this edition will be a will be a two-day workshop.
It will be held as an open full-day workshop with paper presentations and discussions on October 14th and continue on the 15th with focused discussion for attendees interested to follow-up on topics and challenges raised during the first day. The second day is meant as an extended opportunity for participants and organizers to re-engage most active discussion topics of the first day. The second day of the workshop will take place in the beautiful Seattle Public Library where we are kindly sponsored by Google and Microsoft.

AGENDA AND PAPERS

Morning Sessions

1. Evaluation at Design (InfoVis): How do we learn from users at the design stage to correct mistakes before building a full prototype?

• Experiences in Involving Analysts in Visualisation Design (Position Paper)

Aidan Slingsby, Jason Dykes

• An Integrated Approach for Evaluating the Visualization of Intensional and Extensional Levels of Ontologies (Research Paper)

Isabel Silva, Carla Dal Sasso Freitas, Giuseppe Santucci

2. Evaluation at Design (SciVis): How do we learn from users at the design stage to correct mistakes before building a full prototype?

• Which Visualizations Work, for What Purpose, for Which Audiences? Visualization of Terrestrial and Aquatic Systems (VISTAS) Project – A Preliminary Report (Position Paper)

Judith Cushing, Kirsten Winters, Denise Lach, Michael Bailey, Susan Stafford, Evan Hayduk, Jerilyn Walley, Christoph Thomas

• Toward Mixed Method Evaluations of Scientific Visualizations and Design Process as an Evaluation Tool (Position Paper)

Bret Jackson, Dane Coffey, Lauren Thorson, David Schroeder, Arin Ellingson, David Nuckley, Daniel Keefe

3. Cognition and evaluation (new metrics / measures): How can we measure user cognition?

• Evaluating Scientific Visualization Using Cognitive Measures (Position Paper)

Erik Anderson

• The ICD3 Model: Individual Cognitive Differences in Three Dimensions (Position Paper)

Evan Peck, Beste Yuksel, Lane Harrison, Alvitta Ottley, Remco Chang

• Interaction Junk: User Interaction-Based Evaluation of Visual Analytic Systems (Position Paper)

Alex Endert, Chris North

4. Evaluating visualizations: How can we measure visualization?

• Spatial Autocorrelation-Based Information Visualization Evaluation (Research Paper)

Joseph Cottam, Andrew Lumsdaine

• The Importance of Tracing Data Through the Visualization Pipeline (Position Paper)

Aritra Dasgupta, Robert Kosara

Afternoon Sessions:

5. Why evaluate?: What are the goals and motivations of evaluations? How should these be conveyed in reporting evaluation?

• Stop The Evaluation Arms Race! A Call to Evaluate Visualization Evaluation (Position Paper)

Michael Gleicher

• The Four-Level Nested Model Revisited: Blocks and Guidelines (Research Paper)

Miriah Meyer, Michael Sedlmair, Tamara Munzner

6. New evaluation framework: What can we learn from patterns and templates and apply to visualization evaluation?

• Patterns for Visualization Evaluation (Research Paper)

Niklas Elmqvist, Ji Soo Yi

• A Reflection on Seven Years of the VAST Challenge (Research Paper)

Jean Scholtz, Mark Whiting, Catherine Plaisant, Georges Grinstein

7. Novel methods

• Reading, Sorting, Marking, Shuffling: Mental Model Formation through Information Foraging (Position Paper)

Laura McNamara, Nancy Orlando-Gay

• Evaluating Analytic Performance (Position Paper)

Brian Fisher, Linda Kaastra, Richard Arias-Hernández

8. Improving existing methods

• How to Filter out Random Clickers in a Crowdsourcing-Based Study? (Research Paper)

Sung-Hee Kim, Hyokun Yun, Ji Soo Yi

• Questionnaires for Evaluation in Information Visualization (Position Paper)

Camilla Forsell, Matthew Cooper

• Methodologies for the Analysis of Usage Patterns in Information Visualization (Position Paper)

Margit Pohl

PLANNED ACTIVITIES

Taking feedback from our past participants, BELIV 2012 will continue to emphasize interactions between attendees so that participants can produce tangible outcomes during the workshop that will have impact on the visualization community at large. BELIV 2012 will be a full-day workshop with eight 45 minute sessions. Our goal is to combine discussions and paper presentations into one event in a panel-based format such that we can accommodate a much larger audience than in the past. Presentations will be essential to provide context for our audience, especially given that the workshop is open to all participants of the VisWeek conferences, and we expect a much larger number of attendees than in previous years.

To avoid turning our workshop into a mini-conference, we will tightly integrate the presentations with discussions. We will assign accepted papers (both research and position) to eight groups, one for each workshop session. In each group, paper authors will together create a group presentation for the session prior to the workshop. Rather than providing overviews for each specific paper, the main goal of the overview presentation is to build the basis for panel-based discussions on the session topic afterwards. The presentation will, therefore, be a summary of the state-of-the-art of the topic, illustrated by materials from the accepted papers. Each group will be given 15-20 minutes for the presentation. The remaining hour will be devoted to a panel discussion where authors of the papers will serve as the panel and discuss the topic with the audience. Workshop organizers will take notes during the discussion to tease out potential topics for further and more in-depth discussions after the workshop, ideally conducted in smaller groups.

At the end of the first day of the workshop, the organizers will post topics collected during the panel discussions for workshop attendees to vote and to comment. As the VisWeek open workshop format makes smaller in-depth discussions with groups more challenging, we will invite participation to follow-up discussions on the day after the official workshop on an off-site venue arranged by the workshop organizers. This second day will be complimentary to the first day, and its existence will not offer a diminished experience for those who choose to only attend the first day. Rather, it is an extended opportunity for participants and organizers to re-engage most active discussion topics of the first day. In addition to the selected topics on the second day, the organizers will lead a discussion on broader issues of the workshop such as 1) how to impact the community (e.g., publishing a research agenda in a journal); 2) how to set up an infrastructure to maintain the discussion alive beyond after the workshop; 3) how and where to organize BELIV 2014.

DATES

Deadline for submissions: September 1, 2012, new extended deadline: September 5, 2012
Notification of acceptance: September 24, 2012
Camera ready papers due: October 1, 2012
Workshop: October 14/(15), 2012

HOW TO PARTICIPATE

All registered attendees of VisWeek will be able to attend the workshop. In order to present a paper and participate as a panelist in the discussions, it is necessary to have a paper accepted. The panel discussion itself is open to everybody.

PAPER TYPES

We accept 2 types of submissions—research papers and position papers:

• Research papers present new work and unpublished results on the topic areas of the workshop. Research papers will be selected according to their novelty, quality and relevance.

• Position papers are problem discussions or statements describing the author's relevant experience and ideas that can contribute to the debate during the workshop. Position papers will be selected according to their importance and relevance of their issues and the quality of their discussion.

Both types of papers can be up to eight pages long but the length of a submission needs to correspond to its contribution. All papers will be peer-reviewed by members of the program committee as well as the organizers. We are currently working on making sure that we will be able to publish all accepted papers in the ACM digital library including the assignment of DOIs to the papers. We also plan to invite a selected subset of the papers to contribute to a special issue in a major visualization journal venue. For past BELIV workshops and papers see these links:

• BELIV 2006: published in the ACM DL http://doi.acm.org/10.1145/1168149
  
• BELIV 2008: published in the ACM DL http://doi.acm.org/10.1145/1377966
  
• BELIV 2010: published in the ACM DL http://doi.acm.org/10.1145/2110192
  

SUBMISSION

To submit a paper create an account and submit the paper to the submission system at: https://cmt.research.microsoft.com/BELIV2012/. Please clarify whether you are submitting a position or research paper.

FORMAT

All the submissions should be formatted in the ACM style. Suitable templates, in LaTeX and Word, can be downloaded from: http://www.acm.org/sigs/pubs/proceed/template.html. Submission, however, must be made in PDF format.

TOPICS OF INTEREST

Topics include, but are not limited to:

• Evaluation in the visualization development lifecycle,
• Utility characterization,
• Evaluation metrics,
• Insight characterization,
• Synthetic data sets and benchmarks,
• Taxonomy of tasks,
• Computational evaluation,
• Benchmark development and repositories,
• Methods for longitudinal studies and adoption,
• Evaluation of early prototypes, and
• Evaluation heuristics and guidelines.

ORGANIZERS

Enrico Bertini: University of Konstanz
Adam Perer: IBM Research
Heidi Lam: Google Inc.
Petra Isenberg: INRIA
Tobias Isenberg: INRIA, CNRS, University of Groningen

ADVISORS

Catherine Plaisant: University of Maryland
Giuseppe Santucci: Sapienza Università di Roma, Italy