The BELIV Workshop
BEyond time and errors: novel evaLuation methods for Visualization
Visualization Evaluation
A Special Issue of Information Visualization,
Guest Editors: Enrico Bertini, Petra Isenberg, Tobias Isenberg, Heidi Lam, Adam Perer.
Visualization has recently gained much relevance for its ability to cope with complex data analysis and communication. Thanks to its ability to convey complex concepts in an efficient manner it has been adopted in many contexts; from scientific laboratories to newspapers.
While the overall use of visualization is accelerating, the growth of techniques for the evaluation of these systems has been slow. How do we measure the quality of a visualization? How do we decide on competing designs? How do we know whether a visualization meets its goal? These are only some of the questions researchers and practitioners are confronted with when designing visualization systems.
To understand how, when and why visualization works, evaluation efforts should be targeted at the component level, the system level, and the work environment level. The commonly used evaluation metrics such as task time completion and number of errors, when used in isolation, appear to be insufficient to quantify the complex quality and usage of a visualization system, therefore new metrics and methods to better evaluate visualizations are needed.
This special issue calls for innovative ideas about how to evaluate visualization and reflections about its state of the art. We call for papers dealing with evaluation in the fields of scientific visualization, information visualization, and visual analytics. We explicitly discourage submissions describing exclusively the process and outcome of a given evaluation unless the methodology itself is innovative.
Topics include, but are not limited to:
· Evaluation in the visualization development lifecycle
· Utility characterization
· Evaluation metrics
· Insight characterization
· Synthetic data sets and benchmarks
· Taxonomy of tasks
· Computational evaluation
· Benchmark development and repositories
· Methods for longitudinal studies and adoption
· Evaluation of early prototypes
· Evaluation heuristics and guidelines
· Evaluation of storytelling visualization
· Evaluation of visualization as art
Paper Submission
Submissions due: Feb 8, 2013
Acceptance notices: Apr 15, 2013
Final revisions due: May 13, 2013
Publication: late 2013
Inquiries should be made to the guest editors by sending an email to chairs AT beliv.org. Electronic submissions of manuscripts in PDF should be made using the online submission system and the papers should be formatted according to the journal standards. For details on the submission process, please visit the Journal’s instruction website at: http://ivi.sagepub.com/
Click here to read our notes from Day 1, which we will use as a discussion point for tomorrow at the Seattle Public Library:
WHAT IS NEW (since the 1st cfp):
———————————-
* Accepted papers will appear in the ACM digital library
* There will be a special issue of the IVS Journal for invited papers
* Position papers can now be between 1-8 pages long
* We have a great PC (see below)
* Format and submissions details added
* We will be sponsored by Google and Microsoft (thanks!)
* The paper deadline is just two weeks away!
———————————-
*** 2nd Call for Papers ***
BELIV 2012: Beyond Time and Errors – Novel Evaluation Methods for Visualization
A Workshop at the VisWeek 2012 Conference
*** October 14(/15), 2012 – Seattle, WA, USA ***
DESCRIPTION
Visualization has recently gained much relevance for its ability to cope with complex data analysis tasks and communication. While the overall use of visualizations is accelerating, the growth of techniques for the evaluation of these systems has been slow. To understand these complex behaviors, evaluation efforts should be targeted at the component level, the system level, and the work environment level. The commonly used evaluation metrics such as task time completion and number of errors appear often insufficient to quantify the quality of a visualization system; thus the name of the workshop: “beyond time and errors …”.
The BELIV workshop series is a bi-annual event focusing on the challenges of evaluation in visualization. While it has been focused on information visualization in the past, BELIV 2012 aims at gathering researchers in all fields of visualization to continue the exploration of novel evaluation methods, and to structure the knowledge on evaluation in visualization around a schema, where researchers can easily identify unsolved problems and research gaps.
This is the fourth edition of the BELIV workshop series. It will be held as an open full-day workshop with paper presentations and discussions on October 14th and continue on the 15th with focused discussion for attendees interested to follow-up on topics and challenges raised during the first day. The second day is meant as an extended opportunity for participants and organizers to re-engage most active discussion topics of the first day. The second day of the workshop will take place in the Seattle Public Library where we are kindly sponsored by Google and Microsoft.
DATES
Deadline for submissions: September 5, 2012
Notification of acceptance: September 24, 2012
Camera ready papers due: October 1, 2012
Workshop: October 14/(15), 2012
HOW TO PARTICIPATE
All registered attendees of VisWeek will be able to attend the workshop. In order to present a paper and participate as a panelist in the discussions, it is necessary to have a paper accepted. The panel discussion itself is open to everybody.
PAPER TYPES
We accept 2 types of submissions—research papers and position papers:
* Research papers present new work and unpublished results on the topic areas of the workshop. Research papers will be selected according to their novelty, quality and relevance.
* Position papers are problem discussions or statements describing the author’s relevant experience and ideas that can contribute to the debate during the workshop. position papers will be selected according to their importance and relevance of their issues and the quality of their discussion.
Both types of papers can be up to eight pages long but the length of a submission needs to correspond to its contribution. All papers will be peer-reviewed by members of the program committee as well as the organizers. We are currently working on making sure that we will be able to publish all accepted papers in the ACM digital library including the assignment of DOIs to the papers. We also plan to invite a selected subset of the papers to contribute to a special issue of the Information Visualization journal. For past BELIV workshops and papers see these links:
* BELIV 2006: published in the ACM DL http://doi.acm.org/10.1145/1168149
* BELIV 2008: published in the ACM DL http://doi.acm.org/10.1145/1377966
* BELIV 2010: published in the ACM DL http://doi.acm.org/10.1145/2110192
SUBMISSION
To submit a paper create an account and submit the paper to the submission system at: https://cmt.research.microsoft.com/BELIV2012/. Please clarify whether you are submitting a position or research paper.
FORMAT
All the submissions should be formatted in the ACM style. Suitable templates, in LaTeX and Word, can be downloaded from: http://www.acm.org/sigs/pubs/proceed/template.html. Submission, however, must be made in PDF format.
TOPICS OF INTEREST
Topics include, but are not limited to:
* Evaluation in the visualization development lifecycle,
* Utility characterization,
* Evaluation metrics,
* Insight characterization,
* Synthetic data sets and benchmarks,
* Taxonomy of tasks,
* Computational evaluation,
* Benchmark development and repositories,
* Methods for longitudinal studies and adoption,
* Evaluation of early prototypes, and
* Evaluation heuristics and guidelines.
ORGANIZERS
Enrico Bertini : University of Konstanz
Adam Perer : IBM Research
Heidi Lam : Google Inc.
Petra Isenberg : INRIA
Tobias Isenberg : INRIA
PROGRAM COMMITTEE:
Remco Chang (UNC Charlotte, USA)
Carla Dal Sasso Freitas (Instituto de Informatica UFRGS, Brazil)
Jean-Daniel Fekete (INRIA, France)
Jeffrey Heer (Stanford, USA)
Nathalie Henry Riche (Microsoft Research, USA)
Victoria Interrante (University of Minnesota, USA)
Silvia Miksch (Vienna Univ. of Technology, Austria)
Tamara Munzner (Univ. of British Columbia, Canada)
Chris North (Virginia Tech, USA)
Jean Scholtz (Pacific Northwest National Lab, USA)
John Stasko (Georgia Tech, USA)
Daniel F. Keefe (University of Minnesota, USA)
Jarke Van Wijk (TU Eindhoven, Netherlands)
Robert Kosara (UNC Charlotte, USA)
Chris Weaver (University of Oklahoma, USA)
Niklas Elmqvist (Purdue University, USA)
Melanie Tory (University of Victoria, Canada)
Michael Sedlmair (University of British Columbia, Canada)
The 2012 edition of BELIV will happen at VisWeek for the first time. The goal is to broaden BELIV to the SciVis, InfoVis, and VAST communities.
The BELIV 2012 organizing committee consists of:
Enrico Bertini is a researcher at the University of Konstanz, Germany. His main research interest is Visual Analytics, with a specific focus on visualization of large data, evaluation, and visualization for security and business data. @FILWD
Adam Perer is a research scientist at IBM Research’s Healthcare Analytics group in New York, USA. His research focuses on visualization and analytic techniques for making sense of electronic medical records and social networks. He received his Ph.D. in Computer Science from the University of Maryland. @adamperer
Heidi Lam is a software engineer at Google Inc. She earned a Ph.D. in computer science from the University of British Columbia. Her main research interest is to explore the roles information visualization in supporting large data sets analysis.
Petra Isenberg is a research scientist in the Aviz group at INRIA, the French National Research Institute for Computer Science. She holds a PhD degree from the University of Calgary (2009). Her main research interests lie at the intersection of information visualization, CSCW, and interactive surfaces. @dr_pi
Tobias Isenberg is an assistant professor for computer graphics and interactive systems at the University of Groningen, the Netherlands. Since September 2010, he also holds a Digiteo Chair of Excellence in collaboration with the French research institutions CNRS and INRIA. He works on topics in interactive non-photorealistic/illustrative rendering and explores applications in visualization.
The BELIV 2010 research papers are available online in the ACM Digital Library.
Extended versions of the research papers were published in a special issue of the Information Visualization Journal in July 2011.
Research papers
Position papers