From BELIV 2010

Abstract: Research in information visualization (InfoViz) has developed considerably during the last 25 years. In particular, the field is now informed by a substantial and growing literature on evaluations of visualizations. To keep advancing InfoViz, I believe we need to address two limitations of our evaluations. On the one hand, few empirical studies are motivated by theory or are comparing equally plausible hypotheses. Mostly, the InfoViz literature proposes radical innovations (in the terms of William Newman) and does little to develop and test concepts. On the other hand, many of the practical, low-level decisions in InfoViz evaluations are problematic. Like most HCI researchers, we evaluate our own interfaces, use mostly simple outcome measures, rarely study the process of interaction, and select tasks somewhat randomly. This talk will outline the conceptual and practical challenges of evaluation and begin a discussion of how to overcome them.

Biography: Kasper Hornbæk received his M.Sc. and Ph.D. in Computer Science from the University of Copenhagen, in 1998 and 2002, respectively. Since 2009 he has been a professor with special duties in Human-centered Computing at University of Copenhagen. His core research interests are human-computer interaction, usability research, search user interfaces, and information visualization; detours include eye tracking, cultural usability, and reality-based interfaces. Kasper serves on the editorial boards of Journal of Usability Studies, Interacting with Computers, and International Journal of Human-Computer Studies (IJHCS). He has published at CHI, UIST, ACM Transactions on Computer-Human Interaction, and Human-Computer Interaction, and won IJHCS’s most cited paper award 2006-2008.

Presentation/Notes

Motivation and Goals

The purpose of information visualization is to provide users with accurate visual representations of data and natural interaction tools to support discovery and sense making. These activities are often exploratory in nature and can take place over days, weeks or months and rarely follow a predefined or linear workflow. While the overall use of information visualizations is accelerating, the growth of techniques for the evaluation of these systems has been slow. To understand these complex behaviors, evaluation efforts should be targeted at the component level, the system level, and the work environment level. The commonly used evaluation metrics such as task time completion and number of errors appear insufficient to quantify the quality of an information visualization system; thus the name of the workshop: “beyond time and errors …”.

BELIV 2010 aims at gathering researchers in the field to continue the exploration of novel evaluation methods, and to structure the knowledge on evaluation in information visualization around a schema, where researchers can easily identify unsolved problems and research gaps.

This is the third edition of the BELIV workshop series. Based on feedback from past workshop participants, BELIV 2010 will be a 2-day workshop to provide a more interactive environment where participants can produce a research agenda to be published online.

BELIV 2008 was held in Florence, Italy. Proceedings in ACM DL.
BELIV 2006 was held in Venice, Italy. Proceedings in ACM DL.

Interested? Read How_To_Participate - Note: Submissions Closed.

Schedule

Day 1 - Saturday, April 10th 2010. 9AM - 5PM.

A day of presentations: get an overview of evaluations in Information Visualization from multiple perspectives and prepare for Sunday's working session

09:00 – 09:30	Welcome to BELIV 2010!
09:30 – 10:30	Presentations: #New Metrics I
10:30 – 11:00	Coffee Break
11:00 – 12:30	Presentations: #New Metrics II; #Insight evaluations; #Tasks / Data
12:30 – 14:00	Lunch
14:00 – 15:30	Presentations: #Physiological measurements; #Methods
15:30 – 16:00	Coffee Break
16:00 – 17:00	Presentations: #Mass evaluations; #Evaluation lessons
17:00 – 18:00	Break
18:00 – 20:00	Workshop Dinner (optional)

Day 2 - Sunday, April 11th 2010. 9AM - 5PM.

A day of discussing future research themes in visualization evaluations.

09:00 – 9:10	Introduction to day 2
09:10 – 9:55	Keynote (Kasper Hornbaek)
09:55 – 10:30	Generation of discussion topics and groups
10:30 – 11:00	Coffee Break
11:00 – 11:30	Working session 1
11:30 – 12:00	Working session 2
12:00 – 12:30	Working session 3
12:30 – 14:00	Lunch
14:00 – 14:30	Working session 4
14:30 – 15:30	Group presentations (10 presentation + 5 discussion)
15:30 – 16:00	Coffee Break
16:00 – 17:00	Wrap up, feedback, awards, conclude

Online resources

Post your questions, impressions, feedback throughout the workshop in Google Moderator:

http://www.google.com/moderator/#16/e=52cd

The material collected in GM will be used to generate the topics to discuss on day 2.

Organization

Each research talk is allotted 10 minutes and each position talk, 5 minutes. 5 minutes are reserved for discussion for both types of presentations. We realize this is a short amount of time but please keep in mind that the main purpose of the presentations is to introduce interesting topics to discuss on day 2.

On day 2, we break up into teams, each tasked with fleshing out a particular theme of evaluation research. The goal of each team is to create a set of research agenda for that particular theme. The team presents its topic at the end of day 2. We suggest organizing discussion and presentation around (at least) the following elements: use cases, challenges/open issues, interesting directions for research, proposed initial solutions.

Preliminary Agenda

Day 1 is driven by presentations and discussions. We plan to give 10 min to regular papers and 5 min to position papers. A larger portion of time will be allocated for discussions than for presentation. One of the organizers will be explicitly in charge of taking notes, which will be made available to workshop participants during the day. Organizers will also set up a large poster board with post-it notes for participants to add their comments and ideas. At the end of day 1, organizers will compile a list of discussion topics to be used on day 2 to guide the discussions.

Day 2 will start with a brief summary of the topics collected on day 1. Participants and organizers will select eight of these topics for discussion in the course of the day, with four topics in the morning and four in the afternoon. Participants will be free to form groups based on the topic of their interest. Each group will be led by two participants, who will also give a presentation to summarize their discussion results at the end of day 2. At the end of the day, the organizers will lead a discussion on broader issues like: how to impact the community; how to maintain the discussion alive; how to better organize BELIV'12.

After the workshop, the workshop website will host all the research papers (published in the ACM DL), the position papers, and the presentations created by the working groups. We plan to create a mailing list with the workshop participants so that the discussion can continue after the workshop. Finally, we plan to build a research agenda, based on the presentations prepared by the working groups on day 2, and to publish it in a leading journal.

Organizers

Chairs

Enrico Bertini
University of Konstanz
Konstanz, Germany

Heidi Lam
Google Inc.
Mountain View, CA, USA

Adam Perer
IBM Haifa Research Lab
Mount Carmel, Haifa, Israel

Advisors

Catherine Plaisant (Univ. of Maryland, USA)
Giuseppe Santucci (Sapienza Università di Roma, Italy)

Program Committee

Remco Chang (UNC Charlotte, USA)
Alan Dix (Lancaster University, UK)
Carla Dal Sasso Freitas (Instituto de Informatica UFRGS, Brazil)
Jean-Daniel Fekete (INRIA, France)
Georges Grinstein (UMass Lowell, USA)
Jeffrey Heer (Stanford, USA)
Nathalie Henry (Microsoft Research, USA)
Petra Isenberg (Univ. of Calgary, Canada)
Silvia Miksch (Vienna Univ. of Technology, Austria)
Tamara Munzner (Univ. of British Columbia, Canada)
Chris North (Virginia Tech, USA)
George Robertson (Microsoft Research, USA)
Jean Scholtz (Pacific Northwest National Lab, USA)
John Stasko (Georgia Tech, USA)
Jarke Van Wijk (TU Eindhoven, Netherlands)

Main Page

From BELIV 2010

BELIV'10: BEyond time and errors: novel evaLuation methods for Information Visualization.

Latest Posts from the BELIV Blog

Papers and Presentations

New Metrics I

New Metrics II

Insight evaluations

Tasks / Data

Physiological measurements

Methods

Mass evaluations

Evaluation lessons

The BELIV'10 Keynote

Motivation and Goals

Schedule

Day 1 - Saturday, April 10th 2010. 9AM - 5PM.

Day 2 - Sunday, April 11th 2010. 9AM - 5PM.

Online resources

Organization

Preliminary Agenda

Organizers

Chairs

Advisors

Program Committee

Views

Personal tools

Navigation

Search

Toolbox