BELIV: The Workshop Blog

As the BELIV’10 workshop is nearing, we’ve put all of the papers online at http://www.beliv.org/beliv2010/.  Revised versions of the following papers will be published in the ACM Digital Library in the coming months.

Research papers

• Developing Qualitative Metrics for Visual Analytic Environments. (Jean Scholtz)
• Do Mechanical Turks Dream of Square Pie Charts? (Robert Kosara, Caroline Ziemkiewicz)
• Many Roads Lead to Rome. Mapping Users’ Problem Solving Strategies. (Eva Mayr, Michael Smuc, Hanna Risku)
• A Descriptive Model of Visual Scanning. (Stephane Conversy, Christophe Hurter, Stephane Chatty)
• Exploring Information Visualization – Describing Different Interaction Patterns. (Sylvia Wiltner, Margit Pohl, Silvia Miksch, Markus Rester, Klaus Hinum, Christian Popow, Susanne Ohmann)
• Is Your User Hunting or Gathering Insights? Identifying Insight Drivers Across Domains. (Michael Smuc, Eva Mayr, Hanna Risku)
• Comparing Benchmark Task and Insight Evaluation Methods. (Chris North, Purvi Saraiya)
• Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations. (Michael Sedlmair, Petra Isenberg, Dominikus Baur, Andreas Butz)
• Learning-Based Evaluation of Visual Analytic Systems. (Chang)
• Scanning Between Graph Visualizations: An Eye Tracking Evaluation. (Joseph Goldberg, Jonathan Helfman)
• Towards Information-Theoretic Visualization Evaluation Measure: A Practical example for Bertin’s Matrices. (Innar Liiv)
• Generating a synthetic video dataset. (Mark Whiting)

Position papers

• Visualization Evaluation of the Masses, by the Masses, and for the Masses. (Jeffrey Heer)
• Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study. (Peter Kinnaird, Mario Romero)
• Pragmatic Challenges in the Evaluation of Interactive Visualization Systems. (John Stasko, Youn-ah Kang, Carsten Gorg)
• Look Before You Link: Eye Tracking in Multiple Coordinated View Visualization. (Chris Weaver)
• Mutually Linked Studies – Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation. (Niklas Elmqvist)
• Beyond system logging: human logging for evaluating information visualization. (Nathalie Henry)
• How is a graphic like pumpkin pie? A framework for analysis and critique of visualisations. (Hadley Wickham)
• Proposed Working Memory Measures for Evaluating Information Visualization Tools. (Laura Matzen, Laura McNamara, Kerstan Cole, Alisa Bandlow, Courtney Dornburg, Travis Bauer)
• Comparative Evaluation of Two Interface Tools in Performing Visual Analytics Tasks. (Dong Hyun Jeong, William Ribarsky, Tera Green, Remco Chang)
• Implications of Individual Differences on Evaluating Information Visualization Techniques. (Ji Soo Yi)

We are delighted to announce that Kasper Hornbæk will deliver the keynote at BELIV’10.  Here are his keynote details:

Conceptual and Practical Challenges in InfoViz Evaluations

Abstract: Research in information visualization (InfoViz) has developed considerably during the last 25 years. In particular, the field is now informed by a substantial and growing literature on evaluations of visualizations. To keep advancing InfoViz, I believe we need to address two limitations of our evaluations. On the one hand, few empirical studies are motivated by theory or are comparing equally plausible hypotheses. Mostly, the InfoViz literature proposes radical innovations (in the terms of William Newman) and does little to develop and test concepts. On the other hand, many of the practical, low-level decisions in InfoViz evaluations are problematic. Like most HCI researchers, we evaluate our own interfaces, use mostly simple outcome measures, rarely study the process of interaction, and select tasks somewhat randomly. This talk will outline the conceptual and practical challenges of evaluation and begin a discussion of how to overcome them.

Biography: Kasper Hornbæk received his M.Sc. and Ph.D. in Computer Science from the University of Copenhagen, in 1998 and 2002, respectively. Since 2009 he has been a professor with special duties in Human-centered Computing at University of Copenhagen. His core research interests are human-computer interaction, usability research, search user interfaces, and information visualization; detours include eye tracking, cultural usability, and reality-based interfaces. Kasper serves on the editorial boards of Journal of Usability Studies, Interacting with Computers, and International Journal of Human-Computer Studies (IJHCS). He has published at CHI, UIST, ACM Transactions on Computer-Human Interaction, and Human-Computer Interaction, and won IJHCS’s most cited paper award 2006-2008.

The strong quality of BELIV continues with the 2010 edition.  12 research papers and 10 positions papers were accepted to our workshop.  Below is a preview of the titles and authors.  As BELIV gets closer, we’ll publish more information about each of these papers.  Stay tuned!

Research papers

• Developing Qualitative Metrics for Visual Analytic Environments. (Jean Scholtz)
• Do Mechanical Turks Dream of Square Pie Charts? (Robert Kosara, Caroline Ziemkiewicz)
• Many Roads Lead to Rome. Mapping Users’ Problem Solving Strategies. (Eva Mayr, Michael Smuc, Hanna Risku)
• Decomposing Visual Deciphering for Justification and Comparison of Representation. (Stephane Conversy, Christophe Hurter, Stephane Chatty)
• Exploring Information Visualization – Describing Different Interaction Patterns. (Sylvia Wiltner, Margit Pohl, Silvia Miksch, Markus Rester, Klaus Hinum, Christian Popow, Susanne Ohmann)
• Is Your User Hunting or Gathering Insights? Identifying Insight Drivers Across Domains. (Michael Smuc, Eva Mayr, Hanna Risku)
• Comparing Benchmark Task and Insight Evaluation Methods. (Chris North, Purvi Saraiya)
• Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations. (Michael Sedlmair, Petra Isenberg, Dominikus Baur, Andreas Butz)
• Learning-Based Evaluation of Visual Analytic Systems. (Remco Chang)
• Scanning Between Graph Visualizations: An Eye Tracking Evaluation. (Joseph Goldberg, Jonathan Helfman)
• Towards Information-Theoretic Visualization Evaluation Measure: A Practical example for Bertin’s Matrices. (Innar Liiv)
• Generating a synthetic video dataset. (Mark Whiting)

Position papers

• Visualization Evaluation of the Masses, by the Masses, and for the Masses. (Jeffrey Heer)
• Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study. (Peter Kinnaird, Mario Romero)
• Pragmatic Challenges in the Evaluation of Interactive Visualization Systems. (John Stasko, Youn-ah Kang, Carsten Gorg)
• Look Before You Link: Eye Tracking in Multiple Coordinated View Visualization. (Chris Weaver)
• Mutually Linked Studies – Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation. (Niklas Elmqvist)
• Beyond system logging: human logging for evaluating information visualization. (Nathalie Henry)
• How is a graphic like pumpkin pie? A framework for analysis and critique of visualisations. (Hadley Wickham)
• Proposed Working Memory Measures for Evaluating Information Visualization Tools. (Laura Matzen, Laura McNamara, Kerstan Cole, Alisa Bandlow, Courtney Dornburg, Travis Bauer)
• Comparative Evaluation of Two Interface Tools in Performing Visual Analytics Tasks. (Dong Hyun Jeong, William Ribarsky, Tera Green, Remco Chang)
• Implications of Individual Differences on Evaluating Information Visualization Techniques. (Ji Soo Yi)

There has been some confusion about the difference between position and research papers for BELIV’10, as a potential participant asks:

I’m thinking about submitting a paper to BELIV, but I’m not sure what
category to put it in – it’s a bit more than a position paper but not
quite a full research paper – i.e. I think it’s a nice idea and it’s
worked for me in the past, but it doesn’t have any formal validation.
Am I better of submitting it as a position paper (which I presume has
a high change of being accepted) or should I try for a full paper?
(and can I still submit it as a position paper if it is rejected?)

Our co-chair Enrico thoughtfully responds:

I understand we have a problem here, you are the second person to ask.   Of course, even if we have had very good quality papers in the past, BELIV is just a workshop and we don’t expect lots of validation or strong results. What we are aiming at is more well-reasoned, innovative and, if possible, thought provoking, ideas.

So, if you think your paper has these characteristics, go on and send a full paper! Position papers are just a way to say: “hey I want to participate and this is what I have to put on the table when we will discuss”. And if anything goes wrong, you can of course always convert your paper in a position paper, no worries.

In any case I suggest to you to give a look to the papers of past editions. Many good papers don’t have any formal validation at all. You can for instance give a look to Catherine’s paper on MILC methodology (BELIV’06), or the one of Heidi and Tamara in BELIV’08 on the utility of empirical studies, or the one of Petra et al. on Grounded Evaluation.

BELIV 2010 submissions are due November 30.  Similar to 2008, you may submit full research papers or position papers.  The complete details are on the BELIV 2010 website:  http://www.beliv.org/beliv2010/

• Deadline for submissions: November 30, 2009 (5:00pm PDT)
• Notification of acceptance: December 21, 2009
• Camera ready papers due: Mid-March
• Workshop: April 10-11, 2008