User-Centered Evaluation of Visual Analytics / Synthesis Lectures on Visualization (PDF)

Visual analytics has come a long way since its inception in 2005. The amount of data in the world today has increased significantly and experts in many domains are struggling to make sense of their data. Visual analytics is helping them conduct their analyses. While software developers have worked for many years to develop software that helps users do their tasks, this task is becoming more and more onerous, as understanding the needs and data used by expert users requires more than some simple usability testing during the development process. The need for a user-centered evaluation process was envisioned in Illuminating the Path, the seminal work on visual analytics by James Thomas and Kristin Cook in 2005. We have learned over the intervening years that not only will user-centered evaluation help software developers to turn out products that have more utility, the evaluation efforts can also help point out the direction for future research efforts.

This book describes the efforts that go into analysis, including critical thinking, sensemaking, and various analytics techniques learned from the intelligence community. Support for these components is needed in order to provide the most utility for the expert users. There are a good number of techniques for evaluating software that hasbeen developed within the human-computer interaction (HCI) community. While some of these techniques can be used as is, others require modifications. These too are described in the book. An essential point to stress is that the users of the domains for which visual analytics tools are being designed need to be involved in the process. The work they do and the obstacles in their current processes need to be understood in order to determine both the types of evaluations needed and the metrics to use in these evaluations. At this point in time, very few published efforts describe more than informal evaluations. The purpose of this book is to help readers understand the need for more user-centered evaluations to drive both better-designed products and to define areas for future research. Hopefully readers will view this work as an exciting and creative effort and will join the community involved in these efforts.