Conversation-and-Control

Controlling graphical user interfaces (GUI) by speech is slow, but proves useful for disabled persons with limitations in operating mouse and keyboard. This book presents a new approach, which promises to achieve shorter task completion times than prior art. The author first introduces speech-controlled GUI research by giving an overview of currently available technology. Then he analyzes the properties of different technological approaches for which he derives a common theoretical model. Motivated by the results, he then develops Conversation-and-Control, a new approach, which facilitates direct manipulation of graphical objects by spoken commands. The approach employs mixed-initiative dialogs based on heuristics for inferring the meaning of speech recognition results. The model also includes the ability for the system to ask clarification questions, thus, avoids the rejection of recognition errors where possible. The author describes the internal algorithms of Conversation-and-Control and discusses a user study which indicates a significant reduction of task completion time over prior art. The book is directed to researches and practitioners in speech-controlled GUI research.