The Handbook of Multimodal-Multisensor Interfaces, Volume 1

The Handbook of Multimodal-Multisensor Interfaces, Volume 1

Foundations, User Modeling, and Common Modality Combinations

Sharon Oviatt, Bjorn Schuller, Philip Cohen, Daniel Sonntag, Gerasimos Potamianos, Antonio Kruger
ISBN: 9781970001648 | PDF ISBN: 9781970001655
Hardcover ISBN:9781970001679
Copyright © 2017 | 636 Pages | Publication Date: May, 2017

BEFORE YOU ORDER: You may have Academic or Corporate access to this title. Click here to find out: 10.1145/3015783


Ordering Options: Paperback $99.95   E-book $79.96   Paperback & E-book Combo $124.94
Hardcover $149.95   Hardcover & E-book Combo $187.44

Why pay full price? Members receive 15% off all orders.
Learn More Here

Read Our Digital Content License Agreement (pop-up)

Purchasing Options:

Read Chapter 1 and Glossary

The Handbook of Multimodal-Multisensor Interfaces, Volume 1 provides the first authoritative resource on what has become the dominant paradigm for new computer interfaces-user input involving new media (speech, multi-touch, gestures, writing) embedded in multimodal-multisensor interfaces. These interfaces support smart phones, wearables, in-vehicle and robotic applications, and many other areas that are now highly competitive commercially. This edited collection is written by international experts and pioneers in the field. It provides a textbook, reference, and technology roadmap for professionals working in this and related areas. This first volume of the handbook presents relevant theory and neuroscience foundations for guiding the development of high-performance systems. Additional chapters discuss approaches to user modeling and interface designs that support user choice, that synergistically combine modalities with sensors, and that blend multimodal input and output. This volume also highlights an in-depth look at the most common multimodal-multisensor combinations-for example, touch and pen input, haptic and non-speech audio output, and speech-centric systems that co-process either gestures, pen input, gaze, or visible lip movements. A common theme throughout these chapters is supporting mobility and individual differences among users. These handbook chapters provide walkthrough examples of system design and processing, information on tools and practical resources for developing and evaluating new systems, and terminology and tutorial support for mastering this emerging field. In the final section of this volume, experts exchange views on a timely and controversial challenge topic, and how they believe multimodal-multisensor interfaces should be designed in the future to most effectively advance human performance.


Table of Contents

1. Introduction
2. Theoretical Foundations of Multimodal Interfaces and Systems
3. The Impact of Multimodal-Multisensory Learning on Human Performance and Brain Activation Patterns
4. Multisensory Haptic Interactions: Understanding the Sense and Designing for It
5. A Background Perspective on Touch as a Multimodal (and Multi-Sensor) Construct
6. Understanding and Supporting Modality Choices
7. Using Cognitive Models to Understand Multimodal Processes: The Case for Speech and Gesture Production
8. Multimodal Feedback in HCI: Haptics, Non-Speech Audio, and Their Applications
9. Multimodal Technologies for Seniors: Challenges and Opportunities
10. Gaze Informed Multimodal Interaction
11. Multimodal Speech and Pen Interfaces
12. Multimodal Gesture Recognition
13. Audio and Visual Modality Combination in Speech Processing Applications
14. Multidisciplinary Challenge Topic: Perspectives on Learning with Multimodal Technology
Contributors' Brief Biographies

About the Author(s)

Sharon Oviatt, Incaa Designs
Sharon Oviatt is internationally known for her multidisciplinary work on multimodal and mobile interfaces, human-centered interfaces, educational interfaces, and learning analytics. She has been the recipient of the inaugural ACM-ICMI Sustained Accomplishment Award, National Science Foundation Special Creativity Award, ACM SIGCHI CHI Academy Award, and an ACM Fellow Award. She has published over 160 scientific articles in a wide range of venues, and is Associate Editor of the main journals and edited book collections in the field of human-centered interfaces. Her other books include The Design of Future Educational Interfaces (2013, Routledge) and The Paradigm Shift to Multimodality in Contemporary Computer Interfaces (2015, Morgan Claypool).

Bjorn Schuller, University of Passau and Imperial College London
Bjorn Schuller is currently Chair of Complex and Intelligent Systems at University of Passau and Reader in Machine Learning at Imperial College. He is best known for his work on multisensorial/multimodal intelligent signal processing for affective, behavioral, and human-centered computing. In 2015 and 2016, he was honored by the World Economic Forum as one of 40/50 extraordinary scientists under age 40. His further awards include the CHiME, MediaEval, and MIREX competitions. He has published over 500 peer-reviewed scientific contributions across a range of disciplines and venues, and is Editor-in-Chief of IEEE Transactions on Affective Computing. His books include Intelligent Audio Analysis (2013, Springer) and Computational Paralinguistics (2013, Wiley).

Philip Cohen, VoiceBox Technologies
Philip R. Cohen is Senior Vice President and Chief Scientist for Artificial Intelligence, whose research interests include multimodal interaction, human-computer dialogue, and multi-agent systems. He is a Fellow of the American Association for Artificial Intelligence, past President of the Association for Computational Linguistics, and recipient (with Hector Levesque) of an Inaugural Influential Paper Award by the International Foundation for Autonomous Agents and Multi-Agent Systems. He was the Founder of Adapx, Professor and Co-Director of the Center for Human-Computer Communication in Computer Science at Oregon Health and Science University, and Director of Natural Language in the Artificial Intelligence Center at SRI International.

Daniel Sonntag, German Research Center for Artificial Intelligence
Daniel Sonntag is a Principal Researcher and Research Fellow. His research interests include multimodal and mobile AI-based interfaces, common-sense modeling, and semantic machine learning methods for cognitive computing and improved usability. He has published over 120 scientific articles, and was the recipient of the German High Tech Champion Award in 2011 and the AAAI Recognition and IAAI Deployed Application Award in 2013. He is an editorial board member of the German Journal on Artificial Intelligence (KI). Currently, he leads both national and European projects from the Federal Ministry of Education and Research, the Federal Ministry for Economic Affairs and Energy, and Horizon 2020.

Gerasimos Potamianos, University of Thessaly
Gerasimos Potamianos is Associate Professor and Director of Graduate Studies in Electrical and Computer Engineering. His research spans multisensory and multimodal speech processing and scene analysis, with applications to human-computer interaction and ambient intelligence. He has authored over 120 articles, and has 7 patents. He received a Diploma degree from the National Technical University of Athens, and a M.Sc. and Ph.D. from Johns Hopkins University, all in electrical and computer engineering. In addition to his academic experience, he has worked at AT&T Research Labs, IBM Thomas J.Watson Research Center (US), and at the FORTH and NCSR 'Demokritos' Research Centers in Greece.

Antonio Kruger, German Research Center for Artificial Intelligence
Antonio Kruger is Professor of Computer Science and Director of the Media Informatics Program at Saarland University, as well as Scientific Director of the Innovative Retail Laboratory at the German Research Center for Artificial Intelligence (DFKI). His research areas focus on intelligent user interfaces, and mobile and ubiquitous context-aware systems. He has been General Chair of the Ubiquitous Computing Conference, and Program Chair of MobileHCI, IUI, and Pervasive Computing. He is also on the Steering Committee Chair of Intelligent User Interfaces (IUI), and an Associate Editor of the journals User Modeling and User-Adapted Interaction and ACM Interactive, Mobile, Wearable and Ubiquitous Technologies.


Customers who bought this product also purchased
Research in the Wild
Research in the Wild
Browse by Subject
Case Studies in Engineering
ACM Books
IOP Concise Physics
SEM Books
0 items

Note: Registered customers go to: Your Account to subscribe.

E-Mail Address:

Your Name: