The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events

Jörn Anemüller, Jörg Hendrik Bach, Barbara Caputo, Michal Havlena, Luo Jie, Hendrik Kayser, Bastian Leibe, Petr Motlicek, Tomas Pajdla, Misha Pavel, Akihiko Torii, Luc Van Gool, Alon Zweig, Hynek Hermansky

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ("Detection and Identification of Rare Audio-visual Cues") we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ("AWEAR" - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.

Original languageEnglish (US)
Title of host publicationICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces
Pages289-292
Number of pages4
DOIs
StatePublished - 2008
Event10th International Conference on Multimodal Interfaces, ICMI'08 - Chania, Crete, Greece
Duration: Oct 20 2008Oct 22 2008

Other

Other10th International Conference on Multimodal Interfaces, ICMI'08
CountryGreece
CityChania, Crete
Period10/20/0810/22/08

Fingerprint

Hearing aids
Hardware
Stereo vision
Microphones
Acoustic waves
Sensors

Keywords

  • Audio-visual
  • Augmented cognition
  • Event detection
  • Multimodal interaction
  • Sensor platform

ASJC Scopus subject areas

  • Computer Science Applications
  • Human-Computer Interaction

Cite this

Anemüller, J., Bach, J. H., Caputo, B., Havlena, M., Jie, L., Kayser, H., ... Hermansky, H. (2008). The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events. In ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces (pp. 289-292) https://doi.org/10.1145/1452392.1452451

The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events. / Anemüller, Jörn; Bach, Jörg Hendrik; Caputo, Barbara; Havlena, Michal; Jie, Luo; Kayser, Hendrik; Leibe, Bastian; Motlicek, Petr; Pajdla, Tomas; Pavel, Misha; Torii, Akihiko; Gool, Luc Van; Zweig, Alon; Hermansky, Hynek.

ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces. 2008. p. 289-292.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Anemüller, J, Bach, JH, Caputo, B, Havlena, M, Jie, L, Kayser, H, Leibe, B, Motlicek, P, Pajdla, T, Pavel, M, Torii, A, Gool, LV, Zweig, A & Hermansky, H 2008, The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events. in ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces. pp. 289-292, 10th International Conference on Multimodal Interfaces, ICMI'08, Chania, Crete, Greece, 10/20/08. https://doi.org/10.1145/1452392.1452451
Anemüller J, Bach JH, Caputo B, Havlena M, Jie L, Kayser H et al. The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events. In ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces. 2008. p. 289-292 https://doi.org/10.1145/1452392.1452451
Anemüller, Jörn ; Bach, Jörg Hendrik ; Caputo, Barbara ; Havlena, Michal ; Jie, Luo ; Kayser, Hendrik ; Leibe, Bastian ; Motlicek, Petr ; Pajdla, Tomas ; Pavel, Misha ; Torii, Akihiko ; Gool, Luc Van ; Zweig, Alon ; Hermansky, Hynek. / The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events. ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces. 2008. pp. 289-292
@inproceedings{a48979352a2548e398ae1e94d02ce8ec,
title = "The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events",
abstract = "It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ({"}Detection and Identification of Rare Audio-visual Cues{"}) we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ({"}AWEAR{"} - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.",
keywords = "Audio-visual, Augmented cognition, Event detection, Multimodal interaction, Sensor platform",
author = "J{\"o}rn Anem{\"u}ller and Bach, {J{\"o}rg Hendrik} and Barbara Caputo and Michal Havlena and Luo Jie and Hendrik Kayser and Bastian Leibe and Petr Motlicek and Tomas Pajdla and Misha Pavel and Akihiko Torii and Gool, {Luc Van} and Alon Zweig and Hynek Hermansky",
year = "2008",
doi = "10.1145/1452392.1452451",
language = "English (US)",
isbn = "9781605581989",
pages = "289--292",
booktitle = "ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces",

}

TY - GEN

T1 - The DIRAC AWEAR audio-visual platform for detection of unexpected and incongruent events

AU - Anemüller, Jörn

AU - Bach, Jörg Hendrik

AU - Caputo, Barbara

AU - Havlena, Michal

AU - Jie, Luo

AU - Kayser, Hendrik

AU - Leibe, Bastian

AU - Motlicek, Petr

AU - Pajdla, Tomas

AU - Pavel, Misha

AU - Torii, Akihiko

AU - Gool, Luc Van

AU - Zweig, Alon

AU - Hermansky, Hynek

PY - 2008

Y1 - 2008

N2 - It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ("Detection and Identification of Rare Audio-visual Cues") we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ("AWEAR" - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.

AB - It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ("Detection and Identification of Rare Audio-visual Cues") we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ("AWEAR" - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.

KW - Audio-visual

KW - Augmented cognition

KW - Event detection

KW - Multimodal interaction

KW - Sensor platform

UR - http://www.scopus.com/inward/record.url?scp=63449125358&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=63449125358&partnerID=8YFLogxK

U2 - 10.1145/1452392.1452451

DO - 10.1145/1452392.1452451

M3 - Conference contribution

AN - SCOPUS:63449125358

SN - 9781605581989

SP - 289

EP - 292

BT - ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces

ER -