Group Gated Fusion On Attention-based Bidirectional Alignment For Multimodal Emotion Recognition | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs. Maintained by Sean Moran.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Group Gated Fusion On Attention-based Bidirectional Alignment For Multimodal Emotion Recognition

Pengfei Liu, Kun Li, Helen Meng . Interspeech 2020 2020 – 42 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Affective Computing Datasets Image Text Integration Interspeech Model Architecture Visual Contextualization

Emotion recognition is a challenging and actively-studied research area that plays a critical role in emotion-aware human-computer interaction systems. In a multimodal setting, temporal alignment between different modalities has not been well investigated yet. This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states to explicitly capture the alignment relationship between speech and text, and a novel group gated fusion (GGF) layer to integrate the representations of different modalities. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly, and the proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.

Similar Work