SBAT: Video Captioning With Sparse Boundary-aware Transformer | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

SBAT: Video Captioning With Sparse Boundary-aware Transformer

Tao Jin, Siyu Huang, Ming Chen, Yingming Li, Zhongfei Zhang . Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence 2020 – 50 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Compositional Generalization Datasets Evaluation IJCAI Image Text Integration Interactive Environments Interdisciplinary Approaches Model Architecture Neural Machine Translation RAG Visual Contextualization Visual Question Answering

In this paper, we focus on the problem of applying the transformer structure to video captioning effectively. The vanilla transformer is proposed for uni-modal language generation task such as machine translation. However, video captioning is a multimodal learning problem, and the video features have much redundancy between different time steps. Based on these concerns, we propose a novel method called sparse boundary-aware transformer (SBAT) to reduce the redundancy in video representation. SBAT employs boundary-aware pooling operation for scores from multihead attention and selects diverse features from different scenarios. Also, SBAT includes a local correlation scheme to compensate for the local information loss brought by sparse operation. Based on SBAT, we further propose an aligned cross-modal encoding scheme to boost the multimodal interaction. Experimental results on two benchmark datasets show that SBAT outperforms the state-of-the-art methods under most of the metrics.

Similar Work