Prompting Large Language Models With Rationale Heuristics For Knowledge-based Visual Question Answering | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs. Maintained by Sean Moran.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Prompting Large Language Models With Rationale Heuristics For Knowledge-based Visual Question Answering

Zhongjian Hu, Peng Yang, Bing Li, Fengyuan Liu . 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023 – 150 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
3d Representation CVPR Compositional Generalization Interdisciplinary Approaches Multimodal Semantic Representation Prompting Question Answering Tools Visual Question Answering

Recently, Large Language Models (LLMs) have been used for knowledge-based Visual Question Answering (VQA). Despite the encouraging results of previous studies, prior methods prompt LLMs to predict answers directly, neglecting intermediate thought processes. We argue that prior methods do not sufficiently activate the capacities of LLMs. We propose a framework called PLRH that Prompts LLMs with Rationale Heuristics for knowledge-based VQA. The PLRH prompts LLMs with Chain of Thought (CoT) to generate rationale heuristics, i.e., intermediate thought processes, and then leverages the rationale heuristics to inspire LLMs to predict answers. Experiments show that our approach outperforms the existing baselines by more than 2.2 and 2.1 on OK-VQA and A-OKVQA, respectively.

Similar Work