Both Text And Images Leaked! A Systematic Analysis Of Multimodal LLM Data Contamination | Awesome LLM Papers Contribute to Awesome LLM Papers

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Both Text And Images Leaked! A Systematic Analysis Of Multimodal LLM Data Contamination

Dingjie Song, Sicheng Lai, Shunian Chen, Lichao Sun, Benyou Wang . No Venue 2024

[Paper] [Paper]   Search on Google Scholar   Search on Semantic Scholar
Datasets Evaluation Fine Tuning Tools Training Techniques

The rapid progression of multimodal large language models (MLLMs) has demonstrated superior performance on various multimodal benchmarks. However, the issue of data contamination during training creates challenges in performance evaluation and comparison. While numerous methods exist for detecting dataset contamination in large language models (LLMs), they are less effective for MLLMs due to their various modalities and multiple training phases. In this study, we introduce a multimodal data contamination detection framework, MM-Detect, designed for MLLMs. Our experimental results indicate that MM-Detect is sensitive to varying degrees of contamination and can highlight significant performance improvements due to leakage of the training set of multimodal benchmarks. Furthermore, We also explore the possibility of contamination originating from the pre-training phase of LLMs used by MLLMs and the fine-tuning phase of MLLMs, offering new insights into the stages at which contamination may be introduced.

https://huggingface.co/discussions/paper/672c2ca8dbc16559ba1d8fd1

Similar Work