Bitfit: Simple Parameter-efficient Fine-tuning For Transformer-based Masked Language-models | Awesome LLM Papers Contribute to Awesome LLM Papers

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Bitfit: Simple Parameter-efficient Fine-tuning For Transformer-based Masked Language-models

Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg . Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) 2022 – 407 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
ACL Ethics & Fairness Fine Tuning Model Architecture Training Techniques

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Similar Work