Raise A Child In Large Language Model: Towards Effective And Generalizable Fine-tuning · Awesome LLM Papers Contribute to LLM-Bible

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Raise A Child In Large Language Model: Towards Effective And Generalizable Fine-tuning

Runxin Xu et al.. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021 – 46 citations

[Paper]    
Training Techniques Fine-Tuning RAG Evaluation

Recent pretrained language models extend from millions to billions of parameters. Thus the need to fine-tune an extremely large pretrained model with a limited training corpus arises in various downstream tasks. In this paper, we propose a straightforward yet effective fine-tuning technique, Child-Tuning, which updates a subset of parameters (called child network) of large pretrained models via strategically masking out the gradients of the non-child network during the backward process. Experiments on various downstream tasks in GLUE benchmark show that Child-Tuning consistently outperforms the vanilla fine-tuning by 1.5~8.6 average score among four different pretrained models, and surpasses the prior fine-tuning techniques by 0.6~1.3 points. Furthermore, empirical results on domain transfer and task transfer show that Child-Tuning can obtain better generalization performance by large margins.

Similar Work