To Pretrain Or Not To Pretrain: Examining The Benefits Of Pretraining On Resource Rich Tasks · Awesome LLM Papers Contribute to LLM-Bible

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

To Pretrain Or Not To Pretrain: Examining The Benefits Of Pretraining On Resource Rich Tasks

Sinong Wang, Madian Khabsa, Hao Ma. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020 – 15 citations

[Paper]    
BERT Training Techniques Model Architecture Masked Language Model

Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.

Similar Work