How To Train BERT With An Academic Budget
Peter Izsak, Moshe Berchansky, Omer Levy . Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021 – 70 citations
While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.