Indobertweet: A Pretrained Language Model For Indonesian Twitter With Effective Domain-specific Vocabulary Initialization | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A collection of research papers on LLMs.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Indobertweet: A Pretrained Language Model For Indonesian Twitter With Effective Domain-specific Vocabulary Initialization

Fajri Koto, Jey Han Lau, Timothy Baldwin . Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021 – 53 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Datasets EMNLP Evaluation Interdisciplinary Approaches Model Architecture

We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.

Similar Work