Text-to-text Pre-training For Data-to-text Tasks · Awesome LLM Papers Contribute to LLM-Bible

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Text-to-text Pre-training For Data-to-text Tasks

Mihir Kale, Abhinav Rastogi. Proceedings of the 13th International Conference on Natural Language Generation 2020 – 50 citations

[Paper]    
Language Modeling Model Architecture GPT Transformer Fine-Tuning Pre-Training BERT Training Techniques Evaluation

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

Similar Work