Assessing The Factual Accuracy Of Generated Text | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs. Maintained by Sean Moran.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Assessing The Factual Accuracy Of Generated Text

Ben Goodrich, Vinay Rao, Mohammad Saleh, Peter J Liu . Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2019 – 145 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Datasets KDD

We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy). We introduce and release a new large-scale dataset based on Wikipedia and Wikidata to train relation classifiers and end-to-end fact extraction models. The end-to-end models are shown to be able to extract complete sets of facts from datasets with full pages of text. We then analyse multiple models that estimate factual accuracy on a Wikipedia text summarization task, and show their efficacy compared to ROUGE and other model-free variants by conducting a human evaluation study.

Similar Work