Evaluating Theory Of Mind In Question Answering | Awesome LLM Papers Add your paper to Awesome LLM Papers

Evaluating Theory Of Mind In Question Answering

Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, Thomas L. Griffiths . Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing 2018 – 51 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Datasets EMNLP Interdisciplinary Approaches Question Answering

We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality. We evaluate a number of recent neural models with memory augmentation. We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models’ accuracy decreases notably when random sentences are introduced to the tasks at test.

Similar Work