Negated And Misprimed Probes For Pretrained Language Models: Birds Can Talk, But Cannot Fly | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A continuously updated collection of research papers on LLMs.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Negated And Misprimed Probes For Pretrained Language Models: Birds Can Talk, But Cannot Fly

Nora Kassner, Hinrich Schütze . Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020 – 68 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
ACL Compositional Generalization Interdisciplinary Approaches Multimodal Semantic Representation

Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated (“Birds cannot [MASK]”) and non-negated (“Birds can [MASK]”) cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add “misprimes” to cloze questions (“Talk? Birds can [MASK]”). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

Similar Work