Benchmarking Zero-shot Text Classification: Datasets, Evaluation And Entailment Approach
Wenpeng Yin, Jamaal Hay, Dan Roth . Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019 – 219 citations
Zero-shot text classification (0Shot-TC) is a challenging NLU problem to
which little attention has been paid by the research community. 0Shot-TC aims
to associate an appropriate label with a piece of text, irrespective of the
text domain and the aspect (e.g., topic, emotion, event, etc.) described by the
label. And there are only a few articles studying 0Shot-TC, all focusing only
on topical categorization which, we argue, is just the tip of the iceberg in
0Shot-TC. In addition, the chaotic experiments in literature make no uniform
comparison, which blurs the progress.
This work benchmarks the 0Shot-TC problem by providing unified datasets,
standardized evaluations, and state-of-the-art baselines. Our contributions
include: i) The datasets we provide facilitate studying 0Shot-TC relative to
conceptually different and diverse aspects: the topic'' aspect includes
sports’’ and politics'' as labels; theemotion’’ aspect includes joy''
andanger’’; the situation'' aspect includesmedical assistance’’ and
``water shortage’’. ii) We extend the existing evaluation setup
(label-partially-unseen) – given a dataset, train on some labels, test on all
labels – to include a more challenging yet realistic evaluation
label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text
snippets without seeing task specific training data at all. iii) We unify the
0Shot-TC of diverse aspects within a textual entailment formulation and study
it this way.
Code & Data: https://github.com/yinwenpeng/BenchmarkingZeroShot