Evaluating The Text-to-sql Capabilities Of Large Language Models | Awesome LLM Papers Add your paper to Awesome LLM Papers

Awesome LLM Papers

A collection of research papers on LLMs.

Stay Updated

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Evaluating The Text-to-sql Capabilities Of Large Language Models

Nitarshan Rajkumar, Raymond Li, Dzmitry Bahdanau . Arxiv 2022 – 48 citations

[Paper]   Search on Google Scholar   Search on Semantic Scholar
Compositional Generalization Evaluation Few Shot Interdisciplinary Approaches Multimodal Semantic Representation Prompting

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

Similar Work