As background, while working on text-to-SQL as a general problem, we came to believe one of its most powerful applications is as a search tool because:
- SQL is hard to write by hand and prone to errors
- It allows you to iterate quickly if you’re exploring a new dataset
- A lot of contextual information that you’d normally have to internalize (e.g. your data’s schema) can be automatically generated and offloaded to the language model
Using large language models (LLMs) like Codex to write the SQL for you means you don’t have to worry about the nitty gritty language details, but still benefit from the power of a language like SQL. Also, after seeing the results of the query, you can inspect (and if necessary, change) the SQL. The lack of this sort of explainability of the query result is one of the more notorious challenges of returning the output of an LLM directly to the user. Additionally, using LLMs in this way makes these kinds of queries over structured data accessible to people who know little or no SQL.
While Bird SQL shares significant infrastructure with our more general LLM-powered search engine over unstructured data (Ask Perplexity - https://perplexity.ai[1]), the two approaches and their respective challenges are quite different. For example, the type of models are different (GPT3.5 vs Codex), obviously the model prompts have different structure, and how to verify model output when its text vs when it's code is different. We are currently exploring ways to combine the two approaches, such as using the results of retrieving information from a structured source (as in Bird SQL) as one of the inputs for the LLM to interpret or summarize (as in Ask Perplexity).
We would love to hear your questions, suggestions, and feedback!