How AskTable Generates Accurate SQL: Meta Brain, Meta Retrieval, and Data Retrieval

Query your company data with a single sentence, just like chatting. Generate charts easily, get business insights quickly. ☀️

Many users say that it's better than expected, and are curious about how we achieved this. Taking this opportunity, I'd like to explain: AskTable is a SQL-based query AI. Therefore, for good query results, SQL is key; and to generate accurate SQL, besides choosing a reliable large language model, knowing how to control these large/small models is equally important. It's like cooking—ingredients matter, but the chef matters more.

Inside AskTable, the system for generating SQL can be divided into three core modules: Meta Brain, Meta Retrieval, and Data Retrieval, as shown in the diagram below.

Intelligence Center—Meta Brain: Building the Metadata Graph

Meta Brain is AskTable's intelligence center, serving as the knowledge base for the entire data system. It stores all metadata in the database along with necessary keyword data, including database names, table names, field names, and their annotations. It uses AI-driven automated construction to create a comprehensive database metadata graph. This enables AskTable to understand data structures across different database systems (such as MySQL, Oracle, PostgreSQL, Dameng, Hive, etc.) and efficiently and accurately generate corresponding SQL queries when needed.

Meta Brain is also the storage and computation center for embedding and various related/similarity searches. All queries are searched and computed here to find the data tables, fields, and data values that best match the user's query intent. The combination of vector search and graph database technology enables Meta Brain to quickly locate target data in large-scale data, improving query efficiency.

Reconnaissance Team—Meta Retrieval: Entity Recognition and Metadata Retrieval

When a user initiates a query, the system first enters the Meta Retrieval module. Here, the question goes through natural language processing to extract key entities or relationships, such as location names, user names, etc. This step uses entity recognition technology to help the system identify and understand the core content of the user's query.

Next, the system calls an embedding model to convert the extracted entities into vector representations. These vector representations are queried in Meta Brain to find the most relevant database fields and tables. Semantic vector retrieval enables the system to efficiently match within complex data structures, ensuring query accuracy and recognition of colloquial expressions.

Additionally, the system uses built-in search algorithms specifically optimized for Meta Data and Short Value structures to match real entity names from Meta Brain. This prevents Value Missing during SQL generation.

Furthermore, Meta Retrieval is also responsible for query permission management. Based on user roles and permissions, the system filters out data that doesn't meet access requirements.

Action Team—Data Retrieval: Intelligent Learning and Dynamic Error Correction

The Data Retrieval module is responsible for executing generated SQL queries and retrieving data. Unlike traditional database queries, AskTable adds intelligent learning and error correction mechanisms to this process. When query results don't match expectations, the system uses AI models to dynamically adjust or rewrite the SQL to optimize results. This dynamic error correction mechanism significantly improves response accuracy in complex query scenarios.

At the same time, AskTable continuously optimizes SQL generation effectiveness across different database dialects and table structures through analysis of Good/Bad Cases, reinforcement learning and training based on user feedback, further improving query accuracy.

Before data retrieval, the system also performs further permission verification. Each row of data is filtered again according to permission rules before being returned to the user, ensuring data compliance and security. Through this strict multi-layer filtering, AskTable not only guarantees data security but also greatly reduces data redundancy.

Other

In addition to the above, we also introduced advanced features such as "Term Library," "Trainer," and "Private Preferences" in the SQL generation process. Click the original article to learn more, or scan the QR code below to join the group for discussion. If you have cooperation needs, you can also follow our official account and click the button at the bottom to contact us.

We believe in AI, believe in LLM, believe that they make us forever innovative and forward-looking. We respect time, respect privacy, respect that makes us cautious, pragmatic, and objective.