The File prioritizer capability is validated by the test_file_prioritizer test case.

The batch processing pipeline orchestration is validated by the test_progress_status test case.

The Question router capability is validated by the test_question_router test case.

The DataLens Master Implementation Plan includes the Testing, Benchmarks, Polish epic. Testing, Benchmarks, Polish phase uses the DABStep dataset for query accuracy measurement. Testing, Benchmarks, Polish references DABStep for data analysis benchmarks. Testing, Benchmarks, Polish uses KramaBench for data analysis benchmarking.

Architectural pattern for semantic boundary detection using Python libraries (python-docx, python-pptx) and JSON for table embedding, enabling better document structure preservation.

Implements a Transform-Extract-Transform pipeline for standardized data processing across CSV, Excel, and PDF files. DataLens Development implements the TET Pipeline pattern for data extraction standardization as a core business process.

Text chunk storage persists extracted text chunks including DOCX and PPTX chunks into the DuckDB database's doc_text_chunks table.

The Text chunks table in DuckDB is a physical table supporting DuckDB SQL queries. The text chunks table in DuckDB with full-text search support is mapped to the DuckDB SQL data store.

The DataLens Master Implementation Plan includes the Text-to-SQL epic. Text-to-SQL integrates with Vanna.AI. Text-to-SQL integrates with Ollama. Text-to-SQL uses DuckDB for schema auto-training. The file backend/app/services/text_to_sql.py is part of the TEXT-TO-SQL Service. The QuestionRouter class uses the TEXT-TO-SQL Service for SQL query generation. The Backend implements the Text-to-SQL capability. The DataLens Platform uses the Text-to-SQL capability. Text-to-SQL is integrated within the DataLens platform backend. The Text-to-SQL feature uses Ollama hosted on elin accessible via the Ollama URL. The Text-to-SQL capability integrates with the external system Vanna.AI for best practice implementations and query optimization. Text-to-SQL capability integrates with Ollama external system for AI model usage within schema auto-training and natural language to SQL pipelines. The Text-to-SQL capability uses DuckDB as a data platform for schema auto-training and SQL query operations. The DataLens platform backend includes the Text-to-SQL feature. QuestionRouter uses Text-to-SQL capability for structured query execution through TextToSQLService. The Frontend uses the Text-to-SQL analysis capability. The DataLens Platform incorporates Text-to-SQL functionality for natural language queries. SQLAgent realizes the Text-to-SQL capability by converting natural language to SQL executions. The DataLens platform backend uses Text-to-SQL capabilities.

The FastAPI backend exposes a Text-to-SQL query API for natural language queries. Text-to-SQL incorporates the natural language to SQL pipeline for query translation.

AI Core capability requires the Text-to-SQL with Ollama capability. DS-STAR AI cataloging system uses Text-to-SQL with Ollama for natural language query translation.

text_to_sql.py is changed to generate PostgreSQL-compatible SQL instead of DuckDB SQL by modifying the prompt to specify PostgreSQL dialect. The text_to_sql.py prompt and code were modified to generate PostgreSQL-specific SQL compatible with PgDataService. The backend code file text_to_sql.py uses SQLCoder-7B as the default model for Text-to-SQL queries.

Entity representing text chunks used in extraction and semantic search processes.

TextToSQLService converts natural language questions into SQL queries using LLMs, building prompts with schema info and parsing responses for execution. QuestionRouter uses TextToSQLService for generating SQL from natural language queries on the structured path. TextToSQLService integrates with Ollama LLM by calling its generate API on qwen3-coder-next model to generate SQL queries. QuestionRouter uses TextToSQLService for generating SQL from natural language queries on the structured path. TextToSQLService integrates with Ollama LLM by calling its generate API on qwen3-coder-next model to generate SQL queries. TextToSQLService depends on IronClawClient integration for agent logic involving SQL generation. QuestionRouter depends on TextToSQLService to convert natural language queries to SQL. DataLens Development uses TextToSQLService which connects to Ollama for generating SQL from natural language queries. TextToSQLService enables QueryDataSkill to convert natural language queries to SQL queries. Text-to-SQL Service is defined in backend/app/services/text_to_sql.py. The generate_sql method is part of the Text-to-SQL Service.

The TextToSQLService class is defined within backend/app/services/text_to_sql.py. The TextToSQLService class integrates with the Ollama API for LLM-based SQL generation. The generate_sql method is part of the Text-to-SQL Service.

DataLens is deployed on theo, which runs containerized APIs and web frontend via Coolify, coordinating GPU-based document extraction on elin through SSH tunnels. theo manages backend services including FastAPI, RQ workers, PostgreSQL, Redis, and DuckDB, orchestrating document processing, storage, and agent interactions including the DS-STAR API and firewall configurations. The theo backend integrates with the elin GPU server via SSH to orchestrate Docling extraction and embedding jobs on the GPU hardware running on elin. SQLCoder-7B is used by the FastAPI backend on theo for Text-to-SQL queries. theo hosts the FastAPI backend which runs file extraction, DuckDB storage, and uses SQLCoder-7B for query processing. The theo backend server connects to the IronClaw service running on elin to delegate agent message processing via IronClaw Gateway API. theo Backend depends on IronClaw Gateway running on elin server to manage agent sessions and route requests. theo Backend uses the OpenClaw Skill API via HTTP on elin to execute ringfenced database queries safely for the agent. theo Backend accesses PostgreSQL database to manage budget files metadata as part of the data platform. User accesses the theo Backend as main API and service endpoint to interact with the platform for budget analysis. theo Backend is implemented with FastAPI for its web API and service operations. Frontend depends on the theo Backend API to obtain data and agent responses. theo Backend communicates with OpenClaw HTTP API for streaming agent sessions and ringfenced skill executions. Theo maintains an SSH tunnel to elin for connectivity.

Timeout issues with long-running queries are mitigated by streaming responses via the /ask-stream endpoint to prevent HTTP client timeouts. Timeout issues during query classification are mitigated by switching to keyword classification which takes under 10ms compared to prior 120s LLM classification.

Recent code changes to improve system loading have caused responses to now return empty in the OpenClaw integration. Reverting to last known good commit is planned to restore proper budget data extraction.

Vanna 2.0 implements a Tool Registry that supports extensible tools with access control via access_groups. Vanna 2.0 uses a tool registry to extend tools with access group permissions. Vanna 2.0 architecture uses a tool registry for extensible tools with permissions

DataLens requires a Tool Registry pattern to support extensible custom actions beyond SQL, such as email and notifications, with permissions. DataLens requires implementing the tool registry pattern to support extensible custom tools. DataLens requires a Tool Registry pattern to add extensible custom actions beyond SQL

DataLens Development implements the Transform-Extract-Transform Pipeline to standardize data extraction workflows.