The Data Discovery feature architecture uses Qwen3 with large context for table selection and Arctic with smaller context for SQL generation. The Data Discovery feature architecture uses Qwen3 with large context for table selection and Arctic with smaller context for SQL generation. Arctic is integrated within DataLens as the LLM model used for SQL query generation. The Data Discovery System integrates with Arctic LLM inference component for SQL query generation from selected tables. The analysis pipeline uses Arctic to generate SQL queries over the unified schema. The Discovery Service passes unified schemas to Arctic LLM for SQL generation after table consolidation. The Data Discovery Architecture uses Arctic for SQL generation with a limited context window.

Arctic SQL queries are executed against PostgreSQL instead of DuckDB after migration, requiring minimal SQL translation for compatibility. Transient views created during consolidation are used as input schema by Arctic for SQL generation. The Data Discovery System requires the Arctic SQL generation capability to generate SQL queries on consolidated tables. Arctic SQL currently generates SQL queries targeting DuckDB. Post-migration Arctic SQL will generate SQL queries targeting PostgreSQL with minor translation.

Request model for submitting questions to API.

Auth system successfully integrated with login/auth endpoints. User login fixed with correct credentials, enabling secure access. Frontend and backend authentication are functional, supporting Danish language and user sessions in production.

Backend uses FastAPI, supports multi-tenant auth, and integrates DS-STAR, Text-to-SQL, and PostgreSQL, Redis, Qdrant, Ollama on elin. Critical components include question_router, findings_generator, and document extractors. It manages project data, files, and analysis API, with fallback mechanisms for Qdrant. It currently uses DuckDBService; after migration, it will replace this with PgDataService to manage data in PostgreSQL.

The backend service depends on the PostgreSQL 16 service as its metadata storage with connection configured via environment variable. The backend service depends on the Redis 7 service for job queueing with connection configured via environment variable. The backend service uses DS-STAR agents and their Python virtual environment from the mounted /home/ops/datalens directory for execution. The Backend implements the DS-STAR intelligence capability. The Backend implements the Text-to-SQL capability. The Backend implements the Document RAG capability. The DataLens Platform contains the Backend component. Coolify Application depends on the Backend for API functionality. Backend integrates with PostgreSQL for authentication and project metadata. Backend includes a production-ready Dockerfile for deployment. The Data Discovery feature includes the Discovery backend service. The Discovery backend service uses defined API endpoints for its operation. The Backend Service uses the defined API endpoints including the discovery endpoints. The Backend API integrates with the Discovery backend service via the defined routes. The Data Discovery feature includes the Discovery backend service. The Discovery backend service uses defined API endpoints for its operation. The Backend Service uses the defined API endpoints including the discovery endpoints. The Backend API integrates with the Discovery backend service via the defined routes. Integration requires registering and using the Backend service's API endpoints for discovery.

The file backend/app/api/analysis.py is part of the API LAYER. The QuestionRouter class is used by the API LAYER including the backend/app/api/analysis.py endpoint for processing queries. The API Layer contains the backend/app/api/analysis.py file. The file backend/app/api/analysis.py contains the /analysis/ask endpoint. The file backend/app/api/analysis.py contains the /analysis/query endpoint. The file backend/app/api/analysis.py contains the /analysis/history endpoint. The analysis.py API layer uses QuestionRouter to route questions to appropriate services.

The backend app api discovery router is part of the Backend discovery service to provide API integration.

The generate-goal API endpoint is implemented in backend/app/api/projects.py.

The file backend/app/services/text_to_sql.py is part of the TEXT-TO-SQL Service. The TextToSQLService class is defined within backend/app/services/text_to_sql.py. The streaming responses via the /ask-stream endpoint realize the Text-to-SQL analysis query endpoint by enabling streaming of query results to avoid client timeouts. The backend service text_to_sql.py uses SQLCoder-7B as the configured default model for Text-to-SQL queries after the code change. Coolify rebuilds the backend container to deploy updated Python code including changes in text_to_sql.py. backend/app/services/text_to_sql.py implements schema compression and SQL generation with error retry for the Multi-Stage Text-to-SQL Architecture. The text to SQL service depends on the DuckDB service to execute generated SQL queries against the extracted data. The code in backend/app/services/agent_skills.py depends on backend/app/services/text_to_sql.py for SQL extraction functionality within agent workflows. Text-to-SQL Service is defined in backend/app/services/text_to_sql.py.

Abstract base client in backend/app/services/ironclaw_client.py for communicating with IronClaw API.

A large-scale GPU-first extraction process targeting 214 files, expected to complete in about 15-20 minutes, using Docling and Ollama for document parsing and embeddings. The batch processor orchestrator manages background workers including catalog, extract, vectorize, and prioritize jobs. The file prioritizer component is utilized by the batch processor orchestrator to assign tiers and manage processing order. Batch extraction subprocess is part of Backend batch processing strategy using subprocess calls instead of RQ for extraction jobs.

The Batch upload API endpoint provides the backend interface for the batch upload pipeline. The batch upload API endpoint is used to receive batch uploads for processing by the batch upload pipeline.

The batch upload pipeline with smart processing is implemented as a feature within DataLens, supporting phases like AI cataloging and file processing. It utilizes new extractors (DOCX, PPTX, MSG) and backend services for orchestrating uploads, extraction, and vectorization, integrated into a comprehensive UX model.

The extraction worker chains to the batch vectorize job to process GPU embeddings after extraction. Extracted text chunks from DOCX and PPTX files are vectorized in batch and indexed into Qdrant for semantic search. batch_vectorize_job depends on EmbeddingService to perform batch vectorization of extracted document chunks. The batch_vectorize_job processes text chunks to generate embeddings using Ollama on the GPU. The batch_vectorize_job performs vectorization of text chunks and depends on Qdrant to store vector embeddings for retrieval-augmented generation.

The /api/v1/discovery endpoints include the consolidate endpoint.

Docker Deployment integrates with the Coolify Application deployment system. Coolify Application incorporates PostgreSQL for metadata storage and authentication. Coolify Application depends on the Backend for API functionality. Coolify Application depends on the Frontend for user interface.

Coolify Integration includes deployment for datalens.exerun.com domain. Coolify Integration includes deployment for api.datalens.exerun.com domain. The production deployment on theo uses the Coolify platform for container orchestration and deployment. The production deployment on theo uses PostgreSQL 16 for metadata storage. The production deployment on theo uses Redis 7 as job queue infrastructure. The Production Deployment on theo includes running the Backend container. The Production Deployment on theo includes running the PostgreSQL container. The deployment on theo lacks local GPU features, which constrains the availability of AI features like DS-STAR and Ollama inference. DataLens Project integrates with Coolify for deployment management and environment configuration.

DataLens Platform deployment depends on Coolify server for production hosting. Coolify server integrates with Ollama running on elin server at port 11434. Coolify server integrates with Qdrant running on elin server at port 6333. Coolify server uses DS-STAR agents located at /home/ops/datalens/agents/ on elin server. Coolify server deployment method references Git repository for code or uploads files directly.

Contains PostgreSQL URL for authentication and project tracking. Backend API requires database connection string configured in DATABASE_URL environment variable. Backend API requires database connection string configured in DATABASE_URL environment variable.

DataLens uses PostgreSQL, DuckDB, Ollama (qwen3-coder-next model), and Qdrant for semantic search, supporting hybrid methods. It handles cross-file queries with standard schemas, is scalable, multi-tenant, privacy-focused, and primarily local, with document extraction and autonomous AI cataloging via DS-STAR and Text-to-SQL endpoint, integrated with file analysis agents. DataLens implements the batch upload pipeline capability. DataLens implements smart processing as part of its data handling capabilities. DataLens provides the unified ask interface for natural language question routing over structured and semantic data.

The Discovery API router integrates with the DataLens backend system. Coolify auto-deploy integrates with the DataLens backend to deploy code changes automatically. IronClaw Agent backend endpoints are integrated within the DataLens backend API services. The DataLens backend depends on SQL migration files to create and update necessary database tables, including those for agent features. The DataLens platform backend uses the DS-STAR pipeline for various processing tasks. The DataLens platform backend uses Text-to-SQL capabilities. The DataLens platform backend uses Document RAG functionalities. The DataLens platform backend contains 11 API endpoints. The DataLens platform backend includes the schema.sql database schema. The DataLens platform backend uses a multi-org data model. The DataLens platform backend integrates the brainstorming skill. The DataLens platform backend integrates the shadcn-ui skill. The DataLens platform backend integrates the ui-audit skill. The DataLens platform backend integrates the test-engineering skill. The DataLens platform backend integrates the docker-essentials skill. The DataLens platform backend integrates the coolify skill for deployment. The DataLens platform backend includes an Agent config. INFRASTRUCTURE.md reserves port 8002 for DataLens backend.

Coolify Integration includes deployment for datalens.exerun.com domain.