DataLens needs to implement SSE streaming to provide streaming UI with progress updates and partial results for better user experience. The SSE progress endpoint implements real-time streaming via Redis pub/sub mechanisms. DataLens requires SSE streaming to provide streaming UI responses like Vanna

GPU-first document extraction integrates with Docling on elin GPU via SSH and SCP for remote execution and file transfer.

DataLens Development has added Standard Models as logical data entities for normalized cross-file queries and AI mapping.

DataLens Development implements Standard Schemas to enable cross-file data analysis via AI-powered schema mapping. The Standard Schemas capability requires the SchemaMapper Service for AI-powered column mapping.

Platform supports 5 standard schemas (Salary, Health, Financial, Budget, Geographic) with optional mappings. Infrastructure for storing mappings and applying them during extraction is being developed to enable cross-file analysis and improve AI-assisted schema detection.

StandardBudgetRecord data entity is used for modeling government budget and spending data in projects. StandardBudgetRecord data entity relates to StandardFinancialTransaction data entity as both involve financial data management.

StandardFinancialTransaction data entity represents financial transactions relevant to projects. StandardSalaryRecord and StandardFinancialTransaction data entities are related as they both model financial compensation and transactions data. StandardBudgetRecord data entity relates to StandardFinancialTransaction data entity as both involve financial data management.

StandardGeographicData data entity models geographic data used in project contexts. StandardHealthRecord data entity uses StandardGeographicData data entity fields for location-related health data analysis.

StandardHealthRecord data entity represents health-related data used in context of projects. StandardHealthRecord data entity uses StandardGeographicData data entity fields for location-related health data analysis.

StandardSalaryRecord entity is a data model for salary and compensation data likely used in context of projects. StandardSalaryRecord and StandardFinancialTransaction data entities are related as they both model financial compensation and transactions data.

Interactive UI allowing users to define, adjust, and test schema consolidation strategies for targeted queries.

The DataLens Platform stores project data and files in the storage directory at /home/ops/datalens-platform/storage/.

Backend API requires mounting persistent storage volumes for uploads and DuckDB files.

Configured as /app/storage for local data storage. Backend API requires a file upload directory configured by STORAGE_ROOT environment variable. Backend API requires a file upload directory configured by STORAGE_ROOT environment variable.

GdprDetector uses StorageService to scan project data for GDPR-related personal data indicators. SchemaMapper uses StorageService to manage file storage when mapping uploaded file columns to standard schemas. GdprDetector uses StorageService to scan project data for GDPR-relevant personal data indicators. StorageService handles file storage and retrieval for FilePrioritizer's prioritization process.

Svelte store managing current language state and persistence across user sessions.

Vanna 2.0 provides streaming server-sent events responses with progress updates and structured UI components. Timeout issues with long-running queries are mitigated by streaming responses via the /ask-stream endpoint to prevent HTTP client timeouts. The streaming responses via the /ask-stream endpoint realize the Text-to-SQL analysis query endpoint by enabling streaming of query results to avoid client timeouts. The streaming responses via /ask-stream endpoint are implemented using the FastAPI Framework to provide asynchronous request handling. Timeout issues with long-running queries are mitigated by streaming responses via the /ask-stream endpoint to prevent HTTP client timeouts. The streaming responses via the /ask-stream endpoint realize the Text-to-SQL analysis query endpoint by enabling streaming of query results to avoid client timeouts. The streaming responses via /ask-stream endpoint are implemented using the FastAPI Framework to provide asynchronous request handling. Vanna 2.0 provides streaming UI components to stream structured response objects like tables and charts. Vanna 2.0 provides streaming UI components to stream structured response objects like tables and charts. The SSE progress endpoint integrates with Redis pub/sub for real-time progress streaming.

Vanna 2.0 architecture supports streaming UI components with progress updates and structured data

StreamingResponse wraps the agent.py event_generator() function to stream data asynchronously to the client.

DataLens supports structured search via Text-to-SQL queries on DuckDB.

A process requirement that now uses parallel subprocesses for real-time GPU-based extraction, replacing fallback methods.

The Backend integrates with Supabase as its project management system.

The DiscoveryFlow component is implemented using the Svelte framework. The npm dev dependency @sveltejs/adapter-auto depends on svelte in the frontend project. The npm dev dependency @sveltejs/adapter-node depends on svelte in the frontend project. The npm dev dependency shadcn-svelte depends on svelte in the frontend project dependencies.

The Frontend is implemented using Svelte 5 framework compatible with production deployment.

The BulkUpload component uses Svelte 5 runes for its frontend implementation.

The Data Discovery feature includes multiple Svelte components.