Background workers include the catalog worker as a component. Background workers include the extract worker as a component. Background workers include the vectorize worker as a component. Background workers include the prioritize worker as a component. Background workers use RQ job chaining for job chaining and orchestration. The DOCX extractor relies on background workers for asynchronous processing. The PPTX extractor operates via background workers to perform extraction tasks. MSG extractor uses background workers to process message files. Background workers utilize RQ job chaining to coordinate sequential processing tasks.

The Frontend uses Badge.svelte UI component.

Abstract base client in backend/app/services/ironclaw_client.py for communicating with IronClaw API.

The bash script tests/test-discovery.sh is used to run the E2E test suite.

Performance optimization to process 50 chunks per GPU request, recently implemented.

A large-scale GPU-first extraction process targeting 214 files, expected to complete in about 15-20 minutes, using Docling and Ollama for document parsing and embeddings. The batch processor orchestrator manages background workers including catalog, extract, vectorize, and prioritize jobs. The file prioritizer component is utilized by the batch processor orchestrator to assign tiers and manage processing order. Batch extraction subprocess is part of Backend batch processing strategy using subprocess calls instead of RQ for extraction jobs.

Batch Upload uses the Batch Processing Strategy for extraction of files. Batch Processing Strategy uses RQ job queue for job management and reliability.

Batch upload is part of the enhanced AI cataloging process in DataLens. The Phase 2 MVP includes the requirement for Batch Upload of all 150 SVGV budget files. The Batch Upload process uses the RQ job queue for reliable background job execution. Batch Upload uses the Batch Processing Strategy for extraction of files.

The Batch upload API endpoint provides the backend interface for the batch upload pipeline. The batch upload API endpoint is used to receive batch uploads for processing by the batch upload pipeline.

The batch upload pipeline with smart processing is implemented as a feature within DataLens, supporting phases like AI cataloging and file processing. It utilizes new extractors (DOCX, PPTX, MSG) and backend services for orchestrating uploads, extraction, and vectorization, integrated into a comprehensive UX model.

The batch upload pipeline workflow is validated by the batch-pipeline-workflow.spec.ts end-to-end frontend test case.

The extraction worker chains to the batch vectorize job to process GPU embeddings after extraction. Extracted text chunks from DOCX and PPTX files are vectorized in batch and indexed into Qdrant for semantic search. batch_vectorize_job depends on EmbeddingService to perform batch vectorization of extracted document chunks. The batch_vectorize_job processes text chunks to generate embeddings using Ollama on the GPU. The batch_vectorize_job performs vectorization of text chunks and depends on Qdrant to store vector embeddings for retrieval-augmented generation.

BatchProcessor uses FilePrioritizer to prioritize project files in processing pipeline orchestration. BatchProcessor uses ExtractionCoordinator to coordinate data extraction in the pipeline. BatchProcessor uses EmbeddingService to vectorize extracted data in the processing pipeline. DSStarService depends on BatchProcessor to orchestrate DS-STAR Agent API workflows. WorkflowService uses BatchProcessor to orchestrate analysis workflows and data processing pipelines. The File prioritizer is used by the Batch processor orchestrator in the processing pipeline. The Batch processor orchestrator uses the Question router to classify questions and route queries. BatchProcessor orchestrates the full pipeline that involves ExtractionCoordinator for extraction tasks. WorkflowService orchestrates pipelines that may be started and monitored by BatchProcessor. The batch processor orchestrator depends on the catalog worker as the first step in the batch processing pipeline. The batch processor orchestrator uses the prioritize worker to assign processing tiers. The batch processor orchestrator relies on the extract worker to perform data extraction from files. The batch processor orchestrator depends on the vectorize worker to generate embeddings for processed chunks. The batch processing pipeline orchestration is validated by the test_progress_status test case.

Passlib depends on bcrypt for password hashing.

The DataLens Platform uses bearer token authentication for security and session management. Bearer token authentication is implemented in the platform using Authorization Bearer tokens standard for session management. The Auth system uses Bearer tokens for simpler authentication management than JWT tokens. The Platform Backend implements bearer token authentication for session management and user access.

The DataLens Platform authenticates users using Bearer tokens instead of JWT.

DataLens requires lifecycle hooks like BeforeQuery and AfterQuery to support quota checking, cost tracking, and content filtering. DataLens requires BeforeQuery and AfterQuery lifecycle hooks for quota checking and cost tracking

DataLens requires adding BeforeQuery hooks to intercept request processing for quota checking etc.

Document RAG uses the embedding model bge-large-en-v1.5 for semantic embedding of documents. LlamaIndex generates embeddings using bge-large-en-v1.5 model. LlamaIndex generates embeddings using bge-large-en-v1.5 model.

The npm dev dependency @playwright/test uses the bits-ui component as part of the frontend dependency set in the project.

The DataLens platform backend uses brainstorming as part of its skills. The DataLens platform backend includes the brainstorming skill. The DataLens platform backend integrates the brainstorming skill.

Bridge Consulting worked as a consultant on the SVGV Budget Analysis Project.

In scope as a core functionality for data analysis; implementation pending.

Budget Analysis Cluster requires the Consolidation Mechanism to combine related budget tables for comprehensive queries.

Budget Analyzer is a DS-STAR agent specialized in financial and budget analysis queries. Agent Selector directs queries to the Budget Analyzer agent