Varnamala Builds Auditable Document Intelligence with

Company Name

Varnamala AI

Overview

Varnamala AI is an enterprise-grade multi-tenant document management and artificial intelligence platform designed to transform unstructured, mixed-script document corpora into verifiable, natural-language insights. By combining hybrid retrieval strategies, advanced vision and language models via Amazon Bedrock, and a ledger-based arithmetic engine, Varnamala ensures every extracted figure and query response remains fully auditable and accurate to its source document.

The Challenge

Modern organizations handle vast repositories of unstructured documents spanning invoices, contracts, receipts, and operational records. While large language models (LLMs) excel at natural-language text processing, using standard LLM architectures directly over complex business records introduces fundamental operational risks:

• Arithmetic Hallucination & Auditability Gaps: Conventional LLM document QA systems attempt to extract text and calculate totals simultaneously in a single generative step. This frequently introduces math hallucinations, incorrect aggregations, and zero provenance tracking for numbers – making standard approaches unreliable for financial, auditing, or operational workflows.

• Mixed-Script Processing Limitations: Real-world document corpora in multilingual regions frequently mix Latin and non-Latin scripts (such as Devanagari) alongside non-Gregorian calendar formats (such as Bikram Sambat). Single-pipeline Optical Character Recognition (OCR) systems optimized for one writing system systematically fail or return silent extraction errors when encountering multi-script pages.

• Trust & Enterprise Readiness: Left unaddressed, organizations are forced to limit document AI adoption to low-stakes text summarizing, preventing the deployment of automated intelligence engines across mission-critical financial and operational records.

Proposed Architecture & Technical Approach

Varnamala addresses these challenges by decoupling text extraction, structured schema modeling, deterministic arithmetic calculations, and generative response synthesis into distinct, specialized layers.

1. Dual Ingestion Pipelines

To handle diverse document scripts effectively, the platform implements dual parallel extraction pathways:

• Text-Centric Pathway: Utilizes Amazon Textract paired with generative text assembly models for high-speed, structured parsing of Latin-script documents.

• Vision-Model Pathway: Routes multi-script and Devanagari-centric pages directly to multi-modal vision models capable of accurately recognizing intricate character layouts, varied fonts, and regional calendar formats.

2. Centralized Model Orchestration & Observability

All generative model calls across extraction, classification, query parsing, and synthesis route through a unified, OpenAI-compatible proxy gateway and tracing server. Amazon Bedrock (utilizing Nova models) powers four core generative tasks:

• Schema Assembly: Parsing raw document text and visual elements into structured JSON objects.

• Document Classification: Categorizing incoming documents to apply domain-specific processing rules.

• Validated Query Planning: Translating natural language user questions into structured, typed database queries, completely eliminating SQL-injection vulnerabilities.

• Answer Synthesis: Combining retrieved context and exact calculated values into fluent, professional responses.

3. Hybrid Retrieval Engine

The platform’s search infrastructure combines dense semantic vector search with exact lexical keyword search built directly on PostgreSQL:

• Dense Retrieval: Uses high-dimensional vector search powered by scalable DiskANN indexing.

• Lexical Retrieval: Uses full-text search with BM25 scoring for exact keyword, reference number, and proper noun matching.

• Fusion & Reranking: Results are merged using Reciprocal Rank Fusion (RRF) followed by a secondary cross-encoder reranking pass to ensure optimal context retrieval prior to answer synthesis.

4. Deterministic Ledger & Arithmetic Guarantee

To eliminate math errors, all numeric calculations (such as invoice subtotals, tax aggregations, or date differences) are strictly stripped from the generative model’s responsibilities. Instead:

• Numeric operations are executed deterministically using Python Decimal data types or precision database queries.

• Every calculated number is logged in an immutable audit ledger containing its origin document, page, and exact coordinate reference.

• During synthesis, the system automatically checks every number present in the generated output against the ledger, flagging any unverified figure before delivering the response to the user.

Production Deployment & Operational Impact

Varnamala is fully deployed in production as a tenant-isolated platform. Key capabilities operating in production today include:

• End-to-End Automated Ingestion: Multi-stage document parsing, schema enforcement, and entity extraction workflows.

• Human-in-the-Loop Review: Dedicated validation interfaces for review and override of auto-extracted schema fields.

• Enterprise Multi-Tenancy: Secure data segregation, role-based access control, workspace sharing, and collaborative thread management.

• Traceable Conversational Search: Verified, natural-language document querying with explicit citations linking back to original source materials.

Core Quality & Reliability Metrics

• Ledger Audit Pass Rate: Measures the proportion of generated responses where 100% of numeric values trace back to an exact ledger entry without unexplained figures. Targeted and maintained at near-absolute completeness.

• Extraction Path Correctness: Tracks the accuracy of routing document pages to the appropriate processing pipeline (Textract vs. Vision) based on script content, ensuring zero loss of readability across multilingual corpora.

Future Roadmap & Technical Continuous Improvements

As part of ongoing system optimization and active engineering development, key areas of advancement include:

• Dynamic Dynamic Script-Level Routing: Transitioning from global tenant-level processing configurations to per-page automated script detection. This will allow every page within a multi-page mixed-script PDF to be dynamically routed to its optimal extraction pipeline.

• Unified Observability & Telemetry: Enhancing internal tracing tools to aggregate API consumption, model latency, and background parsing jobs into a single unified telemetry dashboard across all pipeline stages.

• Microservices Decoupling for Heavy Workflows: Isolating computer vision tasks (such as page orientation detection and image preprocessing) into standalone lightweight microservices, optimizing core system performance during large bulk-ingestion jobs.