How Ksolves Built an AI-Powered Natural Language Search Engine Over a 7-Source Data Lakehouse for a UAE Lighting & Distribution Company

Ksolves architected and delivered the full AI Factory end-to-end: a production-grade Data Lakehouse as the governed data foundation, with an LLM-powered natural language search engine on top as the interface for the entire business. The governing design principle was open-source first and cloud-neutral, every component, from Apache NiFi for ingestion to Apache Iceberg for table format to the LLM layer for search, was selected to avoid vendor lock-in while delivering enterprise-grade reliability and security.

• Data Lakehouse Foundation (Apache NiFi + Kafka + AWS S3 + Apache Iceberg):
  Automated ingestion pipelines were built for the initial three sources: Salesforce (CRM records, leads, opportunities), Microsoft Outlook (email threads and attachments), and DoubleClick ERP (CSV uploads). Apache NiFi handled visual flow orchestration; Apache Kafka provided real-time event streaming and buffering; AWS S3 served as the raw and processed object store; Apache Iceberg was deployed as the open table format unifying structured and unstructured data in a single governed Lakehouse, with TLS/SSL encryption in transit and AES-256 at rest.
• Source Expansion to Seven Connected Systems:
  Four additional sources were integrated, expanding the Lakehouse to the full operational footprint. Apache Spark and dbt handled transformation, enrichment, and indexing. Custom ingestion scripts were developed for WhatsApp and email APIs where standard connectors were insufficient.
• AI-Powered Natural Language Search Engine (LLM + RAG + NLP):
  The AI search layer used Large Language Models combined with Retrieval-Augmented Generation to enable contextual natural language queries across all seven sources. Users could ask "Show me the top 5 customers by margin this quarter" or "What were the last 10 interactions with this account?" and receive accurate, sourced answers without writing SQL or navigating multiple systems. The NLP layer handled query understanding, entity resolution, and response generation with citations to source data.
• Smart Quotation Engine: As the first tangible AI use case, a Smart Quotation Engine was delivered:
  integrating product catalogue, pricing history, customer win rates, and pipeline data from the Lakehouse to generate margin-aware quotes automatically, eliminating the pricing inconsistencies that resulted from manual cross-system data assembly.
• React/Node.js Web Application with Secure Access:
  A chat-like web interface was built on ReactJS and Node.js with PostgreSQL for application state. Authentication used username and password with TLS-secured access. A dedicated QA/Dev environment mirrored the production deployment for consistent testing and release management.
• Grafana Monitoring and Pipeline Observability:
  A full observability stack was deployed covering ingestion success rates, data freshness, processing latency, and system health across all seven source connectors, with alerting for pipeline failures or data quality degradation.

• 7 Sources, 98.6% Pipeline Reliability:
  All seven sources (Salesforce, DoubleClick ERP, Odoo, Microsoft Outlook, Teams, WhatsApp, and MailChimp) are now ingested into a single Apache Iceberg Lakehouse on AWS S3, with Grafana-monitored pipelines sustaining a 98.6% ingestion success rate across all connectors. The business has clean, enriched, and traceable data across structured and unstructured content for the first time.
• 63% Reduction in Quotation Cycle Time via the Smart Quotation Engine:
  What previously required a sales rep to pull pricing history, product availability, and customer context from three or four systems manually now takes seconds. The Smart Quotation Engine generates margin-aware quotes directly from unified Lakehouse data, cutting average quotation cycle time by 63% and removing the pricing inconsistencies that came with manual multi-system assembly.
• 74% Reduction in Per-Report Effort Across Sales, Ops, and Marketing:
  Cross-functional reports on sales performance, campaign ROI, and operational KPIs that previously consumed an average of 3–4 hours per cycle in manual extraction and Excel consolidation are now served through the AI search engine in under 10 minutes, reducing per-report effort by 74% and removing analyst dependency for routine business questions.
• Data Access Expanded from 3 Analysts to the Full Business User Base:
  Before deployment, cross-system data queries were limited to the small number of staff with access to all source systems and the SQL knowledge to use them. The conversational search interface now gives every user in sales, operations, and leadership direct query access across all 7 connected systems, with accurate, source-cited answers returned in seconds, no SQL, no IT ticket.
• 100% Historical Data Continuity Across the DoubleClick-to-Odoo ERP Migration:
  With the company mid-migration between ERP systems, the Lakehouse was designed to ingest DoubleClick as the historic source and transition live data flows to Odoo post-migration. The switchover completed with zero Lakehouse downtime and 100% historical data continuity, preserving over 18 months of ERP records without architectural rework.