Project Name
StarRocks and Iceberg Lakehouse Cut Infrastructure Costs 68% for a Pan-India Logistics Operator
A leading Indian pan-national logistics conglomerate operating 1,200+ fulfilment centres across 26 states, processing 3M+ shipments daily for India’s largest eCommerce, FMCG, and retail enterprises, ran Elasticsearch for real-time shipment tracking and Oracle OLAP for finance reporting. Nightly batch reconciliation between these two siloed systems created 18- to 24-hour data latency. 72% of Elasticsearch cluster capacity was consumed by data queried less than twice per month. The stack could not support the 4x volume surge projected for the next 18 months. Applying its AI-First approach, Ksolves built a StarRocks and Apache Iceberg Lakehouse on MinIO, eliminating both legacy systems and reducing infrastructure spend by 68%.
- No Unified View Across 3M Daily Shipments: Elasticsearch and Oracle OLAP had no shared query layer. Any cross-functional business question triggered multi-day manual reconciliation across two teams.
- 18 to 24 Hour Finance Reporting Lag: Regional cost, courier performance, and SLA reports ran on nightly batch cycles. Finance had no intraday visibility into Rs800Cr+ daily delivery cost exposure.
- Infrastructure Costs Scaling Faster Than Revenue: 72% of Elasticsearch cluster capacity consumed by data queried less than twice per month. Rs2.4Cr/month in wasted infrastructure spend.
- 8 to 14 Minute Historical Queries: 5.8 PB of shipment data across Elasticsearch and Oracle with no partition strategy. Full cluster scans for every historical report.
- Schema Changes Paralysing Operations: Adding a new last-mile event type required 72-hour freeze windows across Elasticsearch and Oracle DDL migrations, blocking 40+ dependent pipelines.
- 11-Hour Compliance Audit Cycles: TRAI and MCA mandated point-in-time queries for any 30-day window within 5 years. The existing stack required full backup restore averaging 11 hours per request.
- Flink CDC Causing P1 Incidents: Concurrent Flink CDC upserts to Oracle OLAP caused dirty reads during analyst queries. Three P1 incidents in 6 months from non-ACID writes on live reporting tables.
Ksolves designed a hot-cold lakehouse using StarRocks as the real-time hot tier and Apache Iceberg on MinIO as the governed cold tier. The governing principle: one SQL surface for everything. Ops dashboards at 0.04 seconds, finance UNION ALL queries at 0.34 seconds, compliance time travel under 2 seconds, and Spark ML workloads on the same Iceberg tables with no ETL duplication.
- StarRocks Native Hot Tier: PRIMARY KEY table ingesting 3M+ events/day via Routine Load from Kafka. Ops queries at 0.04 seconds. 28-state regional rollups under 0.3 seconds. Elasticsearch replaced entirely.
- Apache Iceberg Cold Tier on MinIO: 5.8 PB stored as Parquet partitioned by (dt, region). Partition pruning drops 8 to 14 minute scans to under 2 seconds. ACID transactions end dirty-read P1 incidents.
- Nightly Hot-to-Cold Offload: Automated pipeline moves 2.4M rows/night from StarRocks to Iceberg. Hot tier stays lean. Cold tier grows at Rs18/GB/month versus Rs280/GB on Elasticsearch.
- Iceberg Time Travel for Compliance: FOR VERSION AS OF returns any 30-day historical window in under 2 seconds. 11-hour backup restore cycles eliminated. Full 5-year audit trail maintained automatically.
- Schema Evolution With Zero Downtime: ALTER TABLE ADD COLUMN completes in 0.17 seconds. New event types go live without freeze windows. All 40+ dependent pipelines continue uninterrupted.
Technolgy Stack
| Category | Technology |
|---|---|
| Hot Tier | StarRocks Native Tables |
| Cold Tier | Apache Iceberg on MinIO |
| Ingestion | Apache Kafka + Flink CDC + Routine Load |
| Catalog | Iceberg REST Catalog |
| Compute | Apache Spark + StarRocks FE |
- Infrastructure Costs Cut 68%: MinIO cold tier at Rs18/GB/month versus Rs280/GB on Elasticsearch. Rs2.4Cr/month infrastructure saving from moving cold data to the correct storage tier.
- Historical Queries From 8 Minutes to 2 Seconds: Iceberg partition pruning eliminates full cluster scans. 8 to 14 minute historical reports complete in under 2 seconds across the full 5.8 PB dataset.
- Ops Dashboards at 0.04 Seconds Across 28 States: StarRocks delivers delayed-shipment counts, SLA alerts, and courier leaderboards for 800+ ops managers at 3M events/day.
- Finance Reporting Intraday: UNION ALL across StarRocks hot and Iceberg cold returns in 0.34 seconds. Rs800Cr+ daily delivery cost exposure visible intraday for the first time.
- Compliance Audit From 11 Hours to 2 Seconds: Iceberg time travel returns any 30-day historical window in under 2 seconds. Full 5-year TRAI and MCA audit trail maintained automatically.
- Schema Evolution in 0.17 Seconds: ALTER TABLE ADD COLUMN on Iceberg completes in 0.17 seconds. New event types live without freeze windows. All 40+ pipelines uninterrupted.
“We were spending Rs2.4Cr per month storing data nobody was querying, while our finance team waited overnight for reports that are now available in seconds. StarRocks and Iceberg gave us real-time and historical analytics on one platform – and we are paying 68% less than before.”
– Chief Data Officer / VP Engineering.
A pan-India logistics operator processing 3M+ daily shipments, paying Rs2.4Cr/month for wasted Elasticsearch capacity, and waiting 8 to 14 minutes for historical reports was transformed through Ksolves big data services. StarRocks and Apache Iceberg on MinIO now deliver one unified lakehouse. Costs down 68%. Historical queries in 2 seconds. Ops dashboards at 0.04 seconds. Finance reporting intraday. Compliance audits in 2 seconds. Schema changes in 0.17 seconds. 5.8 PB on open storage. 800+ users on one SQL layer.
Is Your Logistics Data Platform Paying Premium Rates for Cold Data While Ops and Finance Wait Hours for Reports?
