Project Name
Storage Costs Cut by 40% with ClickHouse Tiered Storage
Our client is a large telecommunications group operating across several Sub-Saharan African markets, providing mobile voice, data, and digital financial services to tens of millions of subscribers. The group runs distinct regional data platform instances, each handling high-volume event streams – call detail records, transaction logs, and network telemetry that must be retained and queryable for regulatory, operational, and revenue assurance purposes.
With data volumes growing year on year and storage budgets under pressure, the organisation sought an architecture that could scale cost-efficiently without sacrificing the analytical capability that revenue assurance and operational dashboards depended on.
Petabytes of event data on a single expensive storage tier, no automated lifecycle, and storage pressure only discovered after it had already caused a service degradation.
- Flat-Tier Storage Explosion: Hot and cold data shared the same high-performance storage, driving infrastructure costs higher as data volumes grew.
- No Automated Data Lifecycle: Data retention relied on manual partition cleanup, creating operational risk, inconsistent policies, and unexpected storage exhaustion.
- Query Performance Risk During Migration: Storage optimisation had to preserve sub-second query performance without disrupting dashboards or revenue assurance workloads.
- Fragmented Multi-Market Deployments: Regional markets used different storage configurations and retention policies, making operations complex and onboarding inconsistent.
- Monitoring Blind Spots: Limited visibility into storage utilisation and no proactive capacity alerts meant disk issues were detected only after service degradation.
- Legacy Platform Migration Debt: Historical data remained locked in a legacy MapR Hive environment, limiting analytics access and delaying migration to ClickHouse.
Ksolves, an AI-first Big Data consulting services company, designed and implemented a ClickHouse-native tiered storage architecture governed by a single principle: every byte of data should live on the cheapest storage tier that can still serve it within SLA. The solution was deployed across a 5-shard, 2-replica 10-node ClickHouse cluster per market, with a standardised storage policy applied consistently across all regions.
- ClickHouse Storage Policy: Implemented a two-tier storage policy with NVMe for hot data and HDD for warm data. A move_factor of 0.3 automatically shifted data at 70% hot-tier utilisation, eliminating manual storage management.
- TTL-Driven Lifecycle Rules: Applied automated TTL rules for data retention and RECOMPRESS policies for intermediate tiers, ensuring the complete data lifecycle was managed without manual intervention.
- SeaTunnel Migration Pipeline: Built a SeaTunnel ETL pipeline to migrate historical data from MapR Hive into ClickHouse, enabling analytics on legacy datasets without rebuilding the query layer.
- ClickHouse Keeper Coordination: Deployed a dedicated three-node ClickHouse Keeper cluster to manage replication, distributed DDL, and data movement across shards with native coordination.
- Prometheus and Grafana Observability: Created comprehensive dashboards to monitor storage utilisation, replication, and capacity across markets, with proactive alerts before storage pressure impacted performance.
- Standardised Runbooks and Knowledge Transfer: Delivered detailed operational runbooks covering storage policies, TTL management, disk expansion, and monitoring, enabling independent client operations after handover.
Technology Stack
| Category | Technology |
|---|---|
| Analytical Database | ClickHouse 24.x+ |
| Stream Ingestion | Apache NiFi 2.4 + Apache Kafka 4.0 |
| ETL Migration | SeaTunnel |
| Coordination | ClickHouse Keeper |
| Observability | Prometheus + Grafana |
| Orchestration | Apache Airflow 3.x |
From a single flat NVMe tier with no lifecycle management and no monitoring to an automated, self-managing tiered architecture that cut storage costs by 40% while maintaining sub-second query performance across every market.
- ~40% Storage Cost Reduction: Automated tiering moved over 60% of data to lower-cost HDD storage, reducing overall storage costs by approximately 40% while maintaining NVMe performance for hot data.
- Sub-Second Query Performance Maintained: Recent data continued to be served from NVMe, preserving sub-second query response times for dashboards and revenue assurance workloads across all markets.
- 100% Automated Data Lifecycle: TTL rules and automated part migration eliminated manual storage management, replacing recurring partition cleanup with a self-managing lifecycle.
- End-to-End Storage Observability: Comprehensive Grafana dashboards provided real-time visibility into storage utilisation, data movement, and capacity forecasts, enabling proactive issue detection.
- Legacy Data Successfully Migrated: Historical MapR Hive datasets were migrated into ClickHouse's warm tier, making previously inaccessible data available for analytics and revenue assurance.
Ksolves transformed the client’s storage architecture from a costly, manually managed environment into an automated, ClickHouse-native tiered storage platform. By intelligently moving data between NVMe and HDD based on access patterns, the solution reduced storage costs by approximately 40% while maintaining sub-second query performance. With automated lifecycle management, proactive monitoring, and standardised deployment across markets, the client gained a scalable, cost-efficient foundation that supports future growth without increasing operational complexity.
Is Your ClickHouse or Big Data Infrastructure Spending More on Storage than it Needs to?
