Project Name
Reactive Incident Detection Replaced with Real-Time APM on Kafka, Spark, and ClickHouse
A high-traffic platform running payments, authentication, order processing, and search had no unified real-time view of system health. Errors and latency spikes were discovered only after customer complaints. Analytics ran on a single node with no replication. Each service logged independently with no queryable cross-service view. Applying its AI-First approach, Ksolves delivered a real-time observability platform that ingests, classifies, stores, and visualises telemetry at scale with fault-tolerant storage and sub-minute system health visibility.
- Reactive Incident Detection: Errors discovered only after customer complaints - operational cost already paid before any internal visibility.
- Fragmented Logs: Each service logged independently with no unified queryable view. Cross-service correlation required manual effort.
- No Historical Trend Visibility: No mechanism to view error-rate or latency trends over time or distinguish spikes from worsening trends.
- Single-Node Storage Risk: Existing analytics storage on a single node with no replication - total data loss risk on any node failure.
- Correct Sharding and Replication Design: 4-node ClickHouse topology required correct sharding for scale and replication for safety without manual per-node table management.
- Pre-Aggregated State Query Correctness: Rollups store mergeable state, not plain numbers. Naive SUM()/AVG() return incorrect results - partial aggregate merging required.
- Secure Transport Without Sacrificing Throughput: SSL + compression combination caused connection failures on wrong driver/protocol. Correct native client required.
- Cross-Container Network Routing: Visualisation containers could not reach the ClickHouse cluster via localhost - explicit container networking configuration required.
- Bounding Storage Growth: Continuous ingestion would grow disk indefinitely without an automated TTL policy.
Ksolves built a three-layer observability platform: Kafka for ingestion, Spark Structured Streaming for in-flight classification, and a sharded, replicated ClickHouse cluster for fault-tolerant storage. The governing principle: classify before you store, replicate before you rely on it, and expire before you overflow.
- Sharded and Replicated ClickHouse Cluster: 4-node cluster (2 shards x 2 replicas) coordinated via ClickHouse Keeper. Distributed table routes queries transparently. 100% data integrity preserved across node restarts.
- Kafka + Spark Structured Streaming: Kafka ingests telemetry. Spark classifies each event's severity (CRITICAL / SLOW / OK) in-flight before writing to ClickHouse - no impact on upstream services.
- AggregatingMergeTree Materialized Views: Summary table updates automatically as new data lands. Pre-computed rollups refresh every 5 seconds with no full-table scans or manual recomputation.
- SSL + LZ4 via Native Protocol Client: ClickHouse native-protocol Python client combines encryption and LZ4 compression reliably without connection failures.
- Date-Based Partitioning and TTL: Fast time-range queries via date partitioning. TTL enforces 30-day automatic data expiry with zero manual deletes.
Technology Stack
| Category | Technology |
|---|---|
| Streaming | Apache Kafka |
| Processing | Apache Spark Structured Streaming |
| Storage | ClickHouse (4-Node Cluster) |
| Query Engine | AggregatingMergeTree Materialized Views |
| Transport | SSL + LZ4 Native Protocol Client |
| Visualization | Grafana |
- Reactive Detection Replaced With Real-Time Visibility: Errors visible within seconds - before customers are impacted. Customer complaints no longer the primary incident detection mechanism.
- 100% Data Integrity Across Node Failures: 2-replica cluster ensures zero data loss on individual node restarts or failures. Single-node risk eliminated.
- 5-Second Aggregation Refresh: Materialized views always current. No full-table scans. Dashboard metrics computed from pre-aggregated state.
- Unified Cross-Service Observability: All services feed one ClickHouse cluster with unified severity classification. Cross-service correlation in seconds.
- 30-Day Automatic Data Retention: TTL enforces expiry automatically. Zero manual delete operations required.
- 3 Independent Layers, One Unified Platform: Streaming, storage, and visualisation independently scalable with zero bytes duplicated by the distributed query layer.
“We were finding out about production problems from customer support tickets. Now our dashboards show us what is happening in real time before anyone notices. The ClickHouse cluster has not lost a single event across restarts and the materialized views mean our metrics are always current without running expensive queries.”
-Head of Engineering / VP Infrastructure.
A high-traffic platform discovering production errors only through customer complaints, with single-node analytics storage and fragmented per-service logs, was transformed through Ksolves Big Data services. A three-layer APM platform on Kafka, Spark, and ClickHouse now classifies, stores, and visualises all telemetry in real time. 100% data integrity. 5-second aggregation refresh. 30-day automatic retention. Zero bytes duplicated. Three independent layers as one unified observability platform.
Still learning about production incidents from customer complaints?
