Project Name
Batch News Analytics Replaced With Apache Druid, Trending Dashboards From Hours to 5 Seconds
A digital news and media publisher operating multiple content properties, generating tens of millions of pageviews, article views, and reader-engagement events every day, had editorial and business teams making trending-content decisions on batch-oriented reporting pipelines that refreshed only once every few hours. By the time a trending story surfaced in a report, its traffic peak had often already passed, and editorial decisions on story placement and push notifications were being made on stale information. Applying its AI-First approach, Ksolves delivered an Apache Druid-based real-time analytics platform capable of ingesting streaming pageview and engagement events, surfacing sub-second trending dashboards, and retaining years of historical readership data at a fraction of the storage cost of the previous system.
- Stale Trending Data: Editorial dashboards refreshed on a multi-hour batch cycle, meaning trending-content decisions were consistently made on data that was already outdated by the time it was actionable.
- Explosive Event Volume: Pageview, scroll-depth, and engagement events were being generated at a scale that overwhelmed the existing row-oriented reporting database, causing dashboard queries to take tens of seconds even for simple time-window aggregations.
- Unbounded Storage Growth: Every individual pageview event was stored at full granularity indefinitely, causing storage costs to grow linearly with traffic, even though most historical detail was never queried again.
- No Prioritisation Between Workloads: Ad-hoc analyst queries against historical archives could monopolise the same database resources as live editorial dashboards, causing trending-story dashboards to slow down whenever a data scientist ran a heavy historical query.
- Choosing the Right Rollup Strategy Without Losing Useful Granularity: Aggregating raw pageview events too aggressively at ingestion time would have permanently discarded the ability to analyse individual reading sessions, while rolling up too conservatively would have defeated the storage savings the platform needed.
- Designing Segment Granularity for a Continuously Streaming Workload: Because events arrived continuously rather than in daily batches, the team had to carefully choose a segment time-partitioning scheme that balanced query-pruning efficiency against an excessive proliferation of very small segment files.
- Preventing Cold Archives from Consuming Premium Infrastructure: With years of historical data accumulating, the team needed a way to physically separate expensive, high-performance hardware from the vast majority of data that was queried only occasionally, without manually managing where each day's data lived.
- Ensuring Cache Correctness Against a Live, Continuously Updating Stream: Because new events were constantly landing for the most recent time windows, naive result caching risked serving editors stale trending numbers for data that was still actively changing.
- Preventing Heavy Analyst Queries from Starving Editorial Dashboards: A single data scientist running a multi-month historical comparison query could consume enough concurrent query capacity to visibly slow down the live trending dashboard used by the entire newsroom.
Ksolves implemented Apache Druid as the unified real-time OLAP engine, replacing the batch-oriented row database entirely. The governing principle: every architectural decision - rollup strategy, segment granularity, storage tiering, caching, and query lane design - was made to guarantee that editorial dashboards always get sub-second responses, regardless of what else is running on the cluster.
- Real-Time Streaming Ingestion via Kafka Indexing Service: Deployed Druid's native Kafka indexing service with tuned minute-level rollup and hourly segment granularity, making new events queryable within seconds while collapsing millions of raw events into a dramatically smaller, dashboard-relevant footprint.
- Hot / Warm / Cold Tiered Historical Servers With Automated Load Rules: Three-tier historical server topology: a high-performance hot tier for the last 48 hours where nearly all live-dashboard traffic landed, a warm tier for the past 90 days used in weekly and monthly reporting, and a cost-optimised cold tier for multi-year archives. Coordinator load rules automatically migrate segments between tiers purely based on age with zero manual intervention.
- Broker-Level Result Caching Scoped to Immutable Time Windows: Enabled Broker result caching relying on Druid's segment-versioning, ensuring only completed, immutable segments were cached while the actively-ingesting most-recent time window was always recomputed fresh - delivering speed on repeated dashboard refreshes without sacrificing correctness on the live-updating edge of the data.
- Query Lanes Separating Editorial Dashboards from Analyst Workloads: Configured dedicated query lanes with independent concurrency limits: a high-priority lane reserved exclusively for live trending dashboards, and a separate capped lane for ad-hoc analyst queries, ensuring a heavy historical query could never block or slow down a newsroom editor's live view.
- Approximate Sketches for Unique-Reader Metrics: Adopted HLL sketches for fast approximate unique-visitor counts per article and Theta sketches for cross-article audience-overlap analysis, avoiding the cost of exact distinct-count computation across a continuously growing event volume.
Technology Stack
| Category | Technology |
|---|---|
| Streaming | Apache Kafka + Druid Kafka Indexing Service |
| Analytics Engine | Apache Druid |
| Storage Tiering | Hot / Warm / Cold Tiered Historical Servers |
| Caching | Broker-Level Result Caching (Segment-Versioned) |
| Query Isolation | Dedicated Query Lanes (Editorial vs Analyst) |
| Sketches | HLL and Theta Sketches |
- Dashboard Visibility From Hours to Under 5 Seconds: Druid's Kafka Indexing Service makes new pageview and engagement events queryable within seconds. Editorial trending dashboards that previously refreshed on a multi-hour batch cycle now reflect events within 5 seconds.
- Sub-Second Trending Dashboard Queries at Peak Traffic: Apache Druid delivers sub-second query responses on trending-story aggregations across tens of millions of daily events, even during peak breaking-news traffic spikes.
- Significant Storage Reduction via Minute-Level Rollup: Minute-level rollup at ingestion time collapses millions of individual raw pageview events into compact summary records. Storage cost reduced significantly versus full-granularity indefinite retention of the legacy system.
- 3-Tier Automated Storage With Zero Manual Intervention: Hot, warm, and cold tiered historical server topology fully automated via Coordinator load rules. Segments migrate between tiers purely based on age with zero manual intervention.
- Editorial Dashboard Performance Fully Isolated from Analyst Load: Dedicated query lanes ensure live editorial trending dashboards always get fast responses regardless of how many heavy historical or ad-hoc analytical queries are running concurrently.
- Accurate Unique-Reader Metrics at Minimal Compute Cost: HLL and Theta sketches deliver fast, accurate approximate unique-visitor and audience-overlap metrics without the prohibitive cost of exact distinct-count computation across a continuously growing event stream.
“By the time our editorial dashboards showed a trending story, the traffic peak had already passed. Our editors were making placement decisions on information that was hours old. Druid changed that entirely – trending dashboards now reflect what is happening right now, queries return in under a second, and our data scientists no longer slow down the newsroom every time they run a historical analysis.”
– Head of Data Engineering / VP Editorial Technology.
A digital news publisher making editorial trending decisions on batch dashboards that were hours out of date, suffering from explosive event volume overwhelming its row-oriented database, unbounded storage growth, and analyst queries starving editorial workloads, was transformed through Ksolves Big Data services. Apache Druid with Kafka streaming ingestion, 3-tier automated storage, segment-versioned result caching, dedicated query lanes, and approximate sketches now delivers sub-second trending queries within 5 seconds of event occurrence. Storage costs reduced significantly. Editorial dashboards fully isolated from analyst load. Historical tier migration fully automated. Zero manual intervention required.
Still Making Editorial Calls on Data That Is Hours Old?
