Project Name
HDP to Apache Bigtop Migration with Disaster Recovery Setup for a Major ISP in South India
A major internet service provider operating across 21 cities in South India had built its operational analytics and revenue assurance capabilities on a 50+ node Hortonworks HDP 2.6.3 cluster hosting between 180 and 190 terabytes of live data. When HDP reached end of life, and vendor security patches stopped entirely, the organization faced a compounding risk: a business-critical platform with no upgrade path, no vulnerability protection, and a single data center footprint offering zero failover capability.
With operational reporting and revenue assurance running continuously on the cluster, any migration approach requiring downtime was commercially unacceptable. The ISP simultaneously needed to implement a disaster recovery site across its Bangalore and Hyderabad data centers for the first time. The organization partnered with Ksolves, an AI-First Company, to design and execute a zero-downtime platform migration to Apache Bigtop while standing up cross-site DR in parallel.
The challenges faced by the client are as follows:
- End-of-Life Platform Risk: Hortonworks HDP 2.6.3 had reached end of support with no security patches, no zero-day vulnerability responses, and no upgrade path available from the original vendor, leaving 180 to 190 TB of live operational data exposed to compounding security and compliance risk.
- Zero-Downtime Migration Mandate: The ISP's operational analytics and revenue reporting ran continuously on the cluster. Any migration approach that required scheduled downtime or a maintenance window was commercially unacceptable and operationally unfeasible.
- No Disaster Recovery Environment: The entire cluster existed in a single Bangalore data center with no DR site, no replication, and no failover capability, representing a complete single-point-of-failure risk for a business-critical platform.
- Component Compatibility Complexity: The migration required validating and reconfiguring all integrated components against the Apache Bigtop compatibility matrix, with no tolerance for data loss or pipeline interruption during the transition.
- Hardware-Dependent Cut-Over Timeline: New nodes had been ordered but were not yet delivered at engagement start, requiring the migration plan to be designed around a blue-green cut-over approach contingent on hardware arrival timelines.
- Lack of Open-Source Stack Expertise: The client's team had operated HDP under vendor-managed support and had no prior experience running a self-managed open-source Hadoop stack, making knowledge transfer a core delivery requirement alongside the technical migration.
Ksolves, an AI-First Company, designed and executed a blue-green migration strategy, standing up the new Apache Bigtop cluster on incoming hardware in parallel with the live HDP cluster, migrating data progressively, and cutting over without a single hour of downtime. Ksolves leveraged its proprietary open-source tooling for automated upgrades and DCDR (Distributed Cluster Disaster Recovery) management to orchestrate the DR site configuration across Bangalore and Hyderabad simultaneously with the primary migration.
- Apache Bigtop Cluster Setup and Configuration: Ksolves provisioned the new Apache Bigtop cluster on incoming hardware nodes, configuring HDFS, YARN, Hive, Spark, and ZooKeeper with validated compatibility settings and enterprise-grade security hardening, replacing the entire HDP stack without disrupting the live cluster.
- Blue-Green Zero-Downtime Data Migration: A progressive blue-green migration approach was executed, running old and new clusters in parallel, migrating workloads and data in sequenced batches, and performing a validated cut-over once the Bigtop environment was fully verified, preserving 100% data integrity across 200 TB.
- Disaster Recovery Setup Across Two Data Centers: Ksolves designed and implemented DR across Bangalore (primary) and Hyderabad (secondary) data centers using its proprietary DCDR management tooling, giving the ISP cross-site replication and failover capability for the first time.
- Apache Ambari for Cluster Management: Apache Ambari was configured as the cluster management and monitoring layer, providing the operations team with a unified interface for service management, performance monitoring, and configuration governance on the new open-source stack.
- Knowledge Transfer and Open-Source Operations Enablement: Structured knowledge transfer was delivered covering Bigtop cluster management, Ambari operations, ZooKeeper maintenance, and performance tuning, transitioning the client from vendor-managed HDP to a fully self-managed open-source platform.
Technology Stack
| Layer | Technology |
|---|---|
| Source Platform | Hortonworks HDP 2.6.3 |
| Target Platform | Apache Bigtop |
| Cluster Management | Apache Ambari |
| Storage and Resource Management | HDFS + Apache YARN |
| Processing and Analytics | Apache Spark + Apache Hive |
| Coordination and DR | Apache ZooKeeper + DCDR |
- 200 TB Migrated With Zero Data Loss and Zero Downtime: CDR and operational data previously processed on an unsupported HDP cluster with no migration path were fully transitioned to Apache Bigtop. The complete blue-green migration was executed across 180 to 190 TB of active data with zero data loss, zero unplanned downtime, and full component compatibility validated across Hive, Spark, YARN, and ZooKeeper.
- End-of-Life Risk Eliminated Across the Entire Platform: HDP 2.6.3 had been running without vendor security patches, zero-day vulnerability protection, or upgrade support, creating compounding security and compliance exposure across the business. The platform is now fully migrated to a supported, community-maintained Apache Bigtop stack with a clear open-source upgrade path and zero ongoing vendor dependency or licensing cost.
- Disaster Recovery Implemented for the First Time: The ISP had operated with a single Bangalore data center, no DR site, no replication, and no failover capability, representing a complete single point of failure for a business-critical platform. A cross-site DR architecture is now live across Bangalore and Hyderabad, providing DCDR-managed replication and failover capability that eliminates the previous single-site risk entirely.
- Internal Team Transitioned to Self-Managed Open-Source Operations: The client's team had operated exclusively under vendor-managed HDP support with no experience running a self-managed open-source stack. Structured knowledge transfer across Bigtop cluster management, Ambari operations, ZooKeeper maintenance, and performance tuning has fully qualified the team to operate the platform independently post-handoff.
Ksolves transformed a stranded, end-of-life HDP platform into a modern, self-managed Apache Bigtop environment with cross-site disaster recovery operational across two data centers. The ISP moved from a vulnerable, vendor-dependent cluster with no failover capability to a fully migrated open-source platform with a trained operations team and a clear path forward.
Zero data loss and zero downtime were achieved across a 200 TB production migration, the highest-stakes outcome metric for a business-critical ISP data platform. The DCDR-based DR architecture is a repeatable blueprint for any enterprise needing to add cross-site resilience to an existing Hadoop estate, applicable across ISPs, banking, and logistics sectors with distributed data center footprints.
With HDP eliminated and Bigtop operational, the client is now positioned to adopt modern open-source Hadoop ecosystem enhancements, including newer Spark versions, cloud-compatible storage APIs, and advanced security integrations without vendor lock-in or licensing constraints. For enterprises still running on unsupported Hadoop distributions, Ksolves Big Data Support Services provide the migration expertise and platform engineering needed to move safely to a future-ready stack.
Is Your Hadoop Platform Still Running on an End-of-Life Distribution With No Clear Migration Path?
