Project Name
RabbitMQ Pub/Sub Decoupled a Node.js Monolith and Eliminated API Timeouts
An enterprise SaaS platform built on a monolithic Node.js architecture hit a critical bottleneck as demand for heavy data operations grew. Users began requesting massive data exports – CSV and XLSX files with hundreds of thousands of rows – and triggering bulk email campaigns. Because Node.js executes JavaScript on a single thread, these CPU-intensive operations hijacked the event loop. A single 500MB XLSX export could block the entire server, causing completely unrelated API endpoints to time out and degrading the experience for every user simultaneously. Applying its AI-First approach, Ksolves decoupled the heavy workloads from the main API using a RabbitMQ Pub/Sub architecture with dedicated Python worker nodes – eliminating API timeouts, restoring 100% uptime during peak load, and enabling independent horizontal scaling.
- Single-Threaded Event Loop Bottleneck: Node.js executes on a single thread. A user triggering a massive 500MB XLSX export could block the event loop entirely, preventing the server from processing any incoming HTTP requests while the file was being constructed in memory.
- Cascading Failures Across Unrelated Endpoints: While the event loop was blocked, completely unrelated users performing simple actions like logging in or loading a dashboard experienced timeouts. One heavy operation penalised every user on the platform simultaneously.
- Vertical Scaling Ceiling: The monolithic architecture could only scale vertically by provisioning larger servers. This was expensive, had a hard ceiling, and still did not address the fundamental single-threaded blocking problem.
- Peak Load Volatility: End-of-month reporting windows were historically the most volatile periods - the platform was most likely to experience degraded performance precisely when reliability mattered most.
- No Fault Isolation: A crashed or hung export had no blast radius limit. The entire API was affected by any single heavy workload that went wrong, with no isolation between export jobs and the core application.
Ksolves redesigned the flow around a Publish/Subscribe model using RabbitMQ as the message broker. The main Node.js API was converted into a pure traffic controller that validates, queues, and acknowledges, never blocks. All CPU-bound work was delegated to a separate cluster of Python worker nodes optimised for data processing. The governing principle was language for the right job: Node.js for I/O and routing, Python for CPU-bound number crunching.
- Publisher - Node.js API as Traffic Controller: Instead of processing exports inline, the Node.js API validates the request, generates a unique job ID, publishes a message payload to the appropriate RabbitMQ exchange, and immediately returns a 202 Accepted response. The event loop is freed in milliseconds.
- Message Broker - RabbitMQ Queue Routing: RabbitMQ receives published messages and routes them to the correct queue (export-tasks or email-tasks). Messages wait safely until a worker is ready. No message is lost if a worker is busy or temporarily unavailable.
- Subscribers - Dedicated Python Worker Nodes: A separate cluster of Python worker environments subscribes to RabbitMQ queues. Workers pull a message, execute CPU-bound file generation using Pandas and openpyxl, upload the completed file to S3, and trigger a webhook back to notify the user.
- Independent Horizontal Scaling: Node.js nodes and Python worker nodes scale independently. Heavy API traffic scales the Node.js tier. A growing export backlog auto-scales the Python worker tier. No vertical scaling ceiling required.
Technology Stack
| Category | Technology |
|---|---|
| Architecture | Node.js (Publisher API) |
| Infrastructure | RabbitMQ (Message Broker) |
| Processing | Python Workers (Subscribers) |
| Integration | Amazon S3 and Webhook Callbacks |
- Zero API Timeouts: Removing CPU-bound tasks from the main thread stabilised Node.js API response times globally. Unrelated endpoint timeouts dropped completely to zero after decoupling.
- 100% Uptime During Peak Load: During end-of-month reporting periods, the core application maintained 100% uptime. The main API remained responsive regardless of how many large exports were queued.
- True Horizontal Scalability: Node.js nodes and Python worker nodes now scale independently. Heavy API traffic scales the Node.js tier. A growing export backlog auto-scales the Python worker tier. No vertical scaling ceiling.
- Isolated Failure Radius: A crashed or hung export worker is fully isolated. The core API stays up. Other workers continue processing. No single heavy job can take down the platform or affect any other user.
“End-of-month used to mean holding your breath. Now it is just another day. The API stays fast no matter how many exports are running, and we can scale the workers up in minutes if the queue starts backing up.”
– VP Engineer
An enterprise SaaS platform whose Node.js monolith was being brought down by its own users’ data export requests, with CPU-bound workloads blocking the event loop and cascading timeouts across unrelated endpoints, was transformed through Ksolves DevOps consulting services. A RabbitMQ Pub/Sub architecture with dedicated Python worker nodes decoupled heavy processing from the core API completely. Zero API timeouts. 100% uptime during peak load. Independent horizontal scaling. Isolated failure radius. Node.js does what it does best. Python handles the number crunching it was built for.
Is Your node.js API Slow Because of Work It Should Never Be Doing?
