Project Name
Ksolves Builds a Custom NiFi Processor to Automate XLSB Ingestion for a Data Analytics Platform
A distributed data analytics platform ingests files from multiple remote and on-premises sources into a Hadoop-based data lake and a real-time analytics database, using Apache NiFi 1.27 as the orchestration backbone. As data volumes grew, some edge systems began producing binary Excel (XLSB) exports, a format Apache NiFi’s native processor library could not parse. That single unsupported format threatened to leave a permanent blind spot in the client’s analytics, forcing manual exports while the rest of the platform’s data flowed through in real time.
Ksolves designed, built, and packaged a custom Java processor that closed the gap natively inside the existing NiFi flow. The client now ingests XLSB files automatically alongside every other source, with no manual conversion step required.
- No Native XLSB Support: Apache NiFi 1.27's out-of-the-box processor library had no way to parse binary Excel workbooks, leaving one edge source completely unintegrated.
- Manual Workaround Risk: Without automated ingestion, the team had to convert files by hand or script one-off jobs outside the governed NiFi flow, undermining consistency and auditability.
- High-Throughput Requirement: Any custom component had to sustain the same volume and velocity as the rest of the flow without becoming a bottleneck.
- Dual-Destination Consistency: Parsed records needed to land reliably in both HDFS and ClickHouse without one write path lagging or failing silently.
- Business-Specific Parsing Logic: Raw spreadsheet rows had to be mapped to the client's specific schema and business rules before being usable downstream.
- NiFi Standards Compliance: The new component had to behave like a first-class NiFi processor, with configurable properties, proper relationships, back-pressure awareness, and provenance tracking.
Ksolves closed the gap by extending Apache NiFi itself rather than working around it, building a custom Java processor that behaves exactly like a native NiFi component: configurable, restartable, and observable through the platform's own provenance and monitoring tooling.
- Custom XLSB Parsing Processor: A Java-based Apache NiFi processor packaged as a NAR artifact via Maven, capable of streaming and parsing binary Excel workbooks without loading entire files into memory.
- Standards-Compliant NiFi Extension: Implemented against NiFi's Processor API with defined relationships, property descriptors, and back-pressure handling so the component integrates natively into existing flows and monitoring.
- Business-Rule Mapping Layer: Parsing logic that translates raw XLSB rows into the client's target schema, applying business-specific validation and transformation before records leave the processor.
- High-Throughput Design: Tuned for concurrent task execution and efficient row-level streaming, keeping pace with volume flowing in from the edge alongside other source types.
- Parallel Dual-Write Flow: The downstream flow fans parsed records out into simultaneous HDFS and ClickHouse write paths, preserving both long-term storage and near-real-time analytics from a single ingestion run.
- Edge-to-Analytics Orchestration: The broader NiFi flow now handles file intake from the edge, routing, and delivery end-to-end, with the new processor running as a drop-in stage alongside existing standard processors.
Technology Stack
| Category | Technology |
|---|---|
| Integration | Apache NiFi 1.27 |
| Processing | Custom Java Processor |
| Architecture | NiFi Processor API |
| Database | HDFS |
| Database | ClickHouse |
| Infrastructure | Maven |
- Zero Manual Conversion Steps: The one edge source producing XLSB files now flows through the same governed Apache NiFi pipeline as every other source, with no manual exports or ad hoc scripts required.
- One Unified File-Type Pipeline: The platform's ingestion coverage gap is closed, with XLSB parsing running natively alongside every existing processor in the flow.
- Two Parallel Write Destinations: Every parsed record lands simultaneously in HDFS and ClickHouse from a single ingestion run, keeping long-term storage and real-time analytics in sync.
- Full Provenance Coverage: The custom processor is tracked through the same NiFi provenance, back-pressure, and monitoring tooling as every other stage in the flow, closing the audit gap left by manual workarounds.
- Native Throughput Parity: The processor sustains the same concurrent volume as the rest of the flow, with no added bottleneck at the XLSB ingestion stage.
- Reusable Extension Pattern: The same custom-processor approach is ready to apply to any future file type NiFi does not support out of the box.
The client chose Ksolves Apache NiFi Development Services to close a format gap without disrupting the throughput or reliability of its existing ingestion flow. Ksolves delivered a custom Java processor, built and packaged with Maven, that ingests XLSB files as a first-class stage inside the same governed NiFi pipeline.
Before this engagement, one edge source producing binary Excel files sat outside the governed flow, requiring manual handling and creating a blind spot in the platform’s analytics. After the custom processor went live, XLSB files are ingested automatically, parsed records flow in parallel into both HDFS and ClickHouse, and the entire path runs under NiFi’s native provenance and monitoring controls.
With the format gap closed, the same extension pattern is ready to be applied to any future file type that the platform’s ingestion layer does not support out of the box.
Have a Data Source Your Ingestion Platform Cannot Parse Out of the Box?
