Migrating from Apache Spark to Apache Flink

Streaming pipelines break in predictable ways. Latency climbs. Windows returns wrong results. State falls out of sync. At some point, Spark Structured Streaming stops being the right tool, and engineering teams start looking at Apache Flink.

Flink is built differently. It processes every event the moment it arrives, not in batches. It handles event time natively, so late-arriving data is managed correctly without manual workarounds. And it keeps state inside the engine itself, backed by RocksDB, rather than depending on an external store like Redis or Cassandra.

But switching engines is not a simple swap. Spark and Flink share no common API. A direct code translation will compile and still produce wrong results in production. The teams that migrate successfully do it in phases: assess first, translate carefully, run both engines in parallel, validate output, then cut over.

That is exactly what this guide covers.

TL;DR for Decision-Makers

• Migration is not a rewrite; it is a rearchitecture. The two engines share no API surface. Budget four to eight weeks for a three-month Spark pipeline.
• Migrate selectively, not wholesale. The business case is strongest for sub-100ms latency, native event-time semantics, or stateful per-entity processing, not every streaming job.
• Run both engines in parallel during transition. Validate output parity for two to four weeks before switching traffic. Many organisations keep both permanently.

Before You Migrate: Is This the Right Move?

Not every Spark Streaming job should be migrated. Answer these questions before committing engineering time.

• What is your actual latency requirement? If the honest answer is “30 seconds,” Spark Structured Streaming handles that comfortably. Migration is clearly justified when your requirement is sub-500ms, and most compelling when it is sub-100ms.
• How much of your pipeline is stateful? Spark handles stateless transformations with no disadvantage compared to Flink. If your pipeline is primarily stateless with a final aggregation, the operational complexity of migration may not be worth the gain.
• Do you have event-time correctness problems today? Missed late events, incorrect window results, or manual backfill runs are clear signals that Flink’s event-time model will solve a real problem you have right now.
• What is your team’s operational bandwidth? Flink has a steeper operational learning curve than Spark. The right time to migrate is when you have four to six weeks of focused engineering capacity, not during a period of high operational pressure.

Migration Roadmap

A well-structured migration has five distinct phases. Each phase has a clear exit criterion before the next begins.

Figure 1. Five migration phases with typical timelines. Assessment and API mapping come first. Build and test follows. The critical phase is parallel running, two to four weeks of validating Flink output against Spark production output before any traffic is switched.

Understanding the Model Differences

The most common migration mistake is treating Flink as a faster version of Spark and attempting a direct API translation. This produces code that compiles but behaves incorrectly, often only apparent under load or when events arrive out of order.

Execution Model

Spark Structured Streaming processes micro-batches. Every result is at least one batch interval old. Flink processes each event individually the moment it arrives. Results can be emitted in milliseconds.

Migration implication: any logic that implicitly relies on batch semantics, processing all events in a window together, or using DataFrame operations that require a complete dataset, needs to be re-expressed as incremental operator logic in Flink.

State Management

Spark externalises state to Redis, HBase, or Cassandra. Flink internalises state co-located with the operator, backed by RocksDB and managed by checkpointing. No external dependency, no network round-trip, no consistency gap.

Time Semantics

Spark defaults to processing time. Flink treats event time as the primary model. Watermarks are precise, allowed lateness is configurable, and late events can be routed to side output streams rather than silently discarded.

Migration Approach: Phased Parallel Running

The safest migration pattern is parallel running, operating the Flink job alongside the existing Spark job, reading from the same Kafka topics, writing to shadow output topics, and validating output parity before switching any downstream consumer.

Figure 2. Both jobs read from the same source topic. Spark writes to the production output topic; Flink writes to a shadow topic. A validation job compares both outputs daily. Any discrepancy is a bug to investigate before traffic is switched — not a tolerance to accept.

Phase 1: API Translations

SparkSession → StreamExecutionEnvironment

Spark – Python session setup:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("order-enrichment") \
    .config("spark.streaming.stopGracefullyOnShutdown", "true") \
    .getOrCreate()

Flink — Java execution environment:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.enableCheckpointing(60_000);
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.setParallelism(12);

Reading from Kafka

Spark – Python readStream from Kafka:

raw_stream = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "kafka:9092") \
    .option("subscribe", "orders.raw") \
    .option("startingOffsets", "latest") \
    .load() \
    .select(from_json(col("value").cast("string"), order_schema).alias("data")) \
    .select("data.*")

Flink – Java KafkaSource with watermark strategy:

KafkaSource<OrderEvent> source = KafkaSource.<OrderEvent>builder()
    .setBootstrapServers("kafka:9092")
    .setTopics("orders.raw")
    .setGroupId("flink-order-enrichment")
    .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.LATEST))
    .setValueOnlyDeserializer(new OrderEventSchema())
    .build();

DataStream<OrderEvent> orders = env.fromSource(
    source,
    WatermarkStrategy.<OrderEvent>forBoundedOutOfOrderness(Duration.ofSeconds(10))
        .withTimestampAssigner((event, ts) -> event.getOrderTimestamp()),
    "orders-source");

Key difference: Flink requires an explicit watermark strategy at the source. In Spark, watermarks are applied downstream with .withWatermark(). Attaching to the source is recommended in Flink because it ensures event-time semantics are applied before any partitioning or keying.

Filter and Map Transformations

Spark – Python filter and column expression:

valid_orders = raw_stream \
    .filter(col("status") == "CONFIRMED") \
    .withColumn("order_value_usd", col("amount") * col("fx_rate")) \
    .select("order_id", "customer_id", "region", "order_value_usd", "event_time")

Flink – Java filter and map on typed objects:

DataStream<EnrichedOrder> validOrders = orders
    .filter(order -> "CONFIRMED".equals(order.getStatus()))
    .map(order -> new EnrichedOrder(
        order.getOrderId(),
        order.getCustomerId(),
        order.getRegion(),
        order.getAmount() * order.getFxRate(),
        order.getEventTime()));

    

        x
        

            
Stop Babysitting Spark Streaming
            

                                    Contact Us
                            
        
    
    

groupBy + Aggregation → keyBy + Window + Aggregate

This is one of the most significant conceptual translations. The Flink version is more explicit: window type, key, allowed lateness, and aggregation function are all separate concerns expressed independently.

Spark – Python windowed aggregation:

regional_revenue = valid_orders \
    .withWatermark("event_time", "30 seconds") \
    .groupBy(
        window(col("event_time"), "5 minutes"),
        col("region")) \
    .agg(
        sum("order_value_usd").alias("total_revenue"),
        count("*").alias("order_count"))

regional_revenue.writeStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "kafka:9092") \
    .option("topic", "revenue.by-region") \
    .option("checkpointLocation", "/tmp/checkpoints/revenue") \
    .start()

Flink – Java keyBy + TumblingEventTimeWindows:

DataStream<RegionalRevenue> revenue = validOrders
    .keyBy(EnrichedOrder::getRegion)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.seconds(30))
    .aggregate(new RevenueAccumulator(), new RevenueWindowFunction());

revenue.sinkTo(
    KafkaSink.<RegionalRevenue>builder()
        .setBootstrapServers("kafka:9092")
        .setRecordSerializer(
            KafkaRecordSerializationSchema.builder()
                .setTopic("revenue.by-region")
                .setValueSerializationSchema(new RegionalRevenueSchema())
                .build())
        .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
        .setTransactionalIdPrefix("revenue-aggregator-v1")
        .build());

mapGroupsWithState → KeyedProcessFunction

This is the most involved translation. The Flink KeyedProcessFunction gives explicit control over the state lifecycle. Timer-based session close is deterministic and inspectable in ways Spark’s timeout mechanism is not.

Spark – Python session tracking with applyInPandasWithState:

def track_user_session(user_id, events, state: GroupState):
    session = (state.get if state.exists else
               {"start_ts": None, "event_count": 0, "last_ts": None})

    for event in events:
        if session["start_ts"] is None:
            session["start_ts"] = event.timestamp
        session["event_count"] += 1
        session["last_ts"] = event.timestamp

    state.setTimeoutDuration("30 minutes")
    state.update(session)

    if session["event_count"] >= 10:
        yield UserSession(user_id, session["start_ts"], session["last_ts"], session["event_count"])
        state.remove()

Flink – Java KeyedProcessFunction with ValueState and timer:

public class UserSessionTracker extends KeyedProcessFunction<String, UserEvent, UserSession> {

    private ValueState<SessionAccumulator> sessionState;

    @Override
    public void open(Configuration cfg) {
        sessionState = getRuntimeContext().getState(
            new ValueStateDescriptor<>("session", SessionAccumulator.class));
    }

    @Override
    public void processElement(UserEvent event, Context ctx, Collector<UserSession> out) throws Exception {
        SessionAccumulator session = sessionState.value();
        if (session == null) {
            session = new SessionAccumulator(event.getTimestamp());
        }
        session.addEvent(event);
        sessionState.update(session);
        ctx.timerService().registerProcessingTimeTimer(
            System.currentTimeMillis() + 30 * 60 * 1000L);
    }

    @Override
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<UserSession> out) throws Exception {
        SessionAccumulator session = sessionState.value();
        if (session != null) {
            out.collect(session.toUserSession(ctx.getCurrentKey()));
            sessionState.clear();
        }
    }
} Get Your Migration Roadmap

Phase 2: Migrating Windowing Logic

Window semantics are where the most subtle correctness bugs are introduced. Three differences demand explicit attention.

Late data handling. Spark’s watermark silently discards late events. Flink’s allowedLateness and sideOutputLateData give you full control.

Flink – side output for late events (no Spark equivalent):

OutputTag<OrderEvent> lateOrderTag = new OutputTag<>("late-orders"){};

SingleOutputStreamOperator<RegionalRevenue> revenue = orders
    .keyBy(OrderEvent::getRegion)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.minutes(1))
    .sideOutputLateData(lateOrderTag)
    .aggregate(new RevenueAccumulator());

DataStream<OrderEvent> lateOrders = revenue.getSideOutput(lateOrderTag);
lateOrders.sinkTo(lateOrderAuditSink);

Phase 3: Migrating State

Strategy 1: Cold Start with Replay

Let Flink start from scratch, reading from the beginning of Kafka’s retention window. Works when the state can be rebuilt from event history, and an incomplete state during warm-up is acceptable. For most aggregation jobs, revenue totals, and event counts, this is the right choice.

Strategy 2: State Bootstrap from Spark Output

When a cold start is not acceptable, for a fraud detector that must have complete per-card history from day one, bootstrap Flink’s state using the State Processor API. This batch job reads the existing state from a database and writes it directly into a Flink savepoint.

Java – State Processor API to bootstrap from existing database:

ExecutionEnvironment batchEnv = ExecutionEnvironment.getExecutionEnvironment();

DataSet<CardRiskProfile> existingProfiles = batchEnv.createInput(
    new CardRiskProfileInputFormat(jdbcUrl, "SELECT * FROM card_risk_profiles"));

Savepoint savepoint = Savepoint.create(new HashMapStateBackend(), 128);

savepoint.withOperator(
    OperatorIdentifier.forUid("fraud-detector"),
    StateBootstrapTransformation.of(existingProfiles, new CardRiskStateBootstrapFunction()));

savepoint.write("s3://your-bucket/flink-savepoints/fraud-detector-bootstrap");

Strategy 3: Dual-Write Transition

Run Spark and Flink simultaneously while Flink builds its own state from the live event stream. Once Flink’s state has converged and output parity is confirmed, switch traffic. Useful when Kafka retention is shorter than the state warm-up period.

Common Migration Mistakes

• Translating batch logic directly. Code that processes a DataFrame of N events at once must be restructured to process one event at a time via KeyedProcessFunction and ValueState.
• Watermark tolerance set too tight. Setting bounded-out-of-orderness to zero assumes perfect event ordering. Real networks do not deliver that. Start with 5 to 10 seconds and adjust based on observation.
• Forgetting operator UIDs. Without explicit UIDs, Flink assigns identifiers based on position in the job graph. Any topology change will break checkpoint restore.

orders
    .keyBy(OrderEvent::getRegion)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .aggregate(new RevenueAccumulator())
    .uid("regional-revenue-aggregator")  // always set this
    .sinkTo(revenueSink)
    .uid("revenue-kafka-sink");

Flink SQL: The Lower-Effort Path for SQL-Heavy Jobs

If your Spark jobs are primarily SQL or DataFrame operations with minimal custom logic, Flink SQL is substantially less work than migrating to the DataStream API. It natively handles event-time windowing, watermarks, and exactly-once semantics.

SQL: Flink SQL tumbling window revenue (equivalent to Spark SQL streaming):

-- Flink SQL — five-minute tumbling window revenue
SELECT
    region,
    TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
    TUMBLE_END(event_time, INTERVAL '5' MINUTE)   AS window_end,
    SUM(order_value_usd)                          AS total_revenue,
    COUNT(*)                                       AS order_count
FROM orders
WHERE status = 'CONFIRMED'
GROUP BY region, TUMBLE(event_time, INTERVAL '5' MINUTE)

How Ksolves Handles Spark-to-Flink Migrations

At Ksolves, we have guided engineering teams through Spark-to-Flink migrations across financial services, e-commerce, logistics, and IoT platforms. The engagements that go well share a common pattern: selective migration, parallel running with rigorous parity validation, and a phased cutover with a tested rollback path.

What we bring to a migration engagement:

• Migration assessment: Reviewing Spark jobs, identifying workloads with a strong Flink business case, and scoping effort realistically before any code is written.
• API translation: Re-expressing Spark DataFrame and Structured Streaming logic as Flink DataStream operators with explicit attention to watermark semantics and state lifecycle.
• State migration: Designing and executing a state bootstrap strategy using the State Processor API, including populating Flink savepoints from existing databases.
• Parallel running infrastructure: Shadow topic, validation job, and parity reporting dashboard, so migration is validated quantitatively before cutover.
• Operator UID governance: Establishing conventions that ensure every future job change can safely restore from checkpoints without state loss.
• Cutover and rollback planning: Writing and rehearsing the production runbook, including the rollback procedure, so the team has confidence on the day.
• Post-migration tuning: Parallelism alignment, RocksDB configuration, checkpoint interval optimisation, and backpressure investigation.

Planning for an Apache Spark to Apache Flink migration? Book a free consultation with our experts.

Conclusion

Migrating from Spark Streaming to Apache Flink is a rearchitecture, not a line-for-line rewrite. The two engines share no API surface, and a direct translation produces code that compiles but misbehaves under real streaming conditions.

The migration earns its cost when your workload has a genuine sub-100ms latency requirement, when native event-time semantics will fix correctness problems you have today, or when the overhead of maintaining an external state store has become a real operational burden.

The proven path is selective migration, parallel running with output-parity validation, deliberate state-bootstrap planning, and a phased cutover with a tested rollback. Teams that follow this path ship migrations that hold up in production, without the weeks of incidents that come from treating Flink as a faster Spark.