Stream Processing with Apache Flink

By Michael Samuel Naeem · September 24, 2025 · 5 min read

Our fraud detection pipeline was batch: Spark jobs every fifteen minutes scanning transaction logs for suspicious patterns. By the time we flagged a stolen card, the attacker had completed six more purchases. Moving to Flink dropped detection latency from fifteen minutes to under two seconds — same rules, same data, but operating on the stream as events arrived instead of waiting for the next batch window.

Apache Flink is a distributed stream processing engine designed for unbounded data. Unlike batch systems that process fixed datasets, Flink treats data as a continuous flow — events arrive indefinitely, and the system maintains state, applies windows, and produces results in real time. It's the engine behind real-time dashboards, fraud detection, anomaly alerting, and event-driven ETL at companies like Alibaba, Uber, and Netflix.

Core programming model

Flink's DataStream API models computation as a directed graph of operators connected by data streams:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

DataStream<Transaction> transactions = env
    .addSource(new FlinkKafkaConsumer<>("transactions", schema, props))
    .assignTimestampsAndWatermarks(
        WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(Duration.ofSeconds(5))
            .withTimestampAssigner((event, ts) -> event.getTimestamp())
    );

DataStream<Alert> alerts = transactions
    .keyBy(Transaction::getAccountId)
    .process(new FraudDetectionFunction());

alerts.addSink(new FlinkKafkaProducer<>("alerts", alertSchema, props));

env.execute("Fraud Detection");

Each operator transforms the stream: map, filter, keyBy, window, process. The keyBy partitions the stream by a key (account ID), ensuring all events for the same account land on the same parallel task — essential for stateful processing.

Event time and watermarks

Batch systems assume all data is present before processing. Streams don't — events arrive late, out of order, and at variable rates. Flink's event-time model handles this with watermarks:

A watermark at time T means "I believe no events with timestamp less than T will arrive." When a window's watermark passes its end time, the window closes and emits results. Late events beyond the allowed lateness are dropped or sent to a side output.

transactions
    .keyBy(Transaction::getMerchantCategory)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.seconds(30))
    .aggregate(new TransactionAggregator())
    .addSink(new DashboardSink());

This five-minute tumbling window counts transactions per merchant category using event timestamps, tolerating up to thirty seconds of lateness. Production pipelines tune watermark delay and allowed lateness based on observed data skew.

Stateful operators

Flink's killer feature is managed, fault-tolerant state. Stateful functions accumulate data across events:

public class FraudDetectionFunction extends KeyedProcessFunction<String, Transaction, Alert> {
    private ValueState<Double> rollingTotal;
    private ValueState<Integer> eventCount;

    @Override
    public void open(Configuration config) {
        rollingTotal = getRuntimeContext().getState(
            new ValueStateDescriptor<>("total", Double.class));
        eventCount = getRuntimeContext().getState(
            new ValueStateDescriptor<>("count", Integer.class));
    }

    @Override
    public void processElement(Transaction tx, Context ctx, Collector<Alert> out) {
        double total = rollingTotal.value() == null ? 0 : rollingTotal.value();
        int count = eventCount.value() == null ? 0 : eventCount.value();
        total += tx.getAmount();
        count++;
        rollingTotal.update(total);
        eventCount.update(count);

        if (count > 10 && total > 5000) {
            out.collect(new Alert(tx.getAccountId(), "Velocity spike"));
        }
    }
}

State is partitioned by key, checkpointed to durable storage (S3, HDFS), and restored on failure. You don't manage Redis or external state stores for pipeline state — Flink handles persistence, consistency, and recovery.

Window types

Windows group events for aggregation:

• Tumbling: Fixed-size, non-overlapping (every 5 minutes).
• Sliding: Fixed-size, overlapping (5-minute window sliding every 1 minute).
• Session: Gap-based — events within 30 seconds of each other form a session.
• Global + trigger: Custom logic for when to emit results.

// Session windows for user activity
clicks
    .keyBy(Click::getUserId)
    .window(EventTimeSessionWindows.withGap(Time.minutes(10)))
    .apply(new SessionSummaryFunction());

Session windows are powerful for user behavior analysis — each user's activity burst becomes one window, regardless of clock alignment.

Deployment and scaling

Flink runs on YARN, Kubernetes, or standalone clusters. Parallelism controls how many task slots process the stream concurrently. Keyed streams scale by adding parallelism — each key maps to exactly one subtask, so state stays consistent.

For Kubernetes, the Flink Kubernetes Operator manages job lifecycle: submit, upgrade, savepoint-and-resume, autoscale. Savepoints are manual or scheduled snapshots for version upgrades without losing state.

Flink SQL and Table API

For teams that prefer SQL over Java/Scala/Python DataStream code, Flink SQL compiles declarative queries into the same runtime:

SELECT account_id, COUNT(*) as tx_count, SUM(amount) as total
FROM transactions
WHERE event_time BETWEEN window_start AND window_end
GROUP BY account_id, TUMBLE(event_time, INTERVAL '5' MINUTE);

The Table API bridges SQL and programmatic pipelines, sharing the same checkpointing, state, and connector ecosystem.

Common production mistakes

Teams get streaming analytics flink wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of streaming analytics flink fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Debugging and triage workflow

When streaming analytics flink misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Apache Flink documentation
• Flink event time and watermarks guide
• Flink stateful stream processing
• Flink Kubernetes Operator
• Flink vs Kafka Streams comparison