Stream Processing with Apache Flink

By Michael Samuel Naeem · September 24, 2025 · 8 min read

Our fraud detection pipeline was batch: Spark jobs every fifteen minutes scanning transaction logs for suspicious patterns. By the time we flagged a stolen card, the attacker had completed six more purchases. Moving to Flink dropped detection latency from fifteen minutes to under two seconds — same rules, same data, but operating on the stream as events arrived instead of waiting for the next batch window.

Apache Flink is a distributed stream processing engine designed for unbounded data. Unlike batch systems that process fixed datasets, Flink treats data as a continuous flow — events arrive indefinitely, and the system maintains state, applies windows, and produces results in real time. It's the engine behind real-time dashboards, fraud detection, anomaly alerting, and event-driven ETL at companies like Alibaba, Uber, and Netflix.

Core programming model

Flink's DataStream API models computation as a directed graph of operators connected by data streams:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

DataStream<Transaction> transactions = env
    .addSource(new FlinkKafkaConsumer<>("transactions", schema, props))
    .assignTimestampsAndWatermarks(
        WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(Duration.ofSeconds(5))
            .withTimestampAssigner((event, ts) -> event.getTimestamp())
    );

DataStream<Alert> alerts = transactions
    .keyBy(Transaction::getAccountId)
    .process(new FraudDetectionFunction());

alerts.addSink(new FlinkKafkaProducer<>("alerts", alertSchema, props));

env.execute("Fraud Detection");

Each operator transforms the stream: map, filter, keyBy, window, process. The keyBy partitions the stream by a key (account ID), ensuring all events for the same account land on the same parallel task — essential for stateful processing.

Event time and watermarks

Batch systems assume all data is present before processing. Streams don't — events arrive late, out of order, and at variable rates. Flink's event-time model handles this with watermarks:

A watermark at time T means "I believe no events with timestamp less than T will arrive." When a window's watermark passes its end time, the window closes and emits results. Late events beyond the allowed lateness are dropped or sent to a side output.

transactions
    .keyBy(Transaction::getMerchantCategory)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.seconds(30))
    .aggregate(new TransactionAggregator())
    .addSink(new DashboardSink());

This five-minute tumbling window counts transactions per merchant category using event timestamps, tolerating up to thirty seconds of lateness. Production pipelines tune watermark delay and allowed lateness based on observed data skew.

Stateful operators

Flink's killer feature is managed, fault-tolerant state. Stateful functions accumulate data across events:

public class FraudDetectionFunction extends KeyedProcessFunction<String, Transaction, Alert> {
    private ValueState<Double> rollingTotal;
    private ValueState<Integer> eventCount;

    @Override
    public void open(Configuration config) {
        rollingTotal = getRuntimeContext().getState(
            new ValueStateDescriptor<>("total", Double.class));
        eventCount = getRuntimeContext().getState(
            new ValueStateDescriptor<>("count", Integer.class));
    }

    @Override
    public void processElement(Transaction tx, Context ctx, Collector<Alert> out) {
        double total = rollingTotal.value() == null ? 0 : rollingTotal.value();
        int count = eventCount.value() == null ? 0 : eventCount.value();
        total += tx.getAmount();
        count++;
        rollingTotal.update(total);
        eventCount.update(count);

        if (count > 10 && total > 5000) {
            out.collect(new Alert(tx.getAccountId(), "Velocity spike"));
        }
    }
}

State is partitioned by key, checkpointed to durable storage (S3, HDFS), and restored on failure. You don't manage Redis or external state stores for pipeline state — Flink handles persistence, consistency, and recovery.

Window types

Windows group events for aggregation:

• Tumbling: Fixed-size, non-overlapping (every 5 minutes).
• Sliding: Fixed-size, overlapping (5-minute window sliding every 1 minute).
• Session: Gap-based — events within 30 seconds of each other form a session.
• Global + trigger: Custom logic for when to emit results.

// Session windows for user activity
clicks
    .keyBy(Click::getUserId)
    .window(EventTimeSessionWindows.withGap(Time.minutes(10)))
    .apply(new SessionSummaryFunction());

Session windows are powerful for user behavior analysis — each user's activity burst becomes one window, regardless of clock alignment.

Deployment and scaling

Flink runs on YARN, Kubernetes, or standalone clusters. Parallelism controls how many task slots process the stream concurrently. Keyed streams scale by adding parallelism — each key maps to exactly one subtask, so state stays consistent.

For Kubernetes, the Flink Kubernetes Operator manages job lifecycle: submit, upgrade, savepoint-and-resume, autoscale. Savepoints are manual or scheduled snapshots for version upgrades without losing state.

Flink SQL and Table API

For teams that prefer SQL over Java/Scala/Python DataStream code, Flink SQL compiles declarative queries into the same runtime:

SELECT account_id, COUNT(*) as tx_count, SUM(amount) as total
FROM transactions
WHERE event_time BETWEEN window_start AND window_end
GROUP BY account_id, TUMBLE(event_time, INTERVAL '5' MINUTE);

The Table API bridges SQL and programmatic pipelines, sharing the same checkpointing, state, and connector ecosystem.

Checkpoint timeout triage runbook

When Flink checkpoints exceed checkpoint.timeout, open the job graph and inspect operator backpressure first — not the checkpoint interval. A source at 100% busy usually means downstream cannot keep pace; adding parallelism to the bottleneck operator beats doubling checkpoint timeout blindly.

We once chased "network blips" for three days while RocksDB incremental checkpoints on gp2 EBS throttled — moving state to local NVMe cut checkpoint duration from 4 minutes to 22 seconds without touching interval settings.

Checkpoint timeout triage runbook

When Flink checkpoints exceed checkpoint.timeout, open the job graph and inspect operator backpressure first — not the checkpoint interval. A source at 100% busy usually means downstream cannot keep pace; adding parallelism to the bottleneck operator beats doubling checkpoint timeout blindly.

We once chased "network blips" for three days while RocksDB incremental checkpoints on gp2 EBS throttled — moving state to local NVMe cut checkpoint duration from 4 minutes to 22 seconds without touching interval settings.

Resources

• Apache Flink documentation
• Flink event time and watermarks guide
• Flink stateful stream processing
• Flink Kubernetes Operator
• Flink vs Kafka Streams comparison

Trade-offs I keep revisiting for streaming analytics flink

Operating streaming analytics flink well means tying design choices to measurable outcomes and explicit owners. Ambiguous ownership is how pages rot.

For streaming analytics flink:

• Write the SLO and the user journey it protects
• Automate the boring verification; reserve humans for judgment calls
• Prefer progressive delivery with fast rollback over big-bang cuts
• Keep runbooks next to the code that can break

Revisit the design when the metric that justified streaming analytics flink stops moving — sunsetting is a feature.

Metrics and alarms for streaming analytics flink

Reviewers should challenge assumptions encoded in streaming analytics flink: defaults copied from tutorials, timeouts that exceed upstream SLAs, and authz checks applied only on the primary UI path. Require a short threat or failure note in the PR when the change touches a trust boundary.

Concrete probes:

1. Scenario C for streaming analytics flink: traffic 3× baseline — prove autoscaling or shedding keeps the golden journey healthy.
2. Scenario A for streaming analytics flink: partial dependency outage — prove clients degrade gracefully and retries do not amplify load.
3. Scenario B for streaming analytics flink: bad config shipped — prove rollback within the declared RTO without data corruption.

Post-incident changes after streaming analytics flink failures

Roll out streaming analytics flink behind a flag or weighted route when possible. Start with internal users or a low-risk geography. Watch the signals in the table for at least one full business cycle before calling the migration done. Keep the previous path warm until error budgets stabilize.

Document the owner, the dashboard, and the single command that reverts the change. If that sentence is hard to write, the design is not ready for production traffic.

Multi-tenant concerns in streaming analytics flink

Detail 1 (673): for streaming analytics flink, define the contract between producers and consumers explicitly — payload shape, timeout, and idempotency key. When multi-tenant concerns in streaming analytics flink becomes painful, it is usually because that contract was implicit.

I keep a short matrix: who can break streaming analytics flink, how we detect it within five minutes, and who is paged. Update the matrix when ownership moves. Add one synthetic check that exercises the failure path, not only the happy path. Prefer checks that run continuously over quarterly manual reviews that everyone skips under deadline pressure.

If you only remember one thing about streaming analytics flink: optimize for reversible decisions. Reversibility beats cleverness when the incident channel is busy and the blast radius is unclear.

Compliance evidence for streaming analytics flink

Detail 2 (255): for streaming analytics flink, define the contract between producers and consumers explicitly — payload shape, timeout, and idempotency key. When compliance evidence for streaming analytics flink becomes painful, it is usually because that contract was implicit.

I keep a short matrix: who can break streaming analytics flink, how we detect it within five minutes, and who is paged. Update the matrix when ownership moves. Add one synthetic check that exercises the failure path, not only the happy path. Prefer checks that run continuously over quarterly manual reviews that everyone skips under deadline pressure.

If you only remember one thing about streaming analytics flink: optimize for reversible decisions. Reversibility beats cleverness when the incident channel is busy and the blast radius is unclear.