Airflow Kubernetes Executor Operations

By Michael Samuel Naeem · August 24, 2026 · 6 min read

Worker pod OOM on wide dataframe—no limit on KubernetesPodOperator.

What broke first on dashboards

Worker pod OOM on wide dataframe—no limit on KubernetesPodOperator.

On-call sees green infrastructure metrics while business KPIs diverge — classic sign the gate is not on the critical path.

Root cause — not the obvious answer

Root cause tied to shared worker image tag latest—non-reproducible task env.

Kubernetes executor was treated as a one-time setup task instead of an operational contract with owners and SLOs.

Fix path we kept

Move Kubernetes executor into the promote path with explicit failure semantics. Add partition-level coverage, not sample-only checks.

Add CI enforcement so misconfigurations cannot merge.

Reference configuration

# Operational hook for Kubernetes executor
@task(retries=3, retry_delay=timedelta(minutes=5))
def run_airflow_kubernetes_executor():
    validate_preconditions()
    execute()
    emit_lineage(run_id=ctx.run_id)

Day-two ownership

Assign a named owner team, review thresholds quarterly, and rehearse rollback.

New hires should execute a safe canary using only the runbook within their first week.

What to do this week

If you only do one thing this week: put Kubernetes executor on the critical path for one tier-1 workflow and measure what it catches.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Handoff to adjacent teams

Data Pipelines pipelines touch ingestion, serving, and finance. Document interfaces where Kubernetes executor gates hand off to downstream owners so failures are not bounced without context.

Operating Kubernetes executor at scale

After the first successful deploy of airflow kubernetes executor operations, most incidents trace to assumptions that stopped being true: traffic doubled, schemas drifted, or credentials rotated without updating consumers. Schedule a quarterly review of Kubernetes executor settings with the on-call rotation — not only the primary author.

Further reading

• https://opentelemetry.io/docs/