Runtime Security with Falco and eBPF

SecurityKubernetesObservability

Most security tooling checks your software before it runs. Runtime security watches what it does once it's actually running — and that's where you catch the attacks that scanning can't predict. Runtime security is the practice of observing live workload behavior (system calls, process execution, file and network activity) and flagging the patterns that indicate compromise: a database container spawning /bin/bash, a process reading /etc/shadow, an unexpected outbound connection to a fresh IP. Falco, a CNCF project, does this by tapping the Linux kernel through eBPF and matching a stream of syscalls against rules in real time.

I added Falco to a cluster after an incident where a compromised dependency did exactly nothing a pre-deploy scan would have caught — the malicious behavior only showed up at runtime. That's the gap this closes. Here's how it works and how to run it without generating an alert firehose nobody reads.

Why syscalls are the right vantage point

Almost everything a process does that matters for security eventually becomes a system call. Opening a file, spawning a child process, making a network connection, changing permissions — all syscalls. If you can observe syscalls reliably and cheaply, you can see the ground truth of what a workload is doing, regardless of what language it's written in or how it was packaged.

The historical problem was cost and safety. Watching every syscall used to mean kernel modules (fragile, dangerous) or ptrace-style interception (slow). eBPF changed the economics: you load a small, verified program into the kernel that observes events and streams them to userspace with minimal overhead and no custom kernel module. The kernel's verifier guarantees your eBPF program can't crash the system. This is the same technology that powers modern eBPF-based observability with OpenTelemetry — the security use case is the same instrumentation aimed at threats instead of latency.

How Falco is wired

The architecture is straightforward: an eBPF program in the kernel captures syscalls, Falco's userspace engine enriches them with container and Kubernetes metadata (which pod, which image, which namespace), and a rules engine evaluates each event. Matches produce alerts routed to stdout, a file, gRPC, or — via Falcosidekick — to Slack, a SIEM, or an alerting pipeline.

That metadata enrichment is what makes it useful in Kubernetes. A raw syscall alert saying "process 4821 opened /etc/shadow" is nearly useless. "Container payments-api (image acme/payments:2.3) in namespace prod read /etc/shadow" is actionable.

Writing rules that don't drown you

Falco ships with a solid default ruleset, and your first instinct will be to enable all of it. Don't — not without tuning. The default rules are broad on purpose, and in a real cluster they'll fire constantly on legitimate behavior (package managers, init scripts, monitoring agents). Alert fatigue kills runtime security faster than any attacker; if every alert is noise, the real one gets ignored.

A Falco rule is YAML: a condition over syscall fields, plus output and priority.

- rule: Shell spawned in web container
  desc: A shell was executed inside a container that should never need one
  condition: >
    spawned_process
    and container
    and shell_procs
    and container.image.repository in (acme/payments, acme/frontend)
  output: >
    Shell in web container
    (user=%user.name container=%container.name
     image=%container.image.repository cmdline=%proc.cmdline)
  priority: CRITICAL
  tags: [container, shell, mitre_execution]

The discipline that keeps this signal-rich:

What it catches that nothing else does

Concretely, the class of threats runtime detection owns:

Threat Runtime signal Falco sees
Reverse shell from exploited service Web process spawns shell + outbound connect
Container escape attempt Sensitive mount access, privileged syscall
Cryptominer Unexpected binary, sustained CPU, mining pool DNS
Credential theft Read of /etc/shadow, cloud metadata endpoint
Supply-chain payload New process not in the image's known set

None of these are visible to a pre-deploy scanner, because they're behaviors, not artifacts. A vulnerable package might sit dormant for months; runtime security only fires when it's actually abused. That's the complement to artifact-based controls like container image security and SBOMs — scanning tells you what could go wrong, runtime tells you what is going wrong.

The honest limitations

Falco detects; it doesn't, by default, prevent. It tells you a shell spawned in your payments container after the fact. You can wire responses (kill the pod, quarantine the node) through Falcosidekick and response engines, but that automation carries its own risk — an aggressive auto-kill rule with a false positive can take down production. I keep enforcement conservative and let humans make the kill decision for anything short of the most unambiguous signals.

There's also overhead. eBPF is cheap, not free; on very high-syscall-rate workloads you'll pay a few percent CPU, and you should measure it rather than assume. And rules are only as good as your understanding of normal — the tuning work is real and ongoing, not a one-time setup.

Where it fits

Runtime security is one layer of defense in depth, not a silver bullet. It sits downstream of secure images, least-privilege RBAC, network policies, and admission control. Its unique contribution is catching the unknown-unknowns — the behavior no pre-deploy check anticipated — with kernel-level ground truth. For any cluster running third-party code or exposed to the internet, that's a layer I now consider mandatory rather than optional. Set it up in audit mode, tune it until the signal is trustworthy, and you'll have visibility into the one phase of the software lifecycle that most tooling ignores: the part where it's actually running.

Resources

Frequently asked questions

What is runtime security?

Runtime security is the practice of detecting and responding to threats while workloads are actually running, as opposed to scanning images or code before deployment. It watches live behavior — system calls, process launches, network connections, file access — and flags activity that indicates compromise, such as a web server suddenly spawning a shell.

How does Falco use eBPF?

Falco uses an eBPF program loaded into the Linux kernel to observe system calls with very low overhead and without kernel modules. Every syscall of interest is streamed to Falco's userspace engine, which evaluates it against a set of rules and raises alerts on matches. eBPF is what makes this deep visibility safe and performant enough for production.

Does runtime security replace image scanning?

No — they're complementary layers. Image scanning and SBOMs catch known-vulnerable components before deploy; runtime security catches what actually happens at execution, including zero-days and misuse that no scan could predict. A mature program runs both, since each covers the other's blind spot.

Hiring a senior Android / Flutter engineer?

I architect and ship production mobile software — Kotlin, Jetpack Compose, Flutter — for robotics, EV infrastructure, fintech, and real-time systems. Open to remote roles in Europe and the US.

Get in touch →