oomkilling myself because of kubernetes event exporter

I run a hobby Kubernetes cluster. A few namespaces, some apps, a CI system with ephemeral runners. All I wanted was Discord notifications when pods crashed.

kubernetes-event-exporter seemed like the obvious choice. It watches Kubernetes events and forwards them to various destinations via webhooks. Over 1,000 GitHub stars, actively forked. I deployed the Bitnami Helm chart, pointed it at Discord, and called it a day.

Then I spent weeks figuring out why it kept running out of memory.

how kubernetes-event-exporter works

The tool uses Kubernetes informers to watch the Event resource. When you start the exporter, the informer syncs all existing events from the API server into memory, then watches for new ones via a persistent connection.

Each event goes through a routing tree you define in YAML. You can drop events based on namespace, type, reason, labels. Events that pass the filters get sent to receivers: webhooks, Elasticsearch, files, whatever.

The red boxes are where memory gets consumed. Drop rules come after the event is already loaded and enriched. The informer has no idea your routing config exists. It syncs everything.

drop rules don't reduce memory

This is the part the docs don't mention.

When you write:

route:
  drop:
    - namespace: ci-runners
    - namespace: kube-system
    - type: "Normal"

You might think you're telling the exporter to ignore those events. You're not. You're telling it to load them into memory, enrich them with API calls, and then throw them away.

The informer callback fires for every event. The exporter fetches the involved object's metadata (labels, annotations, owner references) from the API server. Then it evaluates your routing tree. Then it maybe drops the event. But by then, the memory is already allocated and the API call already happened.

This isn't unique to kubernetes-event-exporter. As one Stack Overflow user noted, informers watching 15k Pods can consume over 1GB of memory because they cache all watched objects before any filtering occurs.

My CI system generates thousands of events per hour from ephemeral runner pods. Every pod create, every container start, every scheduling decision gets loaded and enriched, even the ones I explicitly said to drop.

Increasing memory limits to 1Gi delayed the OOM by a few hours. Not a fix.

the namespace field only takes one string

I tried a different approach: only watch namespaces I care about.

The config has a top-level namespace field that restricts the informer's scope, but it only accepts a single string:

# What I wanted:
namespace: "app-*"  # Doesn't work

# What I tried:
namespace: "app-staging|app-prod"  # Doesn't work

# What actually works:
namespace: "app-staging"  # One namespace only

GitHub issue #232 asks for wildcard or multi-namespace support. Still open with no replies.

If you want to watch multiple namespaces while excluding noisy infrastructure namespaces, you have two options: watch everything and deal with memory issues, or deploy separate exporter instances for each namespace you care about.

no backpressure handling

The informer fires callbacks as fast as events arrive. If your receiver is slow (Elasticsearch indexing, webhook timeouts, network issues), events queue in memory.

There's no filesystem buffer, no way to pause the informer, no retry limits that shed load. Events accumulate until the process hits its memory limit and gets killed.

Fluent Bit handles similar problems with chunk-based processing and filesystem buffering. When memory limits are reached, chunks get written to disk instead of accumulating in RAM. kubernetes-event-exporter has none of that. It assumes receivers can always keep up.

the project is effectively unmaintained

The original Opsgenie repository was archived August 31, 2022. A community fork exists at resmoio/kubernetes-event-exporter with the same codebase and architecture. The last release (v1.7) was February 2024.

The Bitnami Helm chart that most people use for deployment is being discontinued. From GitHub issue #237:

"Bitnami helm chart is not going to be supported anymore after August 28th"

workarounds

Increase memory limits. 512Mi to 1Gi depending on cluster size. Buys time but doesn't fix the underlying problem.

Dedicated instances per namespace. Deploy one exporter per namespace you care about, each with namespace: <name> set. Memory stays bounded because the informer only watches one namespace. Downside: N deployments to manage.

Use Fluent Bit instead. Fluent Bit's kubernetes_events input plugin can capture events and forward them with proper backpressure handling. The config is more complex and it's a different mental model (log pipelines vs event routing), but it scales better.

Accept event loss. Set maxEventAgeSeconds low (5-10 seconds) and accept that events will be discarded during spikes. Better than OOM crashes.

kubesee

I started writing a replacement in Elixir: github.com/aluminyoom/kubesee

The idea is to use the same YAML config format but fix the architectural problems. Each receiver runs in its own Erlang process with its own message queue. Slow receivers don't block fast ones. The supervision tree restarts failed components without taking down the whole process.

It's early and most receivers are still unimplemented. If you need something production-ready today, the Resmo fork with workarounds is your best bet. But if these architectural issues bother you as much as they bothered me, maybe take a look.