Arvo-AI/lastwords

Memory snapshots before OOM kills, and logs and core dumps from crashes, for Kubernetes.
lastwords runs on every node. When a container gets close to its memory limit, it records what is using the memory. When a container dies, it saves the logs of that instance, the pod's events and any core dump. Everything stays on the node (or in S3), and you can read it with kubectl lastwords or from an AI agent over MCP.
sh
helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespace
## The problem
When you look at a pod that died hours ago, most of the evidence is gone:
OOMKilled leaves no heap dump. The kernel kills the process with SIGKILL. -XX:+HeapDumpOnOutOfMemoryError never fires, and Go, Node and Python get no warning either.
kubectl logs --previous only keeps one restart. After a few crash-loop restarts, the logs from the first failure are gone.
Events expire after one hour (--event-ttl, default 1h).
Core dumps go nowhere. corepattern is set per node, so crashing containers either dump nothing or dump to a path inside a container that is about to be deleted.
None of it can be recovered later. It has to be collected while the container is alive or right after it dies, which is what lastwords does.
## What it captures
| When | What lastwords saves |
| A container nears its memory limit (90%, again at 97%, or earlier if it is climbing fast) | Memory breakdown (heap vs page cache vs kernel), a memory timeline, processes. Java: class histogram, heap info, native memory, thread dump. Go: heap profile with the top allocation sites. Optional full heap dump. |
| A container is OOMKilled | The memory curve up to the kill, logs of the instance that died, pod events, and links to the snapshots taken before the kill |
| A process inside a container is OOM-killed but the container survives | What Kubernetes never reports: which process died and the memory state |
| A container crashes (non-zero exit, signal) | Log tail of the dead instance, the crash site parsed from Go/Java/Python stack traces, events, exit code meaning |
| A process dumps core (SIGSEGV, SIGABRT, ...) | The gzipped core, memory maps, binary and image, matched to the pod |
| A pod is evicted | Eviction reason, every container's logs, events |
Every capture comes with a summary.md that says what happened in plain words:
# checkout-5498f94695-526hm/checkout was OOMKilled at 16:51:38 UTC (limit 384.0 MiB); 1 snapshot taken before the kill
## What lastwords found
Memory climbed steadily for 1m1s, the pattern of a leak or an unbounded cache. Raising the limit would only delay the next kill.
Before the kill (67% of limit): Largest application classes on the heap:
com.example.checkout.CheckoutService$CartSnapshot: 2860 instances ...
Before the kill (67% of limit): One class dominates the Java heap: byte[] holds 179.2 MiB (97%).
# order-consumer-76d5fcb94-8vm4p/consumer crashed at 03:09:30 UTC: Error, exit code 2
## What lastwords found
Crashed in main.registerDiscount at /src/main.go:19 (assignment to entry in nil map).
## Use it
sh
kubectl lastwords list # everything from the last 7 days
kubectl lastwords list -n shop --kind oomkill