Cover image credit: Photo by Javier Esteban on Unsplash
Pod Startup Forensics: Joining Lifecycle Events to Syscall Evidence
A design for retaining pod-startup evidence after deletion by putting Kubernetes lifecycle records and eBPF syscall observations on the same time axis.
Series · Pod Startup Forensics
Post 3 of 7
Series · Pod Startup Forensics
Post 3 of 7
- Pod Startup Forensics: The Problem
- Pod Startup Forensics: The Tooling Gap
- Pod Startup Forensics: Joining Lifecycle Events to Syscall Evidence
- Pod Startup Forensics: The Architecture
- Pod Startup Forensics: Surviving Deletion
- Pod Startup Forensics: Closing the Gap
- Pod Startup Forensics: Closure and What's Next
Table of Contents
Tetragon can record configured syscall activity, while the Kubernetes API records lifecycle transitions. Neither record alone tells an operator which startup interval was slow and what the process did during it. The missing capability is the correlation between the two: a durable per-pod timeline that places lifecycle evidence and kernel observations on the same time axis.
Lifecycle records and syscall observations answer different questions
The Kubernetes API supplies two useful, imperfect records of startup. PodStatus records container states such as waiting, running, and terminated; Kubernetes Events may record actions such as scheduling, pulling, and starting. The components that perform those actions commonly report them, but Events are informative, best-effort data and their timestamps are not a precise phase clock. A forensic timeline has to retain the raw records it receives and make its inference rules explicit.
But an Event says that an action occurred; it does not show what a process did during the interval. In the captured slow-init run, the init-container interval lasted eight seconds, while the lifecycle records still said nothing about the process activity inside it. Kubernetes records kubelet- and control-plane-observed state; it does not watch syscalls.
eBPF observes the other side of the problem. A tracepoint on sys_enter_connect and sys_exit_connect can record that a task entered and returned from connect(), along with its timing and result. That does not make the event Kubernetes-aware. A syscall trace identifies kernel activity for a task; pod and lifecycle context live outside that event and must be resolved from container or cgroup metadata plus Kubernetes records. On its own, a trace is a wall of task identifiers and timestamps, with no indication that the activity belonged to a particular startup interval.
Neither source needs to be made into the other. Kubernetes supplies lifecycle records; eBPF supplies process activity. The missing piece is a join keyed on pod identity, with the timeline treating lifecycle boundaries as evidence of an interval and kernel observations as evidence that occurred alongside it. The comparison is useful, but it must not turn coexistence into a causal verdict: a long connect() beside a slow interval is a lead for an investigator, not proof that the call caused the delay.
The division of responsibility is specific. Kubernetes supplies the observed lifecycle records; the tracer supplies activity from selected kernel hooks. A useful output can say that a container-start interval lasted N seconds and show the connect() activity observed for the same pod during the overlapping window. Assigning that activity to a named phase, or treating it as the cause of the delay, requires explicit identity and attribution rules beyond timestamp overlap alone.
The pod that’s already gone by the time you’d query it
There’s a second design constraint that does not come from the tooling gap directly. Part 1 showed a captured short Tekton TaskRun: its step-hello container ran for about two seconds at the available timestamp precision. Any design that answers “which pod, which phase, why” only by querying live cluster state is blind to short-lived workloads after the fact, because by the time a human goes looking, the Pod object—and the status fields a live lookup would read—may already be gone.
This rules out an entire category of otherwise-reasonable design: a tool that watches the API and the kernel live, holds the result in memory or in a dashboard, and answers questions only against whatever is currently running. That shape works for a pod that is slow in front of you. It fails a pod that was slow earlier and has since been deleted. Kubernetes Events also have retention independent of the Pod lifecycle, so the raw material for a manual reconstruction eventually disappears on its own clock.
So durability cannot be an afterthought. The collector has to persist the lifecycle, Event, readiness, and kernel evidence it observes while the Pod still exists, keyed to a durable Pod record. That record is bounded by what the collector saw and by what the API retained, but it lets a later lookup avoid depending on a live Pod object. A live-query tool and a durable forensic record answer different operational questions.
Four requirements for a durable startup investigation
Put the two constraints together and a complete solution needs four capabilities, wired together by pod identity: lifecycle collection, selected kernel tracing, identity-aware correlation, and durable storage. Here, identity means a Pod UID, with namespace and name for display; process membership has to be resolved through container or cgroup metadata rather than inferred from a timestamp.
Keep the lifecycle record. A collector should retain the PodStatus and Event records it receives rather than rely on a fixed polling interval. It must distinguish observed status facts from inferred intervals, because it is not receiving a lossless, semantic feed of every startup transition. The next part shows the CLI, watcher, and tracer that implement this collection; Part 5 covers how the record survives deletion.
Capture kernel activity while it happens. A tracer must capture the syscall classes the investigation supports while a Pod is starting; it cannot recover an event after the fact. The capture policy has to be explicit about what it observes, the overhead it accepts, and evidence it may lose under load. This project traces openat, connect, mount, read, bind, listen, and accept4, plus process exec—not every syscall. A timeline can then present the observations relevant to the investigator’s hypothesis without claiming to explain every kind of startup delay.
Match activity to the right Pod. A timestamp overlap between an API record and a kernel event is not enough on a real node: several Pods can be starting at once for unrelated reasons. The correlation step has to resolve which Pod owns a process and retain that resolution long enough to join the evidence safely. Without that, the right output is an honest unknown, not a confident association based only on simultaneous timestamps.
Keep the answer after deletion. The output cannot live only in memory or behind a live query, because the Pod may be gone before anyone asks about it. The collector should retain the observed lifecycle and kernel evidence against the Pod record so a later lookup has somewhere to go even when kubectl no longer knows the Pod.
None of these four capabilities is individually new. Kubernetes collectors retain lifecycle records. Tetragon, Pixie, and Parca can contribute different kinds of kernel or runtime evidence. Distributed systems persist observations beyond the object that produced them. The tooling in Part 2 can supply individual records, but the investigator still needs a retained per-pod record that connects them without inventing causation. Part 4 shows the collector and tracer that implement that record, including the correlation limits they still have.