# Pod Startup Forensics: Closing the Gap [Part 5](/blog/pod-startup-forensics-surviving-deletion) closed the durability gap: evidence captured while a pod exists, readable after it's gone. This part closes a different gap. In the slow-start capture, `Pulled` and `Started` appear, followed by one coalesced `Unhealthy` record before `Ready`. The event log has little to say about what happened between those records. ## Kubernetes' blind spot A crash-looping container produces Events such as `BackOff`. A container that will become healthy but is slow to start may produce only a coalesced `Unhealthy` Event: the API records a repeat count instead of one Event for every failed readiness probe. kubelet continues polling throughout. Event aggregation, not kubelet going quiet, is why the timeline looks sparse. The eBPF tracer can see work in that gap, but its evidence was not yet part of the timeline. It captures syscalls whether or not Kubernetes emits an Event. Closing the gap meant merging those socket syscalls and kubelet's probe attempts with the Kubernetes record. ## Filling the gap This part explains how the tracer correlates and renders those syscalls, rather than how it hooks them; [Part 4](/blog/pod-startup-forensics-the-architecture) covers the tracepoints. Three pieces make up the work: **What the socket syscalls mean for a readiness probe.** For a server that listens on a TCP port, `listen()` after `bind()` marks the point at which a TCP-based readiness probe can succeed. **kubelet probe correlation.** The correlation path covers TCP or HTTP readiness probes against the pod's own IP. A probe with an overridden host and every exec probe are out of scope. kubelet makes the observed `connect()` call from its own process, so the tracer identifies kubelet calls and joins their destination address to a pod IP instead of attempting cgroup attribution. **A merged, filtered rendering.** The timeline interleaves syscalls, probe attempts, and Kubernetes Events by timestamp. Its default view collapses sub-10-millisecond `openat()`/`mount()`/`read()` bookkeeping into summary lines; `--raw` retains every event. ## `connect()` isn't readiness Building probe correlation surfaced a real mistake in how a `connect()` return code was being interpreted. kubelet's HTTP and TCP readiness probes use Go's non-blocking socket dial, so the `connect()` syscall itself can return `EINPROGRESS` ("in progress, ask again later") regardless of whether the probe eventually succeeds or fails. Treating an `EINPROGRESS` return as `succeeded = true`, on the reasoning that a non-blocking call proceeding normally counts as success, doesn't hold up. A cluster capture falsified that interpretation: all 21 observed HTTP/TCP readiness-probe `connect()` calls returned `EINPROGRESS`, including probes that Kubernetes recorded as `Unhealthy`. The syscall therefore cannot say whether the probe later succeeded or failed; that outcome is determined by a follow-up call the tracer does not hook. The fix treats `succeeded` as true only for an immediate `0`, `failed` only for a real error code other than `EINPROGRESS`, and everything else, the common case, as pending. The authoritative signal for whether a pod actually became ready stays exactly where it belongs: Kubernetes' own `Ready` status condition, fetched directly from the API rather than re-derived from the probe attempts the tracer observed. ## The timeline The test container sleeps for twelve seconds before opening a listening socket, standing in for real startup work. Its readiness probe begins after one second and runs every two seconds. The container itself never crashes and the image never fails to pull; its readiness probe does fail, repeatedly, until the socket actually opens, which is expected. That shape, slow to become ready but never crashing, is different from every crash or stuck-image case the series has covered so far. Each row below shows a wall-clock timestamp, an elapsed-time offset from pod creation, a source tag (a Kubernetes Event, the pod's `Ready` condition, an eBPF-observed kubelet probe attempt, or an eBPF-observed syscall), and what happened: ``` 2026-08-22T20:08:12Z T+ 1.000s [K8S EVENT] Started Container started 2026-08-22T20:08:14Z T+ 3.000s [K8S EVENT] Unhealthy (x5 repeats over 8.000s) Readiness probe failed: dial tcp 10.244.0.21:8080: connect: connection refused 2026-08-22T20:08:16.988402Z T+ 5.988s [EBPF PROBE] kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS] 2026-08-22T20:08:18.987952Z T+ 7.987s [EBPF PROBE] kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS] 2026-08-22T20:08:20.988120Z T+ 9.988s [EBPF PROBE] kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS] 2026-08-22T20:08:22.988318Z T+ 11.988s [EBPF PROBE] kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS] 2026-08-22T20:08:24Z T+ 13.000s [K8S READY] Ready=True (from pod status, not derived from probe attempts above) 2026-08-22T20:08:24.228734Z T+ 13.228s [EBPF SYSCALL] bind() -> [::]:8080 duration=0.000s 2026-08-22T20:08:24.228750Z T+ 13.228s [EBPF SYSCALL] listen() duration=0.000s 2026-08-22T20:08:24.988520Z T+ 13.988s [EBPF PROBE] kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS] ``` This capture predates the correction, so its `FAILED` label is itself evidence of the bug; the corrected build renders those events as `PENDING`. The probe rows fill in attempts that Kubernetes coalesced into one `Unhealthy` Event. The API records `Ready=True` only to whole-second precision. Its displayed T+13.000 value therefore cannot order it relative to the microsecond eBPF rows. The later `FAILED` probe is likewise a pre-fix rendering error: that `EINPROGRESS` result was pending, not a failed readiness check. Filtering makes the timeline usable: a comparable slow-start capture fell from 857 lines to 225, collapsing 643 routine syscalls into eleven summary lines while retaining every Kubernetes Event, probe attempt, and socket syscall. ## Attribution race Correlating socket syscalls exposed a cgroup-attribution race. Only 22.2% of `bind()` calls resolved to a pod cluster-wide; the cgroup index refreshes every three seconds, but a lookup miss was finalized as `unknown` immediately. Startup-time socket calls are rare and early, so a call can arrive before the index catches up and, before this change, never get another chance to resolve. ## Retry on misses On a lookup miss, the tracer now holds the event and retries after each index refresh for up to twelve seconds, then uses the existing `unknown` path. Four refresh cycles let an event survive a missed refresh without retaining it indefinitely. System processes and static control-plane pods still remain unresolved; no retry can join their cgroups to a workload pod. ## Results An unfixed reproduction captured a successful `bind()` for a cgroup known to belong to the test pod, yet stored it as `unknown`; it remained unresolved forever. After the retry change, every captured `bind()` and `listen()` event for a resolvable workload-pod cgroup received an identity: 195 `bind()` events and 17 `listen()` events. The unfiltered cluster-wide `bind()` rate rose more modestly, from 22.2% to 30.0%, because system, stale, and static-control-plane cgroups are not resolvable by this mechanism. ## Isolation test A retry-and-hold buffer introduces a new failure mode worth checking for directly: if two pods are racing through startup at the same time, could a delayed resolution attribute one pod's syscall to the other? The isolation test started two pods in the same second with deliberately different syscall signatures: one ran a file-read loop; the other ran a socket-connect loop that blocked for about a second per attempt. ``` pod_name | syscall | count | sum(duration) ---------------------+---------+-------+----------------------- isolation-pod-a | connect | 2 | 3.2709e-05 isolation-pod-a | openat | 2048 | 0.026864375 isolation-pod-a | read | 1034 | 0.032389957 isolation-pod-b | bind | 13 | 0.000282291 isolation-pod-b | connect | 14 | 12.000398686 isolation-pod-b | openat | 251 | 0.022781171 isolation-pod-b | read | 129 | 0.019244945 ``` Neither pod's syscalls appeared under the other's identity. The capture verifies final attribution, not an uninterrupted event-by-event trace: the tracer restarted after reaching its configured memory limit during the test. ## Remaining limits One concrete gap remains: the correlator cannot match static control-plane pods to their Kubernetes mirror objects because their identifiers differ. The slow-start capture exposes the previously opaque interval and keeps probe outcomes separate from `connect()` observations. In the two-pod isolation test, retry resolution didn't cross-attribute the concurrently starting pods. [Part 7](/blog/pod-startup-forensics-closure-and-future) closes the series by assessing the remaining production questions, including sustained database writes and the tracer's memory ceiling.