Bhawani Singh

Pod Startup Forensics: The Problem

Bhawani Singh — Sat, 18 Jul 2026 00:00:00 GMT

Suppose a pod takes 40 seconds to start instead of 4. It doesn't look like an incident. Nothing pages. kubectl get pods eventually shows Running. The deploy finishes. The only trace is a vague sense that something felt slow, and by the time anyone goes looking, the pod that was actually slow is usually gone.

Kubernetes has a real metric for this. kubelet exposes kubelet_pod_start_sli_duration_seconds, an alpha histogram for Kubernetes' narrower startup SLI: it excludes image pulls and init-container execution. SIG-Scalability's official stateless-pod SLO uses that signal: the 99th percentile, measured over the last five minutes, must stay at or below 5 seconds for a cluster-day. So the instinct is to assume the problem is already solved.

It isn't. That metric tells you the cluster-wide p99 crept up this week. It can't tell you which pod, which phase, or why.

Autoscaling only works if the scale-up is fast

The entire value of a Horizontal Pod Autoscaler or Cluster Autoscaler reacting to load is that new capacity shows up before the old capacity falls over. Consider a scale-up that completes after demand has already outrun the existing pods. The autoscaler technically did its job. It decided to add capacity. But the capacity arrived too late to protect the service: existing pods absorb the overflow, latency degrades, and the new pod can arrive after the pressure has already subsided or triggered a different alarm.

Pod startup latency isn't one number. It's schedule delay, image pull, PVC attach and mount as two separate operations, init containers running in sequence, sidecars starting alongside or before the main container, and the main container itself becoming ready. Every one of those phases is a place autoscaling responsiveness can quietly evaporate, and none of them show up as a single knob to tune. Picture a cluster that scales in a few seconds on a warm node and takes a minute or more on a node that has to pull a fresh image: that cluster doesn't have one autoscaling latency. It has a distribution, and the tail of that distribution is exactly the case autoscaling exists to handle: sudden load, no warm capacity, has to happen fast.

CI/CD pipelines can spend more time starting pods than doing short work

Tekton, and any CI system built the same way, one pod per unit of work, pays the pod startup cost on every Task in a pipeline. A PipelineRun with five sequential Tasks doesn't pay pod startup once. It pays it five times, because each Task runs as its own pod with its own schedule, pull, init, and ready sequence. The individual Steps inside a Task run as containers sequenced within that one pod, so they share the pod's startup cost rather than each paying it separately.

This matters more than it looks like it should, because the actual work inside a Step can be fast. The captured TaskRun pod shows its step-hello container starting at 16:35:55Z and finishing at 16:35:57Z—under two seconds of recorded wall time. That single short-task example illustrates the dynamic: when the startup overhead around a pod runs for several seconds, it is not a rounding error next to the work inside it. A pipeline that feels slow can be one where the build and test commands run fine but the pods wrapping each Task are the bottleneck, and a typical CI dashboard does not separate those two costs.

The on-call cost: Guessing with kubectl describe pod

When a pod is slow to start in production, not failing, not crash-looping, just slow, the standard move is kubectl describe pod followed by reading the Events section top to bottom. That gives a chronological list: Scheduled, Pulling, Pulled, Created, Started, maybe a FailedMount or a repeated Unhealthy if something is actually wrong. What it doesn't give is any indication of why a phase took as long as it did.

So the on-call engineer does arithmetic by hand: subtracting displayed event times—or, when querying the API directly, Event timestamps—to estimate an image-pull interval, while knowing Events are best-effort records rather than a precise phase clock. If the answer isn't obvious from the event list, there's nowhere else to look. In this project's induced connect() stall, the Event record captured lifecycle transitions but did not say that the init container was blocked in connect(). Events record controller-observed transitions, not the syscall that occupied the interval between them. The engineer either already knows this failure mode from having seen it before, or starts guessing: the CNI, DNS, the CSI driver, the registry. That guessing has a real cost: time spent per incident, and because nothing captures the answer, the same ambiguity can get re-diagnosed from scratch the next time it happens.

A fair fraction of the pods worth interrogating are already gone by the time anyone looks. Kubernetes garbage-collects completed pods, CI systems delete TaskRun pods after a retention window, and a pod that failed to schedule and got evicted leaves no long-lived object to describe. kubectl describe pod only works on a pod that still exists. Kubernetes Event objects have their own bounded retention window—one hour by default at the API server—and can outlive the Pod they refer to. Once the Pod is deleted, kubectl describe pod no longer works; once the Event TTL expires, the associated chronology is gone too. "Why was that slow" becomes permanently unanswerable through the tools most people reach for first, and for a Tekton Task pod whose steps finish in a couple of seconds, that clock starts running almost immediately.

Why kubectl get events isn't a phase breakdown

kubectl get events, or describe pod's embedded events section, gives the raw material a phase breakdown would be built from, not the breakdown itself. It's a flat, chronological list of whatever the kubelet and the various controllers happened to report, in the order they happened to report it. There's no concept of a phase in that stream. No line says "image pull took 10.4 seconds" or "this was the slowest part of startup." Pulling shows up at one timestamp and Pulled at another, and it's on the reader to subtract them, notice which containers were init containers versus sidecars versus the main container (which, depending on Kubernetes version, can report state in genuinely different shapes: a native sidecar with restartPolicy: Always shows up as running in the same status array where an ordinary init container shows terminated), and reconstruct the ordering by hand. There's no root cause in that stream by construction, either. An Event is a report that something happened, not an explanation of what the process was doing while it was blocked. If a container spent eight seconds inside a connect() call, no Event says so, because Events cover what Kubernetes' own controllers observed, and no controller is watching syscalls.

Why kubelet's own latency metric doesn't answer this either

kubelet_pod_start_sli_duration_seconds is a genuinely real, useful signal for a specific question, which makes it tempting to assume it solves this one too. The gap is structural, not a matter of granularity.

Each kubelet exposes an unlabeled histogram for the pods it observes. A monitoring system can aggregate those per-node series into a cluster-wide p99, but that roll-up still has no per-pod label, phase attribution, or trace back to the pod that populated a bucket. No timestamp points at a specific second and says where the time went. A cluster-wide p99 can trend upward on a dashboard for a week straight with no way to go from that trend to a single pod name worth investigating.

kubectl get events gives raw, unstructured chronology for one pod at a time, with no phase model and no root cause. kubelet's own metric gives an aggregate trend across the whole cluster, with no per-pod resolution at all. Between a single pod's raw chronology and the cluster's aggregate trend sits the actual question: which pod, which second, why. Neither stock interface covers that middle ground on its own. That gap is the one this series sets out to investigate.

One cgroup budget for a Tekton taskrun's step and its dind sidecar

Bhawani Singh — Sat, 25 Jul 2026 00:00:00 GMT

A Tekton TaskRun that builds container images runs as one pod holding one or more step containers, alongside a privileged Docker-in-Docker sidecar. A step issues docker build and docker run against the sidecar's daemon socket. The dockerd process inside the sidecar is what actually does the work: pulling layers, running RUN instructions, spawning child containers. During a build, the sidecar is the busiest thing in the pod.

If the step and the sidecar each need a 4 GiB peak at different times, separate limits reserve 8 GiB against node capacity even though the combined workload peak is 4 GiB. A shared ceiling would fit that usage.

The step and the sidecar sit in sibling cgroups under the same pod slice. The entrypoint swaps the sidecar hash for the step hash, then passes the resulting path to dockerd as --cgroup-parent. dockerd remains in the sidecar cgroup; containers it creates start beneath the step cgroup. Their memory is charged to the step's limit, and the kernel holds that limit through an OOM kill. The same nesting that makes it work is what erases the build container from the metrics every memory dashboard is built on.

The problem

The unit of budget you want here is the pod: give this whole TaskRun 4 GiB, let the step and the build children draw from it, and stop paying for a peak that never happens. Kubernetes has that concept. PodLevelResources, KEP-2837, sets spec.resources on the pod itself. It went beta and default-on in 1.34. In 1.36, InPlacePodLevelResourcesVerticalScaling graduates to beta and is enabled by default. Two pod-level-resource bug fixes, PodLevelResourcesFixUpdateDefaulting and PodLevelResourcesFixKubeletQOSClass, are beta and enabled by default in 1.36.

But Tekton can't use it. A TaskRun's podTemplate accepts a fixed subset of Pod-spec fields, and that subset excludes resources. Tekton's computeResources field configures individual containers; it cannot set spec.resources on the resulting Pod.

So a platform team that wants one number per TaskRun pod is stuck between a Kubernetes feature that's still beta and a Tekton API surface that doesn't expose it.

The options

Four candidates get the step and the build sharing one ceiling.

PodLevelResources. Set the budget on the pod and let the kernel divide it. Correct, but unavailable in Tekton's podTemplate and still beta.

PID migration via cgroup.procs. Write the sidecar's own dockerd PID into the step container's cgroup.procs, moving the daemon itself under the step's budget. This charges dockerd's own footprint, including image pulls and layer extraction, to the step, which is a different and much blunter thing than charging the build's children.

--cgroup-parent borrowing. Start dockerd with --cgroup-parent set to the step container's scope, so containers that do not override that setting are nested inside the step's cgroup, while dockerd itself stays where the kubelet put it.

Writing memory.max into the pod slice. Have something with node access compute a budget and write it directly into kubelet-kubepods-*-pod<uid>.slice/memory.max, bypassing the API entirely.

I tested --cgroup-parent borrowing. Of the four designs considered, it is the one this prototype validates inside a normal pod while targeting build children rather than the daemon that spawns them.

The chosen method

The sidecar's entrypoint runs before dockerd and does four things: read its own cgroup path, identify its own pod, ask the API server for the step container's containerd ID, and substitute that ID into the path it read.

The critical property is that dockerd's own PID never moves. The daemon stays in the sidecar's cgroup, where the kubelet put it and where the kubelet's own accounting expects it. Only the containers it goes on to create land under the step.

Step one reads the sidecar's own cgroup from procfs, which under the cgroup v2 unified hierarchy, is a single 0:: line:

$ awk -F: '/^0::/ {print $3}' /proc/self/cgroup
/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podcd57a787_d809_4a55_b6f8_ab38de73aeb8.slice/cri-containerd-4354439d2f1668c37baaabaa5a33b5ce71a1b24c182e4ebe0d16455889c14617.scope

Steps two and three read the pod name from /etc/hostname and the namespace from /var/run/secrets/kubernetes.io/serviceaccount/namespace, then poll the API server for the step container's ID, filtering out the sidecar's own entry:

kubectl get pod "$podname" -n "$namespace" -o json | \
  jq -r 'first(.status.containerStatuses[] | select(.name != "sidecar-dind" and .started == true) | .containerID)' | \
  awk -F'://' '{print $2}'

Step four is the whole trick. The sidecar's cgroup path and the step's cgroup path are siblings under the same pod slice, differing only in the container hash, so one substitution converts one into the other:

NEW_CG_PATH=$(echo "$CGROUP_PATH" | sed -E "s/cri-containerd-[a-f0-9]+\.scope/cri-containerd-$CONTAINERD_HASH.scope/")

Then dockerd starts with that path as its parent:

exec /usr/local/bin/dockerd-entrypoint.sh \
  --cgroup-parent="$NEW_CG_PATH" \
  "$@"

The sequence, after the step has started and its container ID is available:

sequenceDiagram
    participant Init as sidecar entrypoint
    participant API as kube-apiserver
    participant D as dockerd (sidecar cgroup)
    participant Step as step container
    participant CG as step .scope cgroup

    Init->>Init: read /proc/self/cgroup
    Init->>API: get pod, read step containerID
    API-->>Init: containerd://<step hash>
    Init->>Init: sed sidecar hash -> step hash
    Init->>D: exec dockerd with --cgroup-parent=<step scope>
    Step->>D: docker run / docker build
    D->>CG: create child cgroup under step scope
    Note over CG: child's memory charges to it and its step ancestor

One detail makes the substitution viable, and it's worth checking before copying any of this. The two sides of the pod run different cgroup drivers:

$ kubectl exec <pod> -c sidecar-dind -- docker info --format '{{.CgroupDriver}} / v{{.CgroupVersion}}'
cgroupfs / v2
$ kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq -r .kubeletconfig.cgroupDriver
systemd

The kubelet builds the systemd-style .slice/.scope paths that the sidecar reads from procfs, while dockerd inside the sidecar mounts cgroupfs and treats --cgroup-parent as a literal directory to create beneath. That mismatch is what lets a path lifted from the kubelet's hierarchy be handed to dockerd unchanged. A dind daemon configured with the systemd driver would expect a slice name and derive its own scope instead, and the borrowed path wouldn't nest the same way.

Two prerequisites beyond that. The sidecar's service account needs to get on pods in its own namespace, because the container ID is only available through the API. And the sidecar needs privileged: true, both for dockerd's normal reasons and because without a host cgroup namespace the container reads only the namespace-relative 0::/ and has no host path to rewrite.

What the kernel does with it

The nesting is literal. After the step tells dockerd to run a container that writes a 200 MB file, that container's cgroup appears as a subdirectory of the step's scope on the node:

$ ls .../cri-containerd-84c3e193...abe1.scope/
4c4095d78188d1ba957135bd72bf672eec1de8a40ed8e011e3b30fb1a811d247
cgroup.controllers
cgroup.subtree_control
...
memory.current
memory.max
...

The kernel charges it to the step. Reading the step scope's memory.current directly on the node, with a 200 MB child running:

$ cat .../cri-containerd-84c3e193...abe1.scope/memory.current
213598208

That's roughly 203.7 MiB against a step container whose own resident footprint before the child started was about 7 MB. The charge landed exactly where the mechanism aimed it.

That happens because of one file. A cgroup can distribute a resource to child cgroups only when it enables that controller in cgroup.subtree_control. The step's scope lists memory as available without enabling it:

$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.subtree_control
cpuset cpu pids
$ cat .../4c4095d78188...247/cgroup.controllers
cpuset cpu pids
$ cat .../4c4095d78188...247/memory.current
cat: ...: No such file or directory

So the child gets no memory.* files at all. It is not a memory-accounting boundary, so memory instantiated by its processes is accounted at the step's .scope. The mechanism works precisely because the child is not a memcg.

There's a second reason, and it's the stronger one. Reading the child's cgroup.procs returns Operation not supported, which the kernel documents for threaded cgroups and cgroup.type confirms it. Probing a separate run for those values: the child reads threaded, the step scope reads domain threaded, and a step scope with no dind child beneath it yet reads a plain domain. Threaded cgroups carry only the threaded controllers, cpu, cpuset, perf_event, and pids, and memory isn't among them. The two facts are the same one seen from either end: delegating only threaded controllers is what let the child become threaded at all, and once threaded it can never carry a memory.current. Its processes show up in cgroup.threads instead.

That's the whole finding, and everything an operator would reach for follows from it. There is no per-child memory object, so nothing reading cgroup memory files can report one. cadvisor's line for the child reads zero on working set, usage, and RSS alike:

container_memory_working_set_bytes{container="",id=".../cri-containerd-84c3e193...abe1.scope/4c4095d78188...247",...} 0
container_memory_usage_bytes{container="",id=".../4c4095d78188...247",...} 0
container_memory_rss{container="",id=".../4c4095d78188...247",...} 0

docker stats inside the sidecar, looking at the same container through the daemon that created it, agrees:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
4c4095d78188   memhog    0.00%     0B / 0B             0.00%     1.32kB / 126B   0B / 0B     1

Neither is wrong. There's no memory.current at the child to read, so both correctly report the absence as zero.

The step's own line is the one that misleads, because it does move, just nowhere near enough. cadvisor's container_memory_working_set_bytes for step-step went from about 7.2 MB before the child to about 13.9 MB with it running: roughly 6.4 MiB of movement against a kernel charge of roughly 204 MiB. Those are sampled gauges, so the individual readings drift a few hundred kB between runs, but the relationship is stable: the observed working-set increase was about one-thirtieth of the step cgroup's memory.current.

One footnote on where the 200 MB went. The step scope's memory.stat shows anon 6541312, file 200101888, and inactive_file 200101888: nearly all of the charge was inactive file cache. A memory-limited cgroup can reclaim those pages before it kills a task, although they still count toward the limit.

What happens when it OOMs

Forcing a kill needs anonymous memory. With the step capped at 64Mi via stepSpecs.computeResources and a child allocating 500 MiB of anon, the kernel does exactly what the budget says.

The kill is a real memcg kill, not a node-pressure eviction, and the cgroup it names is the step's:

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/b132849958c0.../kubelet.slice/.../kubelet-kubepods-burstable-podee4ed01f_c5b6_401d_84e6_cd45e7a94890.slice/cri-containerd-d5a62de9d463d23595850010701c41f8cd07c14626727281560162300d0f7d98.scope,task_memcg=...,task=entrypoint,pid=79431,uid=0
memory: usage 65536kB, limit 65536kB, failcnt 59

The oom_memcg path carries this run's pod UID (ee4ed01f_c5b6_401d_84e6_cd45e7a94890), and this run's step container hash, both matched against what kubectl reported for the same pod. usage 65536kB, limit 65536kB is the 64Mi cap hit exactly, with 59 prior failed charge attempts.

The kernel then killed five processes, not one, because memory.oom.group is set on the step scope:

process	pid	total-vm	anon-rss	oom_score_adj
entrypoint	79431	1301884 kB	5760 kB	996
script-0-fp877	80150	1720 kB	0 kB	996
sleep	80152	1704 kB	0 kB	996
entrypoint	79452	1301884 kB	5760 kB	996
python3	80829	523528 kB	58084 kB	0

The python3 at oom_score_adj: 0 is the child inside dind, the process that actually invoked the killer. Everything at 996 is Tekton's own step machinery: two entrypoint processes, the generated step script, and its sleep. The kernel log states the rule plainly: tasks in the step scope "are going to be killed due to memory.oom.group set." The step is billed for the child's overrun and executed for it.

Note that the killer only got 58084 kB of anon resident before hitting the wall, against a total-vm reservation of 523528 kB. The 500 MiB is what it asked for, not what it held.

Tekton usually reports the failure, but not as an OOM:

$ kubectl get taskrun cgroup-budget-run-oom -o jsonpath='{.status.conditions}'
[{"lastTransitionTime":"2026-08-18T08:32:12Z","message":"\"step-step\" exited with code 137: Error","reason":"StepFailed","status":"False","type":"Succeeded"}]

This is the nuance worth getting right, because the obvious reading is wrong. Tekton does have OOM-specific reasons: TaskRunReasonStepOOM ("StepOOM"), TaskRunReasonSidecarOOM, and TaskRunReasonInitContainerOOM, all of which exist in the v1 TaskRun types. Tekton's getFailureInfo() calls isOOMKilled(), which tests s.State.Terminated.Reason == "OOMKilled" and nothing else. It never looks at the exit code. The step's terminated state here was exitCode: 137 with reason: "Error", so the check correctly declined to fire.

The interesting part is one layer down, because Tekton isn't the component that makes the decision. containerd writes that string, and it reaches the opposite conclusion from the opposite evidence: on a 137 exit it checks whether the cgroup's memory.events counter shows an oom_kill, and only then sets the reason. So containerd gates on the exit code Tekton ignores, and Tekton gates on the string containerd may or may not get around to writing.

It usually doesn't. Polling the step scope's own memory.events in a busy loop through the kill catches the counter arriving and the cgroup disappearing almost simultaneously:

09:17:12.938 | oom_kill 5 | oom_group_kill 1 | current=774144
09:17:12.940 | directory gone

Two milliseconds. At a 20 ms sampling interval the directory was already gone. The kernel's accounting is correct and hierarchical throughout: the pod slice's memory.events reads oom_kill 5 while its memory.events.local stays at 0, which is exactly what a kill charged to a descendant should look like. The counter is there to be read. Under the systemd cgroup driver, the scope unit is garbage-collected as soon as its last process exits, and containerd races that GC to read a file that is about to be removed. containerd's own source comments name this race.

Repeat the same OOM six times; the reason comes back as Error five times and OOMKilled once, with exitCode: 137 every time. So StepOOM isn't dead code, and this isn't a Tekton bug: it's a race one layer below Tekton that Tekton faithfully reports the losing side of. An operator sees the same generic failure most of the time and the correct one occasionally, which is worse than either being consistent.

Why not to run this

The enforcement is real, and the operability isn't, and those are separable properties.

Nothing downstream of the kernel sees the charge at the granularity it happened. Dashboards built on container_memory_working_set_bytes stay roughly flat while the kernel accumulates 200 megabytes per step. This cluster had no metrics-server installed, so it did not test whether kubectl top reproduces the gap. Any downstream system that consumes this cAdvisor per-container working-set series without compensating for the nested charge could inherit it; no autoscaler or sizing recommender was tested here. Recovering the truth means reading memory.current on the node, which means node access and a scrape path that doesn't exist by default.

Tekton's status surface can't distinguish a step that overran its own budget from a step that was killed for a child's. Both usually arrive as StepFailed with exited with code 137: Error, and memory.oom.group means the same set of Tekton processes dies either way, so nothing in the process-exit shape distinguishes them. The occasional run that does win the race and report StepOOM is no better for this purpose: it correctly says the step was OOM-killed, which is still the wrong container to go looking at. An on-call engineer reading only the TaskRun has no signal pointing at the build container, and an intermittent reason string is harder to build an alert on than a consistently wrong one.

The pod also still reserves two numbers, not one. dockerd's own PID stays in the sidecar's cgroup under the sidecar's own limit, so the 8 GiB example in the opening never collapses all the way to 4. What changes is their size: the sidecar's cAdvisor line read 26.9 MB idle and 42.8 MB while a 200 MB child ran, because the child's memory is charged elsewhere. Those measurements cover only this idle and single-child test; a real build's sidecar limit requires measurements of its image pulls, layer extraction, and BuildKit activity.

You can't drop privileged: true later as a hardening pass. Without it the sidecar reads 0::/ instead of a real cgroup path, the substitution has nothing to match, and the daemon would start with --cgroup-parent=/. dockerd never gets that far, dying on a mount failure first, but nothing in the script detects or reports the broken path either way.

Both failure modes leave a dead sidecar inside a green TaskRun. RBAC denial and the unprivileged case both end with the sidecar at exit code 1 and the TaskRun at Succeeded / All Steps have completed executing. The step ran, passed, and was never subject to the budget the pipeline thought it had. There is no annotation, condition, or event distinguishing an enforced run from an unenforced one.

There's a related Docker issue, and it's worth being precise about how it differs. moby/moby#45378, DinD cgroupv2 problem inside K8s, filed 2023-04-21 and still open, reports containers inside DinD exceeding the pod memory limit without being OOM-killed, an enforcement failure not seen on cgroup v1. That's the opposite half of what shows up here, where enforcement works, and attribution is missing. Both point to nested cgroup accounting under DinD being fragile, in different directions.

Everything above ran on a single-node kind cluster, kind v0.32.0, node image kindest/node:v1.36.1, Kubernetes v1.36.1 on Debian 13 with containerd 2.3.1, kernel 6.8.0-117-generic on aarch64, cgroup v2 unified hierarchy, and Tekton Pipelines v1.15.x. The sidecar is docker:28-dind (Docker 28.5.2, Alpine 3.22) with apk add bash kubectl jq on top, since that base image ships no bash and the entrypoint needs it. The step is alpine:3.20. The load was generated by kubectl exec into the sidecar and by running docker run against its own dockerd on a disposable local cluster, with nothing resembling production.

What to use instead

PodLevelResources is the answer when the feature is enabled on a Kubernetes 1.34-or-later cluster. It puts the limit on the pod and keeps the kubelet's accounting and the kernel's accounting pointed at the same object. PodLevelResourcesFixKubeletQOSClass and PodLevelResourcesFixUpdateDefaulting are beta and enabled by default in 1.36. Read the release notes and feature-gate reference before making a platform commitment. What it leaves unsolved is delivery: spec.resources still has to reach the pod, and the TaskRun's podTemplate allowlist won't carry it. A mutating admission webhook could patch spec.resources onto the resulting pod.

A node-level agent could set memory.max on the pod cgroup where the path and controller state are verified. It needs node access and depends on the kubelet's slice naming, which varies with the cgroup driver and cgroupRoot. This prototype did not test that alternative. It would charge the whole pod, sidecar included, which for a dind pod is often the desired boundary.

None of this is really about Tekton or Docker. cgroup.subtree_control decides which level distributes a resource. In this kind/containerd configuration, the kubelet did not delegate memory below the container scope. That made enforcement and observability separable: the kernel held the step to a budget that ordinary per-container memory metrics could not fully explain.

Pod Startup Forensics: The Tooling Gap

Bhawani Singh — Thu, 30 Jul 2026 00:00:00 GMT

kubectl get events gives raw chronology with no phase model. kubelet_pod_start_sli_duration_seconds gives per-kubelet aggregate telemetry with no per-pod label. Neither provides a per-pod startup timeline with syscall attribution. Kubernetes observability isn't short on eBPF-based tooling, though, and eBPF is exactly the mechanism you'd reach for to explain why a phase blocked. So it's worth checking, tool by tool, what each one actually does today, not what its name suggests it might do.

Tetragon: Real syscall visibility, no concept of a phase

Tetragon is Cilium's eBPF-based runtime security and observability tool. Its base installation streams process-execution and process-exit events with Kubernetes identity, node, and time metadata. With a tracing policy configured for a relevant kernel function, it can also report process and syscall-adjacent events. That's real, fine-grained, per-process attribution, and it is exactly the kind of signal you'd want during a slow init container: which operation, which process, which pod, down to the timestamp.

What Tetragon doesn't document is a pod-startup phase to attach that signal to. Each event type, process_exec, process_exit, process_kprobe, process_uprobe, process_tracepoint, is a standalone record. Correlation across events uses Tetragon's exec_id and parent_exec_id relationships; a host PID is also present, but exec_id is the documented process identifier for correlation. With a suitable tracing policy, Tetragon can report a connect()-related event with pod identity and a timestamp. It cannot place that event in an init-container or main-container startup interval without a phase model from somewhere else. Tetragon operates at the process and syscall layer; the Kubernetes lifecycle model remains a separate join. Right primitive, missing the other half.

Pixie: Deep runtime tracing, built for services that are already running

Pixie (px.dev) is a CNCF sandbox project, also eBPF-based, that automatically instruments HTTP, gRPC, DNS, and SQL traffic without code changes, builds service maps from the traffic it observes, and produces CPU flame graphs and resource profiles broken down by pod, node, and namespace. It collects, stores, and queries telemetry in-cluster; its telemetry tables use bounded in-memory storage, so retained data is intentionally short-lived.

Pixie's value proposition is instant visibility into application traffic: request/response bodies, query latencies, who's calling whom. An init container can generate protocol traffic or CPU behavior that Pixie observes while it runs. But those observations do not define schedule-to-ready intervals, distinguish a Kubernetes startup phase, or show whether an observed operation dominated that phase. Pixie can show useful fragments of a startup; its public documentation does not describe a per-pod lifecycle model that assembles schedule, pull, attach, mount, init, and readiness into one timeline.

Parca and Pyroscope: Continuous profilers, sampling code that's already executing

Parca and Grafana Pyroscope are both continuous-profiling systems, but their collection paths are not identical. Parca Agent is an eBPF sampling profiler: it captures user- and kernel-space stacks 19 times per second and discovers targets across Kubernetes and systemd automatically, with no code changes or restarts required. Grafana Pyroscope is a continuous-profiling backend that aggregates profiles collected through language SDKs, auto-instrumentation, and eBPF-based collectors. It supports language-specific profiling integrations for Go, Java, Python, Ruby, Node.js, .NET, and Rust, and correlates profiles with metrics, logs, and traces.

Both tools can answer where a process spent CPU time and how that changed over time. A continuous profile may capture startup work, especially when a process runs long enough to receive samples, but it does not by itself provide Kubernetes phase boundaries or tie sampled CPU stacks to the wall-clock delay between lifecycle events. Startup is a one-shot sequence of distinct intervals—for example scheduling, image pulling, volume work, container initialization, and readiness. Neither product's public documentation describes a Kubernetes lifecycle query that attributes a Created-to-Started interval to a named startup phase. A profiler may still collect samples while the container executes; it does not supply the missing phase boundary or the surrounding pull, attach, and mount timeline.

The generic platforms: Broad coverage, no startup-specific model

The remaining category is the generic Kubernetes observability platform: the kind that bundles logs, metrics, traces, and often eBPF-based profiling into one product and sells breadth. Metoro is representative. It combines eBPF-based kernel telemetry, APM-style dashboards and service maps, log collection, continuous CPU/memory profiling, Kubernetes events, resource data, and deployment context in one installation. That's a real, wide net. The public product material reviewed for this project does not describe a dedicated model of the schedule-to-ready sequence a pod goes through before it starts serving anything.

OpenObserve is primarily a unified logs, metrics, traces, and RUM backend. It can also deploy OpenTelemetry eBPF instrumentation that captures application traces and RED metrics without application code changes. Its public documentation does not describe a purpose-built pod-startup lifecycle model, so collecting or querying those signals alone does not produce a phase-attributed startup timeline.

Across the public documentation reviewed here, broad signal coverage does not amount to a dedicated pod-startup phase model: a per-pod schedule-to-ready timeline that attaches an observation to a named lifecycle phase. These tools are strong at explaining particular signals. The missing question is which startup phase was slow, and why.

What's actually missing

Line these up, and the shape of the gap gets sharper, not vaguer. Tetragon has process and kernel evidence but no phase model to hang it on. Pixie has deep runtime tracing but no documented lifecycle timeline. Parca and Pyroscope can show CPU behavior, including some startup work, but do not supply Kubernetes phase boundaries. The generic platforms provide breadth without a documented startup-specific model. Several could supply valuable inputs to a timeline: Events, logs, traces, profiles, and kernel observations. But turning those fragments into a durable per-pod forensic timeline still requires a lifecycle model, storage after a pod disappears, and joins that connect each fragment to a named startup interval. That's not four small gaps between four adjacent categories of tool. It's one gap, looked at from four angles.

Pod Startup Forensics: Joining Lifecycle Events to Syscall Evidence

Bhawani Singh — Sun, 02 Aug 2026 00:00:00 GMT

Tetragon can record configured syscall activity, while the Kubernetes API records lifecycle transitions. Neither record alone tells an operator which startup interval was slow and what the process did during it. The missing capability is the correlation between the two: a durable per-pod timeline that places lifecycle evidence and kernel observations on the same time axis.

Lifecycle records and syscall observations answer different questions

The Kubernetes API supplies two useful, imperfect records of startup. PodStatus records container states such as waiting, running, and terminated; Kubernetes Events may record actions such as scheduling, pulling, and starting. The components that perform those actions commonly report them, but Events are informative, best-effort data and their timestamps are not a precise phase clock. A forensic timeline has to retain the raw records it receives and make its inference rules explicit.

But an Event says that an action occurred; it does not show what a process did during the interval. In the captured slow-init run, the init-container interval lasted eight seconds, while the lifecycle records still said nothing about the process activity inside it. Kubernetes records kubelet- and control-plane-observed state; it does not watch syscalls.

eBPF observes the other side of the problem. A tracepoint on sys_enter_connect and sys_exit_connect can record that a task entered and returned from connect(), along with its timing and result. That does not make the event Kubernetes-aware. A syscall trace identifies kernel activity for a task; pod and lifecycle context live outside that event and must be resolved from container or cgroup metadata plus Kubernetes records. On its own, a trace is a wall of task identifiers and timestamps, with no indication that the activity belonged to a particular startup interval.

Neither source needs to be made into the other. Kubernetes supplies lifecycle records; eBPF supplies process activity. The missing piece is a join keyed on pod identity, with the timeline treating lifecycle boundaries as evidence of an interval and kernel observations as evidence that occurred alongside it. The comparison is useful, but it must not turn coexistence into a causal verdict: a long connect() beside a slow interval is a lead for an investigator, not proof that the call caused the delay.

The division of responsibility is specific. Kubernetes supplies the observed lifecycle records; the tracer supplies activity from selected kernel hooks. A useful output can say that a container-start interval lasted N seconds and show the connect() activity observed for the same pod during the overlapping window. Assigning that activity to a named phase, or treating it as the cause of the delay, requires explicit identity and attribution rules beyond timestamp overlap alone.

The pod that's already gone by the time you'd query it

There's a second design constraint that does not come from the tooling gap directly. Part 1 showed a captured short Tekton TaskRun: its step-hello container ran for about two seconds at the available timestamp precision. Any design that answers "which pod, which phase, why" only by querying live cluster state is blind to short-lived workloads after the fact, because by the time a human goes looking, the Pod object—and the status fields a live lookup would read—may already be gone.

This rules out an entire category of otherwise-reasonable design: a tool that watches the API and the kernel live, holds the result in memory or in a dashboard, and answers questions only against whatever is currently running. That shape works for a pod that is slow in front of you. It fails a pod that was slow earlier and has since been deleted. Kubernetes Events also have retention independent of the Pod lifecycle, so the raw material for a manual reconstruction eventually disappears on its own clock.

So durability cannot be an afterthought. The collector has to persist the lifecycle, Event, readiness, and kernel evidence it observes while the Pod still exists, keyed to a durable Pod record. That record is bounded by what the collector saw and by what the API retained, but it lets a later lookup avoid depending on a live Pod object. A live-query tool and a durable forensic record answer different operational questions.

Four requirements for a durable startup investigation

Put the two constraints together and a complete solution needs four capabilities, wired together by pod identity: lifecycle collection, selected kernel tracing, identity-aware correlation, and durable storage. Here, identity means a Pod UID, with namespace and name for display; process membership has to be resolved through container or cgroup metadata rather than inferred from a timestamp.

Keep the lifecycle record. A collector should retain the PodStatus and Event records it receives rather than rely on a fixed polling interval. It must distinguish observed status facts from inferred intervals, because it is not receiving a lossless, semantic feed of every startup transition. The next part shows the CLI, watcher, and tracer that implement this collection; Part 5 covers how the record survives deletion.

Capture kernel activity while it happens. A tracer must capture the syscall classes the investigation supports while a Pod is starting; it cannot recover an event after the fact. The capture policy has to be explicit about what it observes, the overhead it accepts, and evidence it may lose under load. This project traces openat, connect, mount, read, bind, listen, and accept4, plus process exec—not every syscall. A timeline can then present the observations relevant to the investigator's hypothesis without claiming to explain every kind of startup delay.

Match activity to the right Pod. A timestamp overlap between an API record and a kernel event is not enough on a real node: several Pods can be starting at once for unrelated reasons. The correlation step has to resolve which Pod owns a process and retain that resolution long enough to join the evidence safely. Without that, the right output is an honest unknown, not a confident association based only on simultaneous timestamps.

Keep the answer after deletion. The output cannot live only in memory or behind a live query, because the Pod may be gone before anyone asks about it. The collector should retain the observed lifecycle and kernel evidence against the Pod record so a later lookup has somewhere to go even when kubectl no longer knows the Pod.

None of these four capabilities is individually new. Kubernetes collectors retain lifecycle records. Tetragon, Pixie, and Parca can contribute different kinds of kernel or runtime evidence. Distributed systems persist observations beyond the object that produced them. The tooling in Part 2 can supply individual records, but the investigator still needs a retained per-pod record that connects them without inventing causation. Part 4 shows the collector and tracer that implement that record, including the correlation limits they still have.

Pod Startup Forensics: The Architecture

Bhawani Singh — Wed, 05 Aug 2026 00:00:00 GMT

Part 3 ended on a design: two sources of truth joined on pod identity, written before the pod disappears. This part shows what that design became when run against a real cluster: a command run by hand, a background watcher that never stops, and a tracer that talks directly to the kernel. Every claim below comes from a command actually run and checked against the tool's behavior.

One lookup path for a pod, and for anything that owns a pod

The core CLI path is Kubernetes-native: give it a Pod name and namespace, and it builds that Pod's timeline directly. The timeline builder does not know or care whether the Pod came from a Deployment, Job, CI system, or a hand-written manifest.

The resolver can be extended for resources that own Pods without changing that core path. Tekton PipelineRun is one example, chosen here because it produces short-lived workload Pods. The extension finds the TaskRun Pods carrying the PipelineRun label and passes them to the same timeline builder used for a direct Pod lookup.

Tekton changes only how the CLI finds the Pod. Once it has one, the timeline path is identical. The two captures below show the direct Kubernetes path and the extension path feeding that same timeline model.

$ ./target/release/profiler-cli lookup --kind v1.pods --name baseline-demo --namespace default
Pod startup timeline: default/baseline-demo
  image_pull                          6.000s
  init_container (init-setup)         1.000s
  sidecar (sidecar-logger)            1.000s
  main_container (main)               0.000s
  TOTAL                               8.000s

That's the direct path: a Pod with an init container, a native Kubernetes sidecar, and a main container. The CLI renders each interval to whole seconds; this capture reports a six-second image-pull interval. The extension path renders the same kind of phase timeline for a Pod found through a PipelineRun:

Pod startup timeline: default/demo-pipeline-run-run-demo-task-pod
────────────────────────────────────────────────────────────
  init_container (prepare)            0.000s
  main_container (step-hello)         1.000s
────────────────────────────────────────────────────────────
  TOTAL                               2.000s
  (pass --metrics-url http://<tracer-pod-ip-or-port-forward>:9091/metrics for eBPF root cause)

The resolver changes discovery, not timeline construction. Once it finds one or more Pods, the same Kubernetes-native timeline builder handles them all.

For this capture, Kubernetes supplied only whole-second status and Event timestamps, so the CLI's one-second display is the appropriate precision.

A container's status field has more shapes than "running" or "terminated"

Getting that exact match required noticing something kubectl describe pod doesn't explicitly call out: a regular init container and a native Kubernetes sidecar report their status differently, even though Kubernetes lists them together. After a regular init container completes, its current status is terminated. A native sidecar—an initContainers entry with restartPolicy: Always, stable in Kubernetes 1.33—is normally running while it stays up alongside the main containers, although it can later restart or terminate. A tool that only looks for terminated silently drops every sidecar from the timeline: no crash, no error, just a container that never shows up in the output. The fix is checking its restartPolicy before deciding which status to trust.

A separate problem arises in calculating phase durations. Container-status timestamps are absolute wall-clock times. For a still-running sidecar or main container, the implementation represents the gap from the latest earlier phase end to its startedAt; otherwise it keeps a terminated container's own startedAt-to-finishedAt duration. This prevents overlapping status times from being rendered as duplicated startup time.

A tracer that watches syscalls, not a wrapper around a log line

Looking up a Pod's history and building its phase timeline can answer "which phase was slow." Neither can answer the syscall-level "why": Kubernetes can report a container state, reason, or message, but it does not record what a process was doing while blocked. That's the other half of Part 3's design: a real tracer that watches the kernel directly, not a script that reads log output and guesses.

It's built with Aya, a Rust framework for writing eBPF programs, and runs on every node as its own background process with the permissions needed to watch the kernel. Fifteen tracepoints observe sched_process_exec plus enter and exit events for seven selected syscalls. Each syscall class was chosen to test a specific hypothesis about why a container might be stuck: opening a file, making a network connection, mounting a filesystem, reading data, or serving traffic through a socket. bind, listen, and accept4 are server-side socket operations; an outbound client commonly calls connect() instead. Paired enter and exit events let the tracer measure the duration of each selected syscall.

Once built, the finished program is copied onto a cluster node and loaded there, a point where many projects like this only work on paper. The captured run showed that the eBPF object attached successfully and emitted duration metrics for all seven traced syscalls. Here is what the tracer reported across several kinds of activity at once:

$ curl -s http://127.0.0.1:19191/metrics | grep pod_startup_blocking_syscall_duration_seconds_count
...,pod="readiness-slow-listen-demo",syscall="bind"} 1
...,pod="readiness-slow-listen-demo",syscall="connect"} 2
...,pod="readiness-slow-listen-demo",syscall="listen"} 1
...,pod="readiness-slow-listen-demo",syscall="mount"} 77
...,pod="readiness-slow-listen-demo",syscall="openat"} 280
...,pod="readiness-slow-listen-demo",syscall="read"} 144

Those counts prove the tracer is capturing syscall durations. They do not, by themselves, show which calls belong to the startup interval under investigation. That correlation boundary is the next problem.

Three programs, three jobs

The CLI, exporter, and tracer are separate binaries with different jobs. The CLI provides a live query and a historical query. The exporter watches and stores lifecycle evidence, then serves the historical query. The tracer captures selected syscall events and sends them to the metrics and storage paths.

For a live query, the CLI resolves a Pod from the Kubernetes API, builds its timeline, prints it, and exits. For a historical query, it instead asks the exporter for the persisted timeline of a Pod; this works after the Pods are deleted. The exporter runs continuously. It watches Pod create, update, and delete events; on a Pod update, it derives the timeline and queries retained Pod and referenced-PVC Events for persistence. The tracer runs on each node, pairs the enter and exit of its selected syscalls, and makes those events available to the metrics and storage paths.

The separation follows from their lifetimes. The captured short-lived Pod completed in about two seconds, leaving little time for an ad-hoc live lookup; once its API record is gone, only the exporter-backed historical path can reconstruct it. The exporter and tracer must keep collecting without an operator waiting at a terminal. Combining them with the one-off CLI would couple an interactive command to two always-on services with different availability and storage needs.

Disk I/O QoS for Kubernetes with cgroup v2 io.weight

Bhawani Singh — Sun, 09 Aug 2026 00:00:00 GMT

Kubernetes can request and limit CPU, memory, and ephemeral-storage capacity. It has no native Pod-level resource semantics for runtime disk-I/O bandwidth, IOPS, or latency QoS.

CPU has requests and limits. Memory has requests, limits, and OOM eviction. ResourceQuota can account for requests.ephemeral-storage and limits.ephemeral-storage, but that is disk capacity, not disk performance. A backup job can stay inside its storage quota and still consume service from the same device as a database. Neither the scheduler nor ResourceQuota arbitrates that contention.

The Linux cgroup v2 I/O controller can. This prototype uses io.weight to give two Pods an explicit, node-local priority policy: low tier gets 10; high tier gets 500. It answers a narrow question: when device service is constrained, which workload should be favored? It does not reserve bandwidth or guarantee a latency target.

The control I started with was not available

The first mechanism I checked was io.latency. It accepts a target completion latency for a cgroup and protects it by throttling peer cgroups whose own target is higher. That is closer to the usual database-versus-backup problem: protect a latency-sensitive workload from bulk I/O. It is still not a hard SLA, but it is a latency-oriented control.

It was absent from this node hierarchy. find /sys/fs/cgroup -name io.latency returned no result anywhere on the kind node, so this implementation could not use it. io.latency and iocost are separate kernel facilities; enabling iocost does not make io.latency appear.

That left io.weight. This article validates its control path and active iocost accounting. The conclusion is deliberately narrower than a benchmark claim: io.weight expresses relative I/O priority, not a reservation.

What `io.weight` actually is

io.weight is a relative share. Pod cgroups expose it only when their parent enables io in cgroup.subtree_control. The default is 100; valid values are 1 through 10000. The interface accepts a cgroup-wide default value and reports the effective value on readback:

$ cat .../pod-<uid>.slice/io.weight
default 100
$ printf 'default 500\n' > .../pod-<uid>.slice/io.weight
$ cat .../pod-<uid>.slice/io.weight
default 500

A 500:10 policy favors the high-tier cgroup over the low-tier cgroup when both compete through the same active I/O controller. It does not promise 500 IOPS, a bandwidth floor, or a completion-latency bound. The workload, request mix, backing device, and device model still determine the observed result.

The controller has separate mechanisms for separate policies:

io.weight is proportional arbitration.
io.max is a per-device rate limit. Temporary bursts are allowed.
io.latency is the latency-target mechanism.

Those distinctions are operational, not semantic trivia. A 50:1 weight ratio changes relative priority. It does not give a bulk writer a fixed latency bound. Check io.latency on the actual node image before designing around it; it is not universally available.

For iocost-backed weighting to operate on a device, the root cgroup must report that device in io.cost.qos with enable=1. Writing io.weight proves configuration. Reading io.cost.qos for enable=1 and cgroup io.stat for cost fields verifies active accounting; inspect io.cost.model to understand the model it is using.

The cgroup path is configuration, not an API

The common examples use /sys/fs/cgroup/kubepods.slice/.... That was not the path on this node. The kind node used the systemd cgroup driver with kubelet cgroupRoot: /kubelet, producing:

/sys/fs/cgroup/kubelet.slice/
  kubelet-kubepods.slice/
    kubelet-kubepods-burstable.slice/
      kubelet-kubepods-burstable-pod<uid_with_underscores>.slice

Guaranteed Pods are direct children of kubelet-kubepods.slice. Burstable and BestEffort Pods have an extra QoS-class slice. The Pod UID is part of the leaf name, with dashes replaced by underscores.

The exact path is not portable. It depends on the cgroup driver, kubelet configuration, and Pod QoS class. A cgroupfs-driver cluster uses a different directory layout. That is why the daemon accepts cgroupRoot as configuration instead of hardcoding a path.

This helper builds the target path for the systemd layout:

func PodSlicePath(cgroupRoot, qosClass, uid string) (string, error) {
	if uid == "" {
		return "", fmt.Errorf("cgroup: empty pod UID")
	}
	base, err := kubepodsBase(cgroupRoot)
	if err != nil {
		return "", err
	}

	normalizedUID := strings.ReplaceAll(uid, "-", "_")
	lowerQOS := strings.ToLower(qosClass)

	switch lowerQOS {
	case "guaranteed":
		return fmt.Sprintf("%s/kubelet-kubepods-pod%s.slice", base, normalizedUID), nil
	case "burstable", "besteffort":
		qosSlice := fmt.Sprintf("kubelet-kubepods-%s.slice", lowerQOS)
		leaf := fmt.Sprintf("kubelet-kubepods-%s-pod%s.slice", lowerQOS, normalizedUID)
		return base + "/" + qosSlice + "/" + leaf, nil
	default:
		return "", fmt.Errorf("cgroup: unknown QOS class %q", qosClass)
	}
}

A per-device `io.weight` write was rejected

The first write form I tried was the per-device form documented by the interface:

$ printf '253:16 200\n' > .../low-priority.../io.weight
sh: 7: printf: I/O error
$ cat .../low-priority.../io.weight
default 10

The write failed and the file retained its prior value. The cgroup-wide form succeeded on the same cgroup:

$ printf 'default 150\n' > .../pod-<uid>.slice/io.weight
$ cat .../pod-<uid>.slice/io.weight
default 150

The daemon therefore writes only the cgroup-wide form. The node used mq-deadline, not BFQ, but that alone does not explain why the kernel returned EIO. The useful fact is simpler: on this node and device, per-device syntax was rejected while the cgroup-wide syntax applied and read back correctly.

The daemon

The prototype polls two named Pods. It reads an io-qos.demo/tier annotation (high or low), maps it to a weight, resolves the cgroup path, writes the value, and reads it back.

func (d *Daemon) reconcileOne(ctx context.Context, t Target) error {
	pod, err := d.Client.CoreV1().Pods(t.Namespace).Get(ctx, t.Name, metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("get pod: %w", err)
	}

	if pod.Status.Phase != corev1.PodRunning {
		log.Printf("qosd: %s/%s: phase=%s, waiting", t.Namespace, t.Name, pod.Status.Phase)
		return nil
	}

	tier, ok := pod.Annotations[TierAnnotation]
	if !ok {
		return fmt.Errorf("missing annotation %s", TierAnnotation)
	}
	weight, ok := TierWeight[tier]
	if !ok {
		return fmt.Errorf("unknown tier %q", tier)
	}
	if applied, ok := d.applied[string(pod.UID)]; ok && applied == weight {
		return nil
	}

	slicePath, err := cgroup.PodSlicePath(d.CgroupRoot, string(pod.Status.QOSClass), string(pod.UID))
	if err != nil {
		return fmt.Errorf("resolve cgroup path: %w", err)
	}
	written, readback, err := cgroup.WriteWeight(slicePath, weight)
	if err != nil {
		return fmt.Errorf("write weight: %w", err)
	}
	log.Printf("qosd: %s/%s uid=%s tier=%s qos=%s path=%s wrote=%q readback=%q",
		t.Namespace, t.Name, pod.UID, tier, pod.Status.QOSClass, slicePath+"/io.weight",
		trimNL(written), trimNL(readback))
	d.applied[string(pod.UID)] = weight
	return nil
}

The write helper is intentionally boring. The readback matters because a successful-looking control loop is useless if the kernel rejects the requested form.

func WriteWeight(slicePath string, weight int) (written string, readback string, err error) {
	line, err := WeightLine(weight)
	if err != nil {
		return "", "", err
	}

	weightFile := slicePath + "/io.weight"
	if err := os.WriteFile(weightFile, []byte(line), 0644); err != nil {
		return line, "", fmt.Errorf("cgroup: write %s: %w", weightFile, err)
	}

	data, err := os.ReadFile(weightFile)
	if err != nil {
		return line, "", fmt.Errorf("cgroup: read back %s: %w", weightFile, err)
	}
	return line, string(data), nil
}

The live daemon runs inside the kind control-plane node. That placement gives it access to the node cgroup hierarchy and the Kubernetes API credentials it needs to resolve Pods. It is not a DaemonSet or a general cluster controller; this is a focused one-node prototype.

Validating the policy path

The test used two Burstable Pods with direct-I/O fio jobs against a shared hostPath on the kind node. Low tier wrote sequential 1 MiB requests with four jobs. High tier issued 4 KiB random writes with one job. The request sizes differ by 256:1, so their IOPS and bandwidth are not directly comparable. They are different workload shapes sharing the same target.

The test sequence was equal weights, weighted policy, equal weights again, then weighted policy again. Before and after each arm it captured the two io.weight files and io.stat. The test VM root cgroup reported iocost enabled for the device.

Here, charged-cost share means the high-tier cgroup's cost.usage delta divided by the combined high-tier and low-tier deltas. I use it because this test validates controller behavior, not storage performance. It is controller accounting, not bandwidth, IOPS, or latency.

Arm	High / low `io.weight`	High-tier charged-cost share
Equal weights, pair 1	100 / 100	48.90%
Weighted, pair 1	500 / 10	60.10%
Equal weights, pair 2	100 / 100	5.87%
Weighted, pair 2	500 / 10	17.52%

Across the two ordered pairs, the high-tier share moved upward after the 500:10 setting. That is consistent with the configured priority policy. It is not a throughput or latency result.

The test found the live Pod cgroups, applied each weight, read it back, and captured iocost accounting before and after the workload. The daemon log separately records the same resolution and write/readback path. That distinction matters: the workload capture shows the cgroup control state; the daemon log shows the daemon's own reconciliation path.

The diagram summarizes configured policy, not measured throughput:

sequenceDiagram
    participant Low as low-tier Pod<br/>(1 MiB sequential write)
    participant Dev as shared block device
    participant High as high-tier Pod<br/>(4 KiB random write)

    Note over Low,High: equal weights: 100 / 100
    Low->>Dev: bulk I/O
    High->>Dev: competing I/O

    Note over Low,High: weighted policy: 10 / 500
    Low->>Dev: lower configured share
    High->>Dev: proportionally favored when constrained

The point of the test is the control boundary. io.weight changes relative priority. It does not establish a throughput number that transfers to another device, filesystem, workload mix, or iocost model. A capacity decision needs repeated randomized arms, synchronized starts, a calibrated model, and the storage stack where the workload will run.

`io.max` is a second lever

io.max handles a different policy. It takes a backing device major:minor and BPS or IOPS limits:

$ printf '253:16 wiops=200\n' > .../low-priority.../io.max
$ cat .../low-priority.../io.max
253:16 rbps=max wbps=max riops=max wiops=200

253:16 is the device identifier from this node. Derive it from the workload storage path before applying a cap on another node.

This is useful for a known offender, such as a bulk job whose write rate must be contained. It can be combined with io.weight: use a rate limit to contain the bulk job, then use weights to arbitrate remaining contention. Neither one creates a latency guarantee.

Scope and production requirements

This prototype runs on one kind node, one virtual block device, and two Pods. It does not coordinate Pods across nodes. It does not establish what a network filesystem, NFS share, or iSCSI volume will do. Tier assignment is annotation-driven and targets two named Pods, not a cluster-wide policy API.

Turning this into a platform feature requires more than moving the code into a DaemonSet:

Verify cgroup v2 and io delegation on every node type.
Resolve the actual Pod cgroup layout instead of assuming this systemd path.
Map the workload storage path to its backing device.
Check that iocost is active for that device where proportional enforcement is required.
Select the control by policy: arbitration (io.weight), rate limit (io.max), or latency target (io.latency).
Validate the chosen policy against the real storage stack and workload mix.

The useful conclusion is narrow. Kubernetes does not provide disk-I/O QoS as a Pod resource, but a node agent can use cgroup v2 io.weight to impose explicit proportional priority on live Pod cgroups. Use it for arbitration. Use io.max when a workload needs a rate limit. Use io.latency only on nodes where that latency control exists.

Pod Startup Forensics: Surviving Deletion

Bhawani Singh — Sat, 15 Aug 2026 00:00:00 GMT

A pod that started slowly is often gone by the time anyone asks why. kubectl describe returns NotFound; its Events may linger briefly, but on a retention schedule unrelated to the pod itself. The evidence needed to explain the startup has become a race against deletion and garbage collection.

To answer after deletion, this exporter has to capture the evidence while the pod exists and reconstruct it from durable storage later. This part shows the storage design behind that reconstruction and the --timeline command that reads it back.

Capture it before deletion

The exporter writes the observations it needs to reconstruct a startup to a database. Postgres is the deployment choice for shared, durable history; SQLite offers the same --timeline behavior in one local file for development or a single-node demo. With either storage option, the owner, child, pod, phase, and event records form one hierarchy, so a deleted pod can still be found under its original PipelineRun, Job, or CronJob.

What actually gets stored

For the deletion reconstruction in this part, two stored record types carry the core answer: phase records for duration and an ordered log of Kubernetes Events for explanation. Phase records retain timings that Events can't provide, especially container start and finish times. Events retain the verbatim reasons, messages, and retry history that explain delays outside those phase boundaries. --timeline joins both when it reconstructs a pod.

The output preserves the precision it actually captured. A populated Event eventTime can carry microseconds, while the older firstTimestamp and lastTimestamp fields are whole-second values; --timeline doesn't invent fractional precision for the latter.

Reconstructing a PipelineRun after its pods are deleted

The payoff is a timeline that remains readable after the pods are gone. A three-TaskRun Tekton PipelineRun produced three pods on a kind cluster. One deliberately blocked for about thirteen seconds against a non-routable address before giving up, giving the timeline a known slow phase to find. All three pods were then deleted directly:

$ kubectl delete pod -l tekton.dev/pipelineRun=multi-taskrun-run-slow
$ kubectl get pods -l tekton.dev/pipelineRun=multi-taskrun-run-slow
No resources found in default namespace.

kubectl has nothing left to say about any of these three pods. A direct query against Postgres confirms the same thing from the storage side: all three pod rows carry a non-null deleted_at, with their phases and events still attached:

         owner          |                   child                    |                      pod                       | deleted | phases | events 
------------------------+--------------------------------------------+------------------------------------------------+---------+--------+--------
 multi-taskrun-run-slow | multi-taskrun-run-slow-first-task-run      | multi-taskrun-run-slow-first-task-run-pod      | t       |      3 |      7
 multi-taskrun-run-slow | multi-taskrun-run-slow-second-task-run     | multi-taskrun-run-slow-second-task-run-pod     | t       |      4 |     18
 multi-taskrun-run-slow | multi-taskrun-run-slow-third-task-run-slow | multi-taskrun-run-slow-third-task-run-slow-pod | t       |      3 |      7
(3 rows)

With no matching live pods in the PipelineRun's namespace, --timeline reconstructs the stored history alone:

$ profiler-cli lookup --kind tekton.dev/v1.pipelinerun \
    --name multi-taskrun-run-slow --namespace default --timeline \
    --exporter-url http://127.0.0.1:19090

Owner historical timeline: default/multi-taskrun-run-slow
(reconstructed from durable storage — works even if every pod below is now deleted)
══════════════════════════════════════════════════════════════════════

Child: multi-taskrun-run-slow-third-task-run-slow
  Pod: multi-taskrun-run-slow-third-task-run-slow-pod  [deleted]  node=pod-startup-profiler-control-plane
  ──────────────────────────────────────────────────────────────────
    schedule                            0.335s
    init_container (prepare)            0.000s
    main_container (step-blocking-step)   13.000s <-- SLOWEST PHASE IN PIPELINE
  ──────────────────────────────────────────────────────────────────
    TOTAL                              14.000s
    ...

══════════════════════════════════════════════════════════════════════
FLAGGED SLOW: main_container (step-blocking-step) in Child multi-taskrun-run-slow-third-task-run-slow / pod multi-taskrun-run-slow-third-task-run-slow-pod took 13.000s (longest phase across the whole owner chain)

All three TaskRuns reconstruct with their pods labeled [deleted]; the excerpt shows the deliberately blocked 13-second step as the owner chain's longest phase. A second read returned the same bytes, and a fresh three-pod run reproduced the same slow-phase flag after deletion.

When no container ever starts

The deleted PipelineRun shows that completed startup history survives deletion. A pod blocked on an unbound PersistentVolumeClaim never records a container phase, so there is no started/finished pair to time. The exporter used to drop that shape entirely. Now --timeline renders no phases recorded and shows the captured Event log.

No container started, so this timeline has no syscall or probe records. The CLI keeps its standard merged-timeline heading, but only captured Events follow:

Pod: pvc-failure-demo  [exists]  node=?
──────────────────────────────────────────────────────────────────
  no phases recorded — this pod never reached container start; see the event log below for why
──────────────────────────────────────────────────────────────────
  TOTAL                               0.000s

  MERGED TIMELINE (K8s Events + eBPF syscalls/probes, interleaved by real timestamp)
  2026-08-30T04:13:26Z  T+  0.000s  [K8S EVENT]  ExternalProvisioning   PersistentVolumeClaim/rook-ceph-stuck-pvc (x2 repeats)
      Waiting for a volume to be created either by the external provisioner
      'rook-ceph.rbd.csi.ceph.com' or manually by the system administrator. If volume creation
      is delayed, please verify that the provisioner is running and correctly registered.
  2026-08-30T04:13:26.572306Z  T+  0.572s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
  2026-08-30T04:13:26.578279Z  T+  0.578s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found

That output names the provisioner handling the claim and records the scheduler's immediate blocker: an unbound immediate PersistentVolumeClaim. It identifies the next component to inspect and explains why the pod can't be scheduled. A phase-durations-only view has no row for this pod, regardless of how long it waits.

This capture stores only two repeats. A stored count can understate a persistent failure because the exporter refreshes the Event set only when it observes the pod; later Event changes can be missed if no further pod observation occurs. This run alone doesn't show that happened here.

The previous case retained phase data; this pod never reached container start. The same lookup --timeline interface covers both shapes.

Open questions

This test doesn't answer whether the database keeps up under sustained production load.

Part 6 follows a flagged slow phase into eBPF syscall evidence, then fixes a bug in assigning those syscalls to the right pod.

Pod Startup Forensics: Closing the Gap

Bhawani Singh — Sun, 16 Aug 2026 00:00:00 GMT

Part 5 closed the durability gap: evidence captured while a pod exists, readable after it's gone. This part closes a different gap. In the slow-start capture, Pulled and Started appear, followed by one coalesced Unhealthy record before Ready. The event log has little to say about what happened between those records.

Kubernetes' blind spot

A crash-looping container produces Events such as BackOff. A container that will become healthy but is slow to start may produce only a coalesced Unhealthy Event: the API records a repeat count instead of one Event for every failed readiness probe. kubelet continues polling throughout. Event aggregation, not kubelet going quiet, is why the timeline looks sparse.

The eBPF tracer can see work in that gap, but its evidence was not yet part of the timeline. It captures syscalls whether or not Kubernetes emits an Event. Closing the gap meant merging those socket syscalls and kubelet's probe attempts with the Kubernetes record.

Filling the gap

This part explains how the tracer correlates and renders those syscalls, rather than how it hooks them; Part 4 covers the tracepoints. Three pieces make up the work:

What the socket syscalls mean for a readiness probe. For a server that listens on a TCP port, listen() after bind() marks the point at which a TCP-based readiness probe can succeed.

kubelet probe correlation. The correlation path covers TCP or HTTP readiness probes against the pod's own IP. A probe with an overridden host and every exec probe are out of scope. kubelet makes the observed connect() call from its own process, so the tracer identifies kubelet calls and joins their destination address to a pod IP instead of attempting cgroup attribution.

A merged, filtered rendering. The timeline interleaves syscalls, probe attempts, and Kubernetes Events by timestamp. Its default view collapses sub-10-millisecond openat()/mount()/read() bookkeeping into summary lines; --raw retains every event.

`connect()` isn't readiness

Building probe correlation surfaced a real mistake in how a connect() return code was being interpreted. kubelet's HTTP and TCP readiness probes use Go's non-blocking socket dial, so the connect() syscall itself can return EINPROGRESS ("in progress, ask again later") regardless of whether the probe eventually succeeds or fails. Treating an EINPROGRESS return as succeeded = true, on the reasoning that a non-blocking call proceeding normally counts as success, doesn't hold up.

A cluster capture falsified that interpretation: all 21 observed HTTP/TCP readiness-probe connect() calls returned EINPROGRESS, including probes that Kubernetes recorded as Unhealthy. The syscall therefore cannot say whether the probe later succeeded or failed; that outcome is determined by a follow-up call the tracer does not hook.

The fix treats succeeded as true only for an immediate 0, failed only for a real error code other than EINPROGRESS, and everything else, the common case, as pending. The authoritative signal for whether a pod actually became ready stays exactly where it belongs: Kubernetes' own Ready status condition, fetched directly from the API rather than re-derived from the probe attempts the tracer observed.

The timeline

The test container sleeps for twelve seconds before opening a listening socket, standing in for real startup work. Its readiness probe begins after one second and runs every two seconds. The container itself never crashes and the image never fails to pull; its readiness probe does fail, repeatedly, until the socket actually opens, which is expected. That shape, slow to become ready but never crashing, is different from every crash or stuck-image case the series has covered so far.

Each row below shows a wall-clock timestamp, an elapsed-time offset from pod creation, a source tag (a Kubernetes Event, the pod's Ready condition, an eBPF-observed kubelet probe attempt, or an eBPF-observed syscall), and what happened:

2026-08-22T20:08:12Z          T+  1.000s  [K8S EVENT]   Started      Container started
2026-08-22T20:08:14Z          T+  3.000s  [K8S EVENT]   Unhealthy    (x5 repeats over 8.000s)
    Readiness probe failed: dial tcp 10.244.0.21:8080: connect: connection refused
2026-08-22T20:08:16.988402Z   T+  5.988s  [EBPF PROBE]  kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS]
2026-08-22T20:08:18.987952Z   T+  7.987s  [EBPF PROBE]  kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS]
2026-08-22T20:08:20.988120Z   T+  9.988s  [EBPF PROBE]  kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS]
2026-08-22T20:08:22.988318Z   T+ 11.988s  [EBPF PROBE]  kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS]
2026-08-22T20:08:24Z          T+ 13.000s  [K8S READY]   Ready=True (from pod status, not derived from probe attempts above)
2026-08-22T20:08:24.228734Z   T+ 13.228s  [EBPF SYSCALL] bind()  -> [::]:8080  duration=0.000s
2026-08-22T20:08:24.228750Z   T+ 13.228s  [EBPF SYSCALL] listen()             duration=0.000s
2026-08-22T20:08:24.988520Z   T+ 13.988s  [EBPF PROBE]  kubelet probe connect() -> 10.244.0.21:8080 FAILED [EINPROGRESS]

This capture predates the correction, so its FAILED label is itself evidence of the bug; the corrected build renders those events as PENDING. The probe rows fill in attempts that Kubernetes coalesced into one Unhealthy Event. The API records Ready=True only to whole-second precision. Its displayed T+13.000 value therefore cannot order it relative to the microsecond eBPF rows. The later FAILED probe is likewise a pre-fix rendering error: that EINPROGRESS result was pending, not a failed readiness check.

Filtering makes the timeline usable: a comparable slow-start capture fell from 857 lines to 225, collapsing 643 routine syscalls into eleven summary lines while retaining every Kubernetes Event, probe attempt, and socket syscall.

Attribution race

Correlating socket syscalls exposed a cgroup-attribution race. Only 22.2% of bind() calls resolved to a pod cluster-wide; the cgroup index refreshes every three seconds, but a lookup miss was finalized as unknown immediately. Startup-time socket calls are rare and early, so a call can arrive before the index catches up and, before this change, never get another chance to resolve.

Retry on misses

On a lookup miss, the tracer now holds the event and retries after each index refresh for up to twelve seconds, then uses the existing unknown path. Four refresh cycles let an event survive a missed refresh without retaining it indefinitely. System processes and static control-plane pods still remain unresolved; no retry can join their cgroups to a workload pod.

Results

An unfixed reproduction captured a successful bind() for a cgroup known to belong to the test pod, yet stored it as unknown; it remained unresolved forever. After the retry change, every captured bind() and listen() event for a resolvable workload-pod cgroup received an identity: 195 bind() events and 17 listen() events. The unfiltered cluster-wide bind() rate rose more modestly, from 22.2% to 30.0%, because system, stale, and static-control-plane cgroups are not resolvable by this mechanism.

Isolation test

A retry-and-hold buffer introduces a new failure mode worth checking for directly: if two pods are racing through startup at the same time, could a delayed resolution attribute one pod's syscall to the other? The isolation test started two pods in the same second with deliberately different syscall signatures: one ran a file-read loop; the other ran a socket-connect loop that blocked for about a second per attempt.

    pod_name        | syscall | count |    sum(duration)
---------------------+---------+-------+-----------------------
 isolation-pod-a     | connect |     2 |          3.2709e-05
 isolation-pod-a     | openat  |  2048 |          0.026864375
 isolation-pod-a     | read    |  1034 |          0.032389957
 isolation-pod-b     | bind    |    13 |          0.000282291
 isolation-pod-b     | connect |    14 |          12.000398686
 isolation-pod-b     | openat  |   251 |          0.022781171
 isolation-pod-b     | read    |   129 |          0.019244945

Neither pod's syscalls appeared under the other's identity. The capture verifies final attribution, not an uninterrupted event-by-event trace: the tracer restarted after reaching its configured memory limit during the test.

Remaining limits

One concrete gap remains: the correlator cannot match static control-plane pods to their Kubernetes mirror objects because their identifiers differ.

The slow-start capture exposes the previously opaque interval and keeps probe outcomes separate from connect() observations. In the two-pod isolation test, retry resolution didn't cross-attribute the concurrently starting pods.

Part 7 closes the series by assessing the remaining production questions, including sustained database writes and the tracer's memory ceiling.

Pod Startup Forensics: Closure and What's Next

Bhawani Singh — Thu, 20 Aug 2026 00:00:00 GMT

Part 6 fixed the last correctness bug this series set out to find: a cgroup-lookup race that left most socket syscalls unattributed to a pod. This part closes the series itself: what six parts of building actually proved, and what remains an open question rather than a finding.

What got built

The problem, back in Part 1, was that a slow pod startup produces almost no explanation of where the time went. Kubernetes Events are coarse, and their retention is controlled by the cluster rather than by a durable forensic store; kubelet's own telemetry doesn't say why a probe kept failing. Part 2 checked whether an existing tool already closed that gap and found that eBPF observability platforms, continuous profilers, and generic APM tools each solve a related but different problem, none of them putting pod lifecycle records and kernel-level syscall evidence on one shared timeline.

The design that followed, in Part 3, was to capture both kinds of evidence and join them by pod identity and timestamp rather than treat them as separate systems a person has to correlate by hand. Part 4 split that design into three cooperating programs: a CLI that queries stored evidence and renders a timeline, an exporter that watches the Kubernetes API and writes lifecycle records, and an eBPF tracer that hooks socket syscalls in the kernel and streams what it observes.

Two problems remained once those pieces existed. The first was durability: a pod that has already been deleted is exactly the pod someone is most likely to ask about, and by then kubectl describe returns nothing. Part 5 added persistent storage and a --timeline command that reconstructs a deleted pod's full startup history from Postgres or SQLite alone, verified by deleting a real three-pod pipeline's pods and rebuilding their timeline afterward with no gaps.

The second was correctness in the merge itself. Building the correlation path that joins eBPF syscalls to the HTTP and TCP probe attempts visible through kubelet's own connect() calls surfaced two real bugs rather than confirming the design worked on the first try: a connect() return code that was being read as a success signal when it was actually a non-blocking call still in progress, and a cgroup-lookup race that left the large majority of syscalls unattributed to any pod because a lookup miss was treated as final instead of retried. Part 6 fixed both: in the fixed run, every captured bind() and listen() event for a resolvable workload-pod cgroup received an identity, 195 bind() events and 17 listen() events, up from a cluster-wide pre-fix rate where most such events landed as unattributed. A concurrent two-pod test then verified the fix doesn't cross-attribute one pod's syscalls to another racing alongside it.

That's what six parts actually established: a working phase-attribution pipeline that matches Kubernetes' own lifecycle ground truth, a syscall-to-pod correlation path verified isolated under concurrent load, and a durable storage layer that survives pod deletion, each backed by a captured before-and-after run rather than a description of the intended behavior.

What was never claimed

None of that's the same as knowing how the system behaves at production scale. Every test in this series ran against a single-node kind cluster with a handful of pods at a time. Two scaling questions matter enough to name directly instead of leaving implicit, and neither one has been answered by anything this series actually did.

Postgres under sustained write load

The tracer records each correlated syscall event to a local buffer on the node and forwards it to the exporter, which attempts to persist it to Postgres or SQLite. That local buffer exists precisely because the forwarding path isn't guaranteed to keep up: on a busy node during this series' earlier storage testing, unrelated background syscall traffic drove that local backlog past 60,000 unsent rows, and replaying all of it in one allocation on reconnect contributed to a real OOM kill, a separate incident from the memory ceiling discussed below. The fix bounded the replay to 2,000-row chunks rather than one unbounded read. The code's own comment on that fix names the exporter's Postgres pool as a suspected contributor to why the backlog built up, but nothing in this series independently measured Postgres itself while that happened: no pool metrics, no connection counts, no insert-latency capture. The confirmed finding is the local buffer's own unbounded-replay bug, now fixed; whether Postgres write throughput was actually a bottleneck that day, as opposed to just downstream of a noisy node, was never isolated and measured on its own.

That distinction matters for what's still genuinely untested: sustained insert throughput, connection pool behavior, and query latency under deliberate, sustained concurrent write pressure, the kind a cluster running hundreds of pods starting at once would produce. Every test in this series that touched Postgres checked correctness, not load: did the right rows land, did a --timeline reconstruction after deletion match what actually happened, did a regression run produce output identical in shape to the original. None of them measured the database under sustained write volume. A platform team adopting this design for that kind of cluster would be the first to find out whether the write path holds up, because nothing here ever generated that load on purpose or measured it directly. Postgres write-volume behavior under sustained high-frequency ingestion remains untested as its own question, separate from the local-buffer bug this series already found and fixed.

The tracer's syscall-rate ceiling

The eBPF tracer, unlike the Postgres question, does have a real finding behind it: the tracer restarted mid-test after hitting its configured memory limit during Part 6's two-pod isolation test, and that restart is worth the detail here as a real, measured limit rather than a footnote.

That test needed two pods generating deliberately different, high-volume syscall patterns at the same time, to prove the cgroup-retry fix wouldn't cross-attribute one pod's events to the other. The first attempt ran both workloads as fast as the shell would allow, with no pacing. The tracer was OOMKilled, confirmed on the node with dmesg, showing the tracer process killed for exceeding a 384Mi memory limit set on its container, with anonymous resident memory around 391MB at the moment of the kill (the quoted dmesg line survives in this project's running build notes; no separate raw capture file was saved for it, unlike the numeric captures elsewhere in this series). The test had destroyed its own instrument before it could produce a result.

No allocation profiler ran during the OOM, so what actually consumed the memory was never measured directly. What's known is that the run exceeded its budget. The kernel ring buffer that delivers raw events is fixed in size, and the outbound gRPC stream is bounded too, so a higher event rate fills those and causes stalling or dropped events rather than growing their own allocation without limit. The more plausible candidates are the parts of the pipeline that aren't capacity-bounded: the unbounded in-process handoff that moves each event from the ring-buffer reader to the correlator, and the correlator's own in-memory state, the map of syscalls entered but not yet matched to their exit, and the retry buffer holding events whose cgroup lookup hasn't resolved yet, both of which can grow with however many events are in flight at once rather than draining at a fixed rate. The correlator also holds a separate in-memory index mapping every pod and container cgroup on the node to its identity, refreshed every few seconds, so pod and container count are candidate contributors too, not just event rate, and this test didn't isolate one from the other.

The two unthrottled loops in this test were deliberately artificial rather than production-shaped; this series never measured how their syscall rate compares with a real pod startup's. The response was to pace the test workloads rather than raise the memory limit again to make the symptom go away. With pacing, the same two-pod test completed correctly, with no cross-attribution between the two pods, which is the finding Part 6 actually reported; the tracer still restarted once mid-test after hitting the same limit, which is why that capture verifies final attribution rather than an uninterrupted trace.

The honest framing of that result is narrower than "the tracer has a memory problem," and narrower still than a general operating threshold. For this specific artificial workload against this specific container's 384Mi budget, the tracer hit a real ceiling, not evidence that its memory usage is unbounded or unpredictable in ordinary operation. What it doesn't establish is where that ceiling sits for a production-shaped workload, or against a different memory budget, because the workload that hit it was constructed deliberately to be unrealistic, not sampled from a real deployment. Where the safe operating envelope actually is, and whether 384Mi is a reasonable default cgroup limit for a real cluster's pod-startup traffic rather than just this test's own two artificial loops, is future validation work, not something this series measured.

What this series was and wasn't

What six parts actually produced is a working mechanism for joining Kubernetes lifecycle records to kernel-level syscall evidence, proven correct at the scale it was tested at, because each fix was caught by testing against a real cluster instead of trusted on paper. The questions that only show up at a different scale, or on infrastructure this series never touched, are named here rather than left implicit: sustained database write volume, the tracer's own memory ceiling, exec-probe correlation, and eBPF portability to a production node kernel.

The next work is validation: load-test sustained database writes and tracer memory under realistic multi-pod, multi-node startup traffic; verify the tracer on production kernels; and add a correlation path for exec probes. Those tests should report observed behavior, not assumed design intent.

I’ll continue evolving the tool and, when it’s ready for production use, I plan to release it under an open-source license so others can evaluate it, adapt it to their clusters, and contribute improvements.