# One cgroup budget for a Tekton taskrun's step and its dind sidecar A Tekton `TaskRun` that builds container images runs as one pod holding one or more step containers, alongside a privileged Docker-in-Docker sidecar. A step issues `docker build` and `docker run` against the sidecar's daemon socket. The dockerd process inside the sidecar is what actually does the work: pulling layers, running `RUN` instructions, spawning child containers. During a build, the sidecar is the busiest thing in the pod. If the step and the sidecar each need a 4 GiB peak at different times, separate limits reserve 8 GiB against node capacity even though the combined workload peak is 4 GiB. A shared ceiling would fit that usage. The step and the sidecar sit in sibling cgroups under the same pod slice. The entrypoint swaps the sidecar hash for the step hash, then passes the resulting path to dockerd as `--cgroup-parent`. dockerd remains in the sidecar cgroup; containers it creates start beneath the step cgroup. Their memory is charged to the step's limit, and the kernel holds that limit through an OOM kill. The same nesting that makes it work is what erases the build container from the metrics every memory dashboard is built on. ## The problem The unit of budget you want here is the pod: give this whole `TaskRun` 4 GiB, let the step and the build children draw from it, and stop paying for a peak that never happens. Kubernetes has that concept. `PodLevelResources`, [KEP-2837](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2837-pod-level-resource-spec/README.md), sets `spec.resources` on the pod itself. It went beta and default-on in 1.34. In 1.36, `InPlacePodLevelResourcesVerticalScaling` graduates to beta and is enabled by default. Two pod-level-resource bug fixes, `PodLevelResourcesFixUpdateDefaulting` and `PodLevelResourcesFixKubeletQOSClass`, are beta and enabled by default in 1.36. But Tekton can't use it. A `TaskRun`'s `podTemplate` accepts a fixed subset of Pod-spec fields, and that subset excludes `resources`. Tekton's `computeResources` field configures individual containers; it cannot set `spec.resources` on the resulting Pod. So a platform team that wants one number per `TaskRun` pod is stuck between a Kubernetes feature that's still beta and a Tekton API surface that doesn't expose it. ## The options Four candidates get the step and the build sharing one ceiling. **PodLevelResources.** Set the budget on the pod and let the kernel divide it. Correct, but unavailable in Tekton's `podTemplate` and still beta. **PID migration via `cgroup.procs`.** Write the sidecar's own dockerd PID into the step container's `cgroup.procs`, moving the daemon itself under the step's budget. This charges dockerd's own footprint, including image pulls and layer extraction, to the step, which is a different and much blunter thing than charging the build's children. **`--cgroup-parent` borrowing.** Start dockerd with `--cgroup-parent` set to the step container's scope, so containers that do not override that setting are nested inside the step's cgroup, while dockerd itself stays where the kubelet put it. **Writing `memory.max` into the pod slice.** Have something with node access compute a budget and write it directly into `kubelet-kubepods-*-pod.slice/memory.max`, bypassing the API entirely. I tested `--cgroup-parent` borrowing. Of the four designs considered, it is the one this prototype validates inside a normal pod while targeting build children rather than the daemon that spawns them. ## The chosen method The sidecar's entrypoint runs before dockerd and does four things: read its own cgroup path, identify its own pod, ask the API server for the step container's containerd ID, and substitute that ID into the path it read. The critical property is that dockerd's own PID never moves. The daemon stays in the sidecar's cgroup, where the kubelet put it and where the kubelet's own accounting expects it. Only the containers it goes on to create land under the step. Step one reads the sidecar's own cgroup from procfs, which under the cgroup v2 unified hierarchy, is a single `0::` line: ```shell $ awk -F: '/^0::/ {print $3}' /proc/self/cgroup /kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podcd57a787_d809_4a55_b6f8_ab38de73aeb8.slice/cri-containerd-4354439d2f1668c37baaabaa5a33b5ce71a1b24c182e4ebe0d16455889c14617.scope ``` Steps two and three read the pod name from `/etc/hostname` and the namespace from `/var/run/secrets/kubernetes.io/serviceaccount/namespace`, then poll the API server for the step container's ID, filtering out the sidecar's own entry: ```shell kubectl get pod "$podname" -n "$namespace" -o json | \ jq -r 'first(.status.containerStatuses[] | select(.name != "sidecar-dind" and .started == true) | .containerID)' | \ awk -F'://' '{print $2}' ``` Step four is the whole trick. The sidecar's cgroup path and the step's cgroup path are siblings under the same pod slice, differing only in the container hash, so one substitution converts one into the other: ```bash NEW_CG_PATH=$(echo "$CGROUP_PATH" | sed -E "s/cri-containerd-[a-f0-9]+\.scope/cri-containerd-$CONTAINERD_HASH.scope/") ``` Then dockerd starts with that path as its parent: ```bash exec /usr/local/bin/dockerd-entrypoint.sh \ --cgroup-parent="$NEW_CG_PATH" \ "$@" ``` The sequence, after the step has started and its container ID is available: ```mermaid sequenceDiagram participant Init as sidecar entrypoint participant API as kube-apiserver participant D as dockerd (sidecar cgroup) participant Step as step container participant CG as step .scope cgroup Init->>Init: read /proc/self/cgroup Init->>API: get pod, read step containerID API-->>Init: containerd:// Init->>Init: sed sidecar hash -> step hash Init->>D: exec dockerd with --cgroup-parent= Step->>D: docker run / docker build D->>CG: create child cgroup under step scope Note over CG: child's memory charges to it and its step ancestor ``` One detail makes the substitution viable, and it's worth checking before copying any of this. The two sides of the pod run different cgroup drivers: ```shell $ kubectl exec -c sidecar-dind -- docker info --format '{{.CgroupDriver}} / v{{.CgroupVersion}}' cgroupfs / v2 $ kubectl get --raw /api/v1/nodes//proxy/configz | jq -r .kubeletconfig.cgroupDriver systemd ``` The kubelet builds the systemd-style `.slice`/`.scope` paths that the sidecar reads from procfs, while dockerd inside the sidecar mounts `cgroupfs` and treats `--cgroup-parent` as a literal directory to create beneath. That mismatch is what lets a path lifted from the kubelet's hierarchy be handed to dockerd unchanged. A dind daemon configured with the systemd driver would expect a slice name and derive its own scope instead, and the borrowed path wouldn't nest the same way. Two prerequisites beyond that. The sidecar's service account needs to `get` on `pods` in its own namespace, because the container ID is only available through the API. And the sidecar needs `privileged: true`, both for dockerd's normal reasons and because without a host cgroup namespace the container reads only the namespace-relative `0::/` and has no host path to rewrite. ## What the kernel does with it The nesting is literal. After the step tells dockerd to run a container that writes a 200 MB file, that container's cgroup appears as a subdirectory of the step's scope on the node: ```shell $ ls .../cri-containerd-84c3e193...abe1.scope/ 4c4095d78188d1ba957135bd72bf672eec1de8a40ed8e011e3b30fb1a811d247 cgroup.controllers cgroup.subtree_control ... memory.current memory.max ... ``` The kernel charges it to the step. Reading the step scope's `memory.current` directly on the node, with a 200 MB child running: ```shell $ cat .../cri-containerd-84c3e193...abe1.scope/memory.current 213598208 ``` That's roughly 203.7 MiB against a step container whose own resident footprint before the child started was about 7 MB. The charge landed exactly where the mechanism aimed it. That happens because of one file. A cgroup can distribute a resource to child cgroups only when it enables that controller in `cgroup.subtree_control`. The step's scope lists `memory` as available without enabling it: ```shell $ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.controllers cpuset cpu io memory hugetlb pids rdma misc $ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.subtree_control cpuset cpu pids $ cat .../4c4095d78188...247/cgroup.controllers cpuset cpu pids $ cat .../4c4095d78188...247/memory.current cat: ...: No such file or directory ``` So the child gets no `memory.*` files at all. It is not a memory-accounting boundary, so memory instantiated by its processes is accounted at the step's `.scope`. The mechanism works precisely because the child is not a memcg. There's a second reason, and it's the stronger one. Reading the child's `cgroup.procs` returns `Operation not supported`, which the kernel documents for threaded cgroups and `cgroup.type` confirms it. Probing a separate run for those values: the child reads `threaded`, the step scope reads `domain threaded`, and a step scope with no dind child beneath it yet reads a plain `domain`. Threaded cgroups carry only the threaded controllers, `cpu`, `cpuset`, `perf_event`, and `pids`, and `memory` isn't among them. The two facts are the same one seen from either end: delegating only threaded controllers is what let the child become threaded at all, and once threaded it can never carry a `memory.current`. Its processes show up in `cgroup.threads` instead. That's the whole finding, and everything an operator would reach for follows from it. There is no per-child memory object, so nothing reading cgroup memory files can report one. cadvisor's line for the child reads zero on working set, usage, and RSS alike: ```shell container_memory_working_set_bytes{container="",id=".../cri-containerd-84c3e193...abe1.scope/4c4095d78188...247",...} 0 container_memory_usage_bytes{container="",id=".../4c4095d78188...247",...} 0 container_memory_rss{container="",id=".../4c4095d78188...247",...} 0 ``` `docker stats` inside the sidecar, looking at the same container through the daemon that created it, agrees: ```shell CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 4c4095d78188 memhog 0.00% 0B / 0B 0.00% 1.32kB / 126B 0B / 0B 1 ``` Neither is wrong. There's no `memory.current` at the child to read, so both correctly report the absence as zero. The step's own line is the one that misleads, because it does move, just nowhere near enough. cadvisor's `container_memory_working_set_bytes` for `step-step` went from about 7.2 MB before the child to about 13.9 MB with it running: roughly 6.4 MiB of movement against a kernel charge of roughly 204 MiB. Those are sampled gauges, so the individual readings drift a few hundred kB between runs, but the relationship is stable: the observed working-set increase was about one-thirtieth of the step cgroup's `memory.current`. One footnote on where the 200 MB went. The step scope's `memory.stat` shows `anon 6541312`, `file 200101888`, and `inactive_file 200101888`: nearly all of the charge was inactive file cache. A memory-limited cgroup can reclaim those pages before it kills a task, although they still count toward the limit. ## What happens when it OOMs Forcing a kill needs anonymous memory. With the step capped at 64Mi via `stepSpecs.computeResources` and a child allocating 500 MiB of anon, the kernel does exactly what the budget says. The kill is a real memcg kill, not a node-pressure eviction, and the cgroup it names is the step's: ```shell oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/b132849958c0.../kubelet.slice/.../kubelet-kubepods-burstable-podee4ed01f_c5b6_401d_84e6_cd45e7a94890.slice/cri-containerd-d5a62de9d463d23595850010701c41f8cd07c14626727281560162300d0f7d98.scope,task_memcg=...,task=entrypoint,pid=79431,uid=0 memory: usage 65536kB, limit 65536kB, failcnt 59 ``` The `oom_memcg` path carries this run's pod UID (`ee4ed01f_c5b6_401d_84e6_cd45e7a94890`), and this run's step container hash, both matched against what `kubectl` reported for the same pod. `usage 65536kB, limit 65536kB` is the 64Mi cap hit exactly, with 59 prior failed charge attempts. The kernel then killed five processes, not one, because `memory.oom.group` is set on the step scope: | process | pid | total-vm | anon-rss | oom_score_adj | |---|---:|---:|---:|---:| | entrypoint | 79431 | 1301884 kB | 5760 kB | 996 | | script-0-fp877 | 80150 | 1720 kB | 0 kB | 996 | | sleep | 80152 | 1704 kB | 0 kB | 996 | | entrypoint | 79452 | 1301884 kB | 5760 kB | 996 | | python3 | 80829 | 523528 kB | 58084 kB | 0 | The `python3` at `oom_score_adj: 0` is the child inside dind, the process that actually invoked the killer. Everything at `996` is Tekton's own step machinery: two `entrypoint` processes, the generated step script, and its `sleep`. The kernel log states the rule plainly: tasks in the step scope "are going to be killed due to `memory.oom.group` set." The step is billed for the child's overrun and executed for it. Note that the killer only got 58084 kB of anon resident before hitting the wall, against a `total-vm` reservation of 523528 kB. The 500 MiB is what it asked for, not what it held. Tekton usually reports the failure, but not as an OOM: ```shell $ kubectl get taskrun cgroup-budget-run-oom -o jsonpath='{.status.conditions}' [{"lastTransitionTime":"2026-08-18T08:32:12Z","message":"\"step-step\" exited with code 137: Error","reason":"StepFailed","status":"False","type":"Succeeded"}] ``` This is the nuance worth getting right, because the obvious reading is wrong. Tekton does have OOM-specific reasons: `TaskRunReasonStepOOM` ("StepOOM"), `TaskRunReasonSidecarOOM`, and `TaskRunReasonInitContainerOOM`, all of which exist in the v1 `TaskRun` types. Tekton's `getFailureInfo()` calls `isOOMKilled()`, which tests `s.State.Terminated.Reason == "OOMKilled"` and nothing else. It never looks at the exit code. The step's terminated state here was `exitCode: 137` with `reason: "Error"`, so the check correctly declined to fire. The interesting part is one layer down, because Tekton isn't the component that makes the decision. containerd writes that string, and it reaches the opposite conclusion from the opposite evidence: on a 137 exit it checks whether the cgroup's `memory.events` counter shows an `oom_kill`, and only then sets the reason. So containerd gates on the exit code Tekton ignores, and Tekton gates on the string containerd may or may not get around to writing. It usually doesn't. Polling the step scope's own `memory.events` in a busy loop through the kill catches the counter arriving and the cgroup disappearing almost simultaneously: ```shell 09:17:12.938 | oom_kill 5 | oom_group_kill 1 | current=774144 09:17:12.940 | directory gone ``` Two milliseconds. At a 20 ms sampling interval the directory was already gone. The kernel's accounting is correct and hierarchical throughout: the pod slice's `memory.events` reads `oom_kill 5` while its `memory.events.local` stays at `0`, which is exactly what a kill charged to a descendant should look like. The counter is there to be read. Under the systemd cgroup driver, the scope unit is garbage-collected as soon as its last process exits, and containerd races that GC to read a file that is about to be removed. containerd's own source comments name this race. Repeat the same OOM six times; the reason comes back as `Error` five times and `OOMKilled` once, with `exitCode: 137` every time. So `StepOOM` isn't dead code, and this isn't a Tekton bug: it's a race one layer below Tekton that Tekton faithfully reports the losing side of. An operator sees the same generic failure most of the time and the correct one occasionally, which is worse than either being consistent. ## Why not to run this The enforcement is real, and the operability isn't, and those are separable properties. Nothing downstream of the kernel sees the charge at the granularity it happened. Dashboards built on `container_memory_working_set_bytes` stay roughly flat while the kernel accumulates 200 megabytes per step. This cluster had no metrics-server installed, so it did not test whether `kubectl top` reproduces the gap. Any downstream system that consumes this cAdvisor per-container working-set series without compensating for the nested charge could inherit it; no autoscaler or sizing recommender was tested here. Recovering the truth means reading `memory.current` on the node, which means node access and a scrape path that doesn't exist by default. Tekton's status surface can't distinguish a step that overran its own budget from a step that was killed for a child's. Both usually arrive as `StepFailed` with `exited with code 137: Error`, and `memory.oom.group` means the same set of Tekton processes dies either way, so nothing in the process-exit shape distinguishes them. The occasional run that does win the race and report `StepOOM` is no better for this purpose: it correctly says the step was OOM-killed, which is still the wrong container to go looking at. An on-call engineer reading only the `TaskRun` has no signal pointing at the build container, and an intermittent reason string is harder to build an alert on than a consistently wrong one. The pod also still reserves two numbers, not one. dockerd's own PID stays in the sidecar's cgroup under the sidecar's own limit, so the 8 GiB example in the opening never collapses all the way to 4. What changes is their size: the sidecar's cAdvisor line read 26.9 MB idle and 42.8 MB while a 200 MB child ran, because the child's memory is charged elsewhere. Those measurements cover only this idle and single-child test; a real build's sidecar limit requires measurements of its image pulls, layer extraction, and BuildKit activity. You can't drop `privileged: true` later as a hardening pass. Without it the sidecar reads `0::/` instead of a real cgroup path, the substitution has nothing to match, and the daemon would start with `--cgroup-parent=/`. dockerd never gets that far, dying on a mount failure first, but nothing in the script detects or reports the broken path either way. Both failure modes leave a dead sidecar inside a green `TaskRun`. RBAC denial and the unprivileged case both end with the sidecar at exit code 1 and the `TaskRun` at `Succeeded` / `All Steps have completed executing`. The step ran, passed, and was never subject to the budget the pipeline thought it had. There is no annotation, condition, or event distinguishing an enforced run from an unenforced one. There's a related Docker issue, and it's worth being precise about how it differs. [moby/moby#45378](https://github.com/moby/moby/issues/45378), `DinD cgroupv2 problem inside K8s`, filed `2023-04-21` and still open, reports containers inside DinD exceeding the pod memory limit without being OOM-killed, an enforcement failure not seen on cgroup v1. That's the opposite half of what shows up here, where enforcement works, and attribution is missing. Both point to nested cgroup accounting under DinD being fragile, in different directions. Everything above ran on a single-node kind cluster, `kind v0.32.0`, node image `kindest/node:v1.36.1`, Kubernetes v1.36.1 on Debian 13 with containerd 2.3.1, kernel `6.8.0-117-generic` on aarch64, cgroup v2 unified hierarchy, and Tekton Pipelines v1.15.x. The sidecar is `docker:28-dind` (Docker 28.5.2, Alpine 3.22) with `apk add bash kubectl jq` on top, since that base image ships no bash and the entrypoint needs it. The step is `alpine:3.20`. The load was generated by `kubectl exec` into the sidecar and by running `docker run` against its own dockerd on a disposable local cluster, with nothing resembling production. ## What to use instead `PodLevelResources` is the answer when the feature is enabled on a Kubernetes 1.34-or-later cluster. It puts the limit on the pod and keeps the kubelet's accounting and the kernel's accounting pointed at the same object. `PodLevelResourcesFixKubeletQOSClass` and `PodLevelResourcesFixUpdateDefaulting` are beta and enabled by default in 1.36. Read the release notes and feature-gate reference before making a platform commitment. What it leaves unsolved is delivery: `spec.resources` still has to reach the pod, and the `TaskRun`'s `podTemplate` allowlist won't carry it. A mutating admission webhook could patch `spec.resources` onto the resulting pod. A node-level agent could set `memory.max` on the pod cgroup where the path and controller state are verified. It needs node access and depends on the kubelet's slice naming, which varies with the cgroup driver and `cgroupRoot`. This prototype did not test that alternative. It would charge the whole pod, sidecar included, which for a dind pod is often the desired boundary. None of this is really about Tekton or Docker. `cgroup.subtree_control` decides which level distributes a resource. In this kind/containerd configuration, the kubelet did not delegate `memory` below the container scope. That made enforcement and observability separable: the kernel held the step to a budget that ordinary per-container memory metrics could not fully explain.