One cgroup budget for a Tekton taskrun's step and its dind sidecar

A Tekton TaskRun that builds container images runs as one pod holding one or more step containers, alongside a privileged Docker-in-Docker sidecar. A step issues docker build and docker run against the sidecar’s daemon socket. The dockerd process inside the sidecar is what actually does the work: pulling layers, running RUN instructions, spawning child containers. During a build, the sidecar is the busiest thing in the pod.

If the step and the sidecar each need a 4 GiB peak at different times, separate limits reserve 8 GiB against node capacity even though the combined workload peak is 4 GiB. A shared ceiling would fit that usage.

The step and the sidecar sit in sibling cgroups under the same pod slice. The entrypoint swaps the sidecar hash for the step hash, then passes the resulting path to dockerd as --cgroup-parent. dockerd remains in the sidecar cgroup; containers it creates start beneath the step cgroup. Their memory is charged to the step’s limit, and the kernel holds that limit through an OOM kill. The same nesting that makes it work is what erases the build container from the metrics every memory dashboard is built on.

The problem

The unit of budget you want here is the pod: give this whole TaskRun 4 GiB, let the step and the build children draw from it, and stop paying for a peak that never happens. Kubernetes has that concept. PodLevelResources, KEP-2837, sets spec.resources on the pod itself. It went beta and default-on in 1.34. In 1.36, InPlacePodLevelResourcesVerticalScaling graduates to beta and is enabled by default. Two pod-level-resource bug fixes, PodLevelResourcesFixUpdateDefaulting and PodLevelResourcesFixKubeletQOSClass, are beta and enabled by default in 1.36.

But Tekton can’t use it. A TaskRun’s podTemplate accepts a fixed subset of Pod-spec fields, and that subset excludes resources. Tekton’s computeResources field configures individual containers; it cannot set spec.resources on the resulting Pod.

So a platform team that wants one number per TaskRun pod is stuck between a Kubernetes feature that’s still beta and a Tekton API surface that doesn’t expose it.

The options

Four candidates get the step and the build sharing one ceiling.

PodLevelResources. Set the budget on the pod and let the kernel divide it. Correct, but unavailable in Tekton’s podTemplate and still beta.

PID migration via cgroup.procs. Write the sidecar’s own dockerd PID into the step container’s cgroup.procs, moving the daemon itself under the step’s budget. This charges dockerd’s own footprint, including image pulls and layer extraction, to the step, which is a different and much blunter thing than charging the build’s children.

--cgroup-parent borrowing. Start dockerd with --cgroup-parent set to the step container’s scope, so containers that do not override that setting are nested inside the step’s cgroup, while dockerd itself stays where the kubelet put it.

Writing memory.max into the pod slice. Have something with node access compute a budget and write it directly into kubelet-kubepods-*-pod<uid>.slice/memory.max, bypassing the API entirely.

I tested --cgroup-parent borrowing. Of the four designs considered, it is the one this prototype validates inside a normal pod while targeting build children rather than the daemon that spawns them.

The chosen method

The sidecar’s entrypoint runs before dockerd and does four things: read its own cgroup path, identify its own pod, ask the API server for the step container’s containerd ID, and substitute that ID into the path it read.

The critical property is that dockerd’s own PID never moves. The daemon stays in the sidecar’s cgroup, where the kubelet put it and where the kubelet’s own accounting expects it. Only the containers it goes on to create land under the step.

Step one reads the sidecar’s own cgroup from procfs, which under the cgroup v2 unified hierarchy, is a single 0:: line:

1
$ awk -F: '/^0::/ {print $3}' /proc/self/cgroup
2
/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podcd57a787_d809_4a55_b6f8_ab38de73aeb8.slice/cri-containerd-4354439d2f1668c37baaabaa5a33b5ce71a1b24c182e4ebe0d16455889c14617.scope

Steps two and three read the pod name from /etc/hostname and the namespace from /var/run/secrets/kubernetes.io/serviceaccount/namespace, then poll the API server for the step container’s ID, filtering out the sidecar’s own entry:

1
kubectl get pod "$podname" -n "$namespace" -o json | \
2
  jq -r 'first(.status.containerStatuses[] | select(.name != "sidecar-dind" and .started == true) | .containerID)' | \
3
  awk -F'://' '{print $2}'

Step four is the whole trick. The sidecar’s cgroup path and the step’s cgroup path are siblings under the same pod slice, differing only in the container hash, so one substitution converts one into the other:

1
NEW_CG_PATH=$(echo "$CGROUP_PATH" | sed -E "s/cri-containerd-[a-f0-9]+\.scope/cri-containerd-$CONTAINERD_HASH.scope/")

Then dockerd starts with that path as its parent:

1
exec /usr/local/bin/dockerd-entrypoint.sh \
2
  --cgroup-parent="$NEW_CG_PATH" \
3
  "$@"

The sequence, after the step has started and its container ID is available:

sequenceDiagram
    participant Init as sidecar entrypoint
    participant API as kube-apiserver
    participant D as dockerd (sidecar cgroup)
    participant Step as step container
    participant CG as step .scope cgroup

    Init->>Init: read /proc/self/cgroup
    Init->>API: get pod, read step containerID
    API-->>Init: containerd://<step hash>
    Init->>Init: sed sidecar hash -> step hash
    Init->>D: exec dockerd with --cgroup-parent=<step scope>
    Step->>D: docker run / docker build
    D->>CG: create child cgroup under step scope
    Note over CG: child's memory charges to it and its step ancestor

One detail makes the substitution viable, and it’s worth checking before copying any of this. The two sides of the pod run different cgroup drivers:

1
$ kubectl exec <pod> -c sidecar-dind -- docker info --format '{{.CgroupDriver}} / v{{.CgroupVersion}}'
2
cgroupfs / v2
3
$ kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq -r .kubeletconfig.cgroupDriver
4
systemd

The kubelet builds the systemd-style .slice/.scope paths that the sidecar reads from procfs, while dockerd inside the sidecar mounts cgroupfs and treats --cgroup-parent as a literal directory to create beneath. That mismatch is what lets a path lifted from the kubelet’s hierarchy be handed to dockerd unchanged. A dind daemon configured with the systemd driver would expect a slice name and derive its own scope instead, and the borrowed path wouldn’t nest the same way.

Two prerequisites beyond that. The sidecar’s service account needs to get on pods in its own namespace, because the container ID is only available through the API. And the sidecar needs privileged: true, both for dockerd’s normal reasons and because without a host cgroup namespace the container reads only the namespace-relative 0::/ and has no host path to rewrite.

What the kernel does with it

The nesting is literal. After the step tells dockerd to run a container that writes a 200 MB file, that container’s cgroup appears as a subdirectory of the step’s scope on the node:

1
$ ls .../cri-containerd-84c3e193...abe1.scope/
2
4c4095d78188d1ba957135bd72bf672eec1de8a40ed8e011e3b30fb1a811d247
3
cgroup.controllers
4
cgroup.subtree_control
5
...
6
memory.current
7
memory.max
8
...

The kernel charges it to the step. Reading the step scope’s memory.current directly on the node, with a 200 MB child running:

1
$ cat .../cri-containerd-84c3e193...abe1.scope/memory.current
2
213598208

That’s roughly 203.7 MiB against a step container whose own resident footprint before the child started was about 7 MB. The charge landed exactly where the mechanism aimed it.

That happens because of one file. A cgroup can distribute a resource to child cgroups only when it enables that controller in cgroup.subtree_control. The step’s scope lists memory as available without enabling it:

1
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.controllers
2
cpuset cpu io memory hugetlb pids rdma misc
3
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.subtree_control
4
cpuset cpu pids
5
$ cat .../4c4095d78188...247/cgroup.controllers
6
cpuset cpu pids
7
$ cat .../4c4095d78188...247/memory.current
8
cat: ...: No such file or directory

So the child gets no memory.* files at all. It is not a memory-accounting boundary, so memory instantiated by its processes is accounted at the step’s .scope. The mechanism works precisely because the child is not a memcg.

There’s a second reason, and it’s the stronger one. Reading the child’s cgroup.procs returns Operation not supported, which the kernel documents for threaded cgroups and cgroup.type confirms it. Probing a separate run for those values: the child reads threaded, the step scope reads domain threaded, and a step scope with no dind child beneath it yet reads a plain domain. Threaded cgroups carry only the threaded controllers, cpu, cpuset, perf_event, and pids, and memory isn’t among them. The two facts are the same one seen from either end: delegating only threaded controllers is what let the child become threaded at all, and once threaded it can never carry a memory.current. Its processes show up in cgroup.threads instead.

That’s the whole finding, and everything an operator would reach for follows from it. There is no per-child memory object, so nothing reading cgroup memory files can report one. cadvisor’s line for the child reads zero on working set, usage, and RSS alike:

1
container_memory_working_set_bytes{container="",id=".../cri-containerd-84c3e193...abe1.scope/4c4095d78188...247",...} 0
2
container_memory_usage_bytes{container="",id=".../4c4095d78188...247",...} 0
3
container_memory_rss{container="",id=".../4c4095d78188...247",...} 0

docker stats inside the sidecar, looking at the same container through the daemon that created it, agrees:

1
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
2
4c4095d78188   memhog    0.00%     0B / 0B             0.00%     1.32kB / 126B   0B / 0B     1

Neither is wrong. There’s no memory.current at the child to read, so both correctly report the absence as zero.

The step’s own line is the one that misleads, because it does move, just nowhere near enough. cadvisor’s container_memory_working_set_bytes for step-step went from about 7.2 MB before the child to about 13.9 MB with it running: roughly 6.4 MiB of movement against a kernel charge of roughly 204 MiB. Those are sampled gauges, so the individual readings drift a few hundred kB between runs, but the relationship is stable: the observed working-set increase was about one-thirtieth of the step cgroup’s memory.current.

One footnote on where the 200 MB went. The step scope’s memory.stat shows anon 6541312, file 200101888, and inactive_file 200101888: nearly all of the charge was inactive file cache. A memory-limited cgroup can reclaim those pages before it kills a task, although they still count toward the limit.

What happens when it OOMs

Forcing a kill needs anonymous memory. With the step capped at 64Mi via stepSpecs.computeResources and a child allocating 500 MiB of anon, the kernel does exactly what the budget says.

The kill is a real memcg kill, not a node-pressure eviction, and the cgroup it names is the step’s:

1
oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/b132849958c0.../kubelet.slice/.../kubelet-kubepods-burstable-podee4ed01f_c5b6_401d_84e6_cd45e7a94890.slice/cri-containerd-d5a62de9d463d23595850010701c41f8cd07c14626727281560162300d0f7d98.scope,task_memcg=...,task=entrypoint,pid=79431,uid=0
2
memory: usage 65536kB, limit 65536kB, failcnt 59

The oom_memcg path carries this run’s pod UID (ee4ed01f_c5b6_401d_84e6_cd45e7a94890), and this run’s step container hash, both matched against what kubectl reported for the same pod. usage 65536kB, limit 65536kB is the 64Mi cap hit exactly, with 59 prior failed charge attempts.

The kernel then killed five processes, not one, because memory.oom.group is set on the step scope:

The python3 at oom_score_adj: 0 is the child inside dind, the process that actually invoked the killer. Everything at 996 is Tekton’s own step machinery: two entrypoint processes, the generated step script, and its sleep. The kernel log states the rule plainly: tasks in the step scope “are going to be killed due to memory.oom.group set.” The step is billed for the child’s overrun and executed for it.

Note that the killer only got 58084 kB of anon resident before hitting the wall, against a total-vm reservation of 523528 kB. The 500 MiB is what it asked for, not what it held.

Tekton usually reports the failure, but not as an OOM:

1
$ kubectl get taskrun cgroup-budget-run-oom -o jsonpath='{.status.conditions}'
2
[{"lastTransitionTime":"2026-08-18T08:32:12Z","message":"\"step-step\" exited with code 137: Error","reason":"StepFailed","status":"False","type":"Succeeded"}]

This is the nuance worth getting right, because the obvious reading is wrong. Tekton does have OOM-specific reasons: TaskRunReasonStepOOM (“StepOOM”), TaskRunReasonSidecarOOM, and TaskRunReasonInitContainerOOM, all of which exist in the v1 TaskRun types. Tekton’s getFailureInfo() calls isOOMKilled(), which tests s.State.Terminated.Reason == "OOMKilled" and nothing else. It never looks at the exit code. The step’s terminated state here was exitCode: 137 with reason: "Error", so the check correctly declined to fire.

The interesting part is one layer down, because Tekton isn’t the component that makes the decision. containerd writes that string, and it reaches the opposite conclusion from the opposite evidence: on a 137 exit it checks whether the cgroup’s memory.events counter shows an oom_kill, and only then sets the reason. So containerd gates on the exit code Tekton ignores, and Tekton gates on the string containerd may or may not get around to writing.

It usually doesn’t. Polling the step scope’s own memory.events in a busy loop through the kill catches the counter arriving and the cgroup disappearing almost simultaneously:

1
09:17:12.938 | oom_kill 5 | oom_group_kill 1 | current=774144
2
09:17:12.940 | directory gone

Two milliseconds. At a 20 ms sampling interval the directory was already gone. The kernel’s accounting is correct and hierarchical throughout: the pod slice’s memory.events reads oom_kill 5 while its memory.events.local stays at 0, which is exactly what a kill charged to a descendant should look like. The counter is there to be read. Under the systemd cgroup driver, the scope unit is garbage-collected as soon as its last process exits, and containerd races that GC to read a file that is about to be removed. containerd’s own source comments name this race.

Repeat the same OOM six times; the reason comes back as Error five times and OOMKilled once, with exitCode: 137 every time. So StepOOM isn’t dead code, and this isn’t a Tekton bug: it’s a race one layer below Tekton that Tekton faithfully reports the losing side of. An operator sees the same generic failure most of the time and the correct one occasionally, which is worse than either being consistent.

Why not to run this

The enforcement is real, and the operability isn’t, and those are separable properties.

Nothing downstream of the kernel sees the charge at the granularity it happened. Dashboards built on container_memory_working_set_bytes stay roughly flat while the kernel accumulates 200 megabytes per step. This cluster had no metrics-server installed, so it did not test whether kubectl top reproduces the gap. Any downstream system that consumes this cAdvisor per-container working-set series without compensating for the nested charge could inherit it; no autoscaler or sizing recommender was tested here. Recovering the truth means reading memory.current on the node, which means node access and a scrape path that doesn’t exist by default.

Tekton’s status surface can’t distinguish a step that overran its own budget from a step that was killed for a child’s. Both usually arrive as StepFailed with exited with code 137: Error, and memory.oom.group means the same set of Tekton processes dies either way, so nothing in the process-exit shape distinguishes them. The occasional run that does win the race and report StepOOM is no better for this purpose: it correctly says the step was OOM-killed, which is still the wrong container to go looking at. An on-call engineer reading only the TaskRun has no signal pointing at the build container, and an intermittent reason string is harder to build an alert on than a consistently wrong one.

The pod also still reserves two numbers, not one. dockerd’s own PID stays in the sidecar’s cgroup under the sidecar’s own limit, so the 8 GiB example in the opening never collapses all the way to 4. What changes is their size: the sidecar’s cAdvisor line read 26.9 MB idle and 42.8 MB while a 200 MB child ran, because the child’s memory is charged elsewhere. Those measurements cover only this idle and single-child test; a real build’s sidecar limit requires measurements of its image pulls, layer extraction, and BuildKit activity.

You can’t drop privileged: true later as a hardening pass. Without it the sidecar reads 0::/ instead of a real cgroup path, the substitution has nothing to match, and the daemon would start with --cgroup-parent=/. dockerd never gets that far, dying on a mount failure first, but nothing in the script detects or reports the broken path either way.

Both failure modes leave a dead sidecar inside a green TaskRun. RBAC denial and the unprivileged case both end with the sidecar at exit code 1 and the TaskRun at Succeeded / All Steps have completed executing. The step ran, passed, and was never subject to the budget the pipeline thought it had. There is no annotation, condition, or event distinguishing an enforced run from an unenforced one.

There’s a related Docker issue, and it’s worth being precise about how it differs. moby/moby#45378, DinD cgroupv2 problem inside K8s, filed 2023-04-21 and still open, reports containers inside DinD exceeding the pod memory limit without being OOM-killed, an enforcement failure not seen on cgroup v1. That’s the opposite half of what shows up here, where enforcement works, and attribution is missing. Both point to nested cgroup accounting under DinD being fragile, in different directions.

Everything above ran on a single-node kind cluster, kind v0.32.0, node image kindest/node:v1.36.1, Kubernetes v1.36.1 on Debian 13 with containerd 2.3.1, kernel 6.8.0-117-generic on aarch64, cgroup v2 unified hierarchy, and Tekton Pipelines v1.15.x. The sidecar is docker:28-dind (Docker 28.5.2, Alpine 3.22) with apk add bash kubectl jq on top, since that base image ships no bash and the entrypoint needs it. The step is alpine:3.20. The load was generated by kubectl exec into the sidecar and by running docker run against its own dockerd on a disposable local cluster, with nothing resembling production.

What to use instead

PodLevelResources is the answer when the feature is enabled on a Kubernetes 1.34-or-later cluster. It puts the limit on the pod and keeps the kubelet’s accounting and the kernel’s accounting pointed at the same object. PodLevelResourcesFixKubeletQOSClass and PodLevelResourcesFixUpdateDefaulting are beta and enabled by default in 1.36. Read the release notes and feature-gate reference before making a platform commitment. What it leaves unsolved is delivery: spec.resources still has to reach the pod, and the TaskRun’s podTemplate allowlist won’t carry it. A mutating admission webhook could patch spec.resources onto the resulting pod.

A node-level agent could set memory.max on the pod cgroup where the path and controller state are verified. It needs node access and depends on the kubelet’s slice naming, which varies with the cgroup driver and cgroupRoot. This prototype did not test that alternative. It would charge the whole pod, sidecar included, which for a dind pod is often the desired boundary.

None of this is really about Tekton or Docker. cgroup.subtree_control decides which level distributes a resource. In this kind/containerd configuration, the kubelet did not delegate memory below the container scope. That made enforcement and observability separable: the kernel held the step to a budget that ordinary per-container memory metrics could not fully explain.