# Disk I/O QoS for Kubernetes with cgroup v2 io.weight

Kubernetes can request and limit CPU, memory, and ephemeral-storage capacity. It has no native Pod-level resource semantics for runtime disk-I/O bandwidth, IOPS, or latency QoS.

CPU has `requests` and `limits`. Memory has `requests`, `limits`, and OOM eviction. `ResourceQuota` can account for `requests.ephemeral-storage` and `limits.ephemeral-storage`, but that is disk *capacity*, not disk *performance*. A backup job can stay inside its storage quota and still consume service from the same device as a database. Neither the scheduler nor `ResourceQuota` arbitrates that contention.

The Linux cgroup v2 I/O controller can. This prototype uses `io.weight` to give two Pods an explicit, node-local priority policy: low tier gets `10`; high tier gets `500`. It answers a narrow question: when device service is constrained, which workload should be favored? It does not reserve bandwidth or guarantee a latency target.

## The control I started with was not available

The first mechanism I checked was `io.latency`. It accepts a target completion latency for a cgroup and protects it by throttling peer cgroups whose own target is higher. That is closer to the usual database-versus-backup problem: protect a latency-sensitive workload from bulk I/O. It is still not a hard SLA, but it is a latency-oriented control.

It was absent from this node hierarchy. `find /sys/fs/cgroup -name io.latency` returned no result anywhere on the kind node, so this implementation could not use it. `io.latency` and iocost are separate kernel facilities; enabling iocost does not make `io.latency` appear.

That left `io.weight`. This article validates its control path and active iocost accounting. The conclusion is deliberately narrower than a benchmark claim: `io.weight` expresses relative I/O priority, not a reservation.

## What `io.weight` actually is

`io.weight` is a relative share. Pod cgroups expose it only when their parent enables `io` in `cgroup.subtree_control`. The default is `100`; valid values are `1` through `10000`. The interface accepts a cgroup-wide default value and reports the effective value on readback:

```shell
$ cat .../pod-<uid>.slice/io.weight
default 100
$ printf 'default 500\n' > .../pod-<uid>.slice/io.weight
$ cat .../pod-<uid>.slice/io.weight
default 500
```

A 500:10 policy favors the high-tier cgroup over the low-tier cgroup when both compete through the same active I/O controller. It does not promise 500 IOPS, a bandwidth floor, or a completion-latency bound. The workload, request mix, backing device, and device model still determine the observed result.

The controller has separate mechanisms for separate policies:

- `io.weight` is proportional arbitration.
- `io.max` is a per-device rate limit. Temporary bursts are allowed.
- `io.latency` is the latency-target mechanism.

Those distinctions are operational, not semantic trivia. A 50:1 weight ratio changes relative priority. It does not give a bulk writer a fixed latency bound. Check `io.latency` on the actual node image before designing around it; it is not universally available.

For iocost-backed weighting to operate on a device, the root cgroup must report that device in `io.cost.qos` with `enable=1`. Writing `io.weight` proves configuration. Reading `io.cost.qos` for `enable=1` and cgroup `io.stat` for cost fields verifies active accounting; inspect `io.cost.model` to understand the model it is using.

## The cgroup path is configuration, not an API

The common examples use `/sys/fs/cgroup/kubepods.slice/...`. That was not the path on this node. The kind node used the systemd cgroup driver with kubelet `cgroupRoot: /kubelet`, producing:

```text
/sys/fs/cgroup/kubelet.slice/
  kubelet-kubepods.slice/
    kubelet-kubepods-burstable.slice/
      kubelet-kubepods-burstable-pod<uid_with_underscores>.slice
```

Guaranteed Pods are direct children of `kubelet-kubepods.slice`. Burstable and BestEffort Pods have an extra QoS-class slice. The Pod UID is part of the leaf name, with dashes replaced by underscores.

The exact path is not portable. It depends on the cgroup driver, kubelet configuration, and Pod QoS class. A cgroupfs-driver cluster uses a different directory layout. That is why the daemon accepts `cgroupRoot` as configuration instead of hardcoding a path.

This helper builds the target path for the systemd layout:

```go
func PodSlicePath(cgroupRoot, qosClass, uid string) (string, error) {
	if uid == "" {
		return "", fmt.Errorf("cgroup: empty pod UID")
	}
	base, err := kubepodsBase(cgroupRoot)
	if err != nil {
		return "", err
	}

	normalizedUID := strings.ReplaceAll(uid, "-", "_")
	lowerQOS := strings.ToLower(qosClass)

	switch lowerQOS {
	case "guaranteed":
		return fmt.Sprintf("%s/kubelet-kubepods-pod%s.slice", base, normalizedUID), nil
	case "burstable", "besteffort":
		qosSlice := fmt.Sprintf("kubelet-kubepods-%s.slice", lowerQOS)
		leaf := fmt.Sprintf("kubelet-kubepods-%s-pod%s.slice", lowerQOS, normalizedUID)
		return base + "/" + qosSlice + "/" + leaf, nil
	default:
		return "", fmt.Errorf("cgroup: unknown QOS class %q", qosClass)
	}
}
```

## A per-device `io.weight` write was rejected

The first write form I tried was the per-device form documented by the interface:

```shell
$ printf '253:16 200\n' > .../low-priority.../io.weight
sh: 7: printf: I/O error
$ cat .../low-priority.../io.weight
default 10
```

The write failed and the file retained its prior value. The cgroup-wide form succeeded on the same cgroup:

```shell
$ printf 'default 150\n' > .../pod-<uid>.slice/io.weight
$ cat .../pod-<uid>.slice/io.weight
default 150
```

The daemon therefore writes only the cgroup-wide form. The node used `mq-deadline`, not BFQ, but that alone does not explain why the kernel returned EIO. The useful fact is simpler: on this node and device, per-device syntax was rejected while the cgroup-wide syntax applied and read back correctly.

## The daemon

The prototype polls two named Pods. It reads an `io-qos.demo/tier` annotation (`high` or `low`), maps it to a weight, resolves the cgroup path, writes the value, and reads it back.

```go
func (d *Daemon) reconcileOne(ctx context.Context, t Target) error {
	pod, err := d.Client.CoreV1().Pods(t.Namespace).Get(ctx, t.Name, metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("get pod: %w", err)
	}

	if pod.Status.Phase != corev1.PodRunning {
		log.Printf("qosd: %s/%s: phase=%s, waiting", t.Namespace, t.Name, pod.Status.Phase)
		return nil
	}

	tier, ok := pod.Annotations[TierAnnotation]
	if !ok {
		return fmt.Errorf("missing annotation %s", TierAnnotation)
	}
	weight, ok := TierWeight[tier]
	if !ok {
		return fmt.Errorf("unknown tier %q", tier)
	}
	if applied, ok := d.applied[string(pod.UID)]; ok && applied == weight {
		return nil
	}

	slicePath, err := cgroup.PodSlicePath(d.CgroupRoot, string(pod.Status.QOSClass), string(pod.UID))
	if err != nil {
		return fmt.Errorf("resolve cgroup path: %w", err)
	}
	written, readback, err := cgroup.WriteWeight(slicePath, weight)
	if err != nil {
		return fmt.Errorf("write weight: %w", err)
	}
	log.Printf("qosd: %s/%s uid=%s tier=%s qos=%s path=%s wrote=%q readback=%q",
		t.Namespace, t.Name, pod.UID, tier, pod.Status.QOSClass, slicePath+"/io.weight",
		trimNL(written), trimNL(readback))
	d.applied[string(pod.UID)] = weight
	return nil
}
```

The write helper is intentionally boring. The readback matters because a successful-looking control loop is useless if the kernel rejects the requested form.

```go
func WriteWeight(slicePath string, weight int) (written string, readback string, err error) {
	line, err := WeightLine(weight)
	if err != nil {
		return "", "", err
	}

	weightFile := slicePath + "/io.weight"
	if err := os.WriteFile(weightFile, []byte(line), 0644); err != nil {
		return line, "", fmt.Errorf("cgroup: write %s: %w", weightFile, err)
	}

	data, err := os.ReadFile(weightFile)
	if err != nil {
		return line, "", fmt.Errorf("cgroup: read back %s: %w", weightFile, err)
	}
	return line, string(data), nil
}
```

The live daemon runs inside the kind control-plane node. That placement gives it access to the node cgroup hierarchy and the Kubernetes API credentials it needs to resolve Pods. It is not a DaemonSet or a general cluster controller; this is a focused one-node prototype.

## Validating the policy path

The test used two Burstable Pods with direct-I/O fio jobs against a shared `hostPath` on the kind node. Low tier wrote sequential 1 MiB requests with four jobs. High tier issued 4 KiB random writes with one job. The request sizes differ by 256:1, so their IOPS and bandwidth are not directly comparable. They are different workload shapes sharing the same target.

The test sequence was equal weights, weighted policy, equal weights again, then weighted policy again. Before and after each arm it captured the two `io.weight` files and `io.stat`. The test VM root cgroup reported iocost enabled for the device.

Here, charged-cost share means the high-tier cgroup's `cost.usage` delta divided by the combined high-tier and low-tier deltas. I use it because this test validates controller behavior, not storage performance. It is controller accounting, not bandwidth, IOPS, or latency.

| Arm | High / low `io.weight` | High-tier charged-cost share |
|---|---:|---:|
| Equal weights, pair 1 | 100 / 100 | 48.90% |
| Weighted, pair 1 | 500 / 10 | 60.10% |
| Equal weights, pair 2 | 100 / 100 | 5.87% |
| Weighted, pair 2 | 500 / 10 | 17.52% |

Across the two ordered pairs, the high-tier share moved upward after the 500:10 setting. That is consistent with the configured priority policy. It is not a throughput or latency result.

The test found the live Pod cgroups, applied each weight, read it back, and captured iocost accounting before and after the workload. The daemon log separately records the same resolution and write/readback path. That distinction matters: the workload capture shows the cgroup control state; the daemon log shows the daemon's own reconciliation path.

The diagram summarizes configured policy, not measured throughput:

```mermaid
sequenceDiagram
    participant Low as low-tier Pod<br/>(1 MiB sequential write)
    participant Dev as shared block device
    participant High as high-tier Pod<br/>(4 KiB random write)

    Note over Low,High: equal weights: 100 / 100
    Low->>Dev: bulk I/O
    High->>Dev: competing I/O

    Note over Low,High: weighted policy: 10 / 500
    Low->>Dev: lower configured share
    High->>Dev: proportionally favored when constrained
```

The point of the test is the control boundary. `io.weight` changes relative priority. It does not establish a throughput number that transfers to another device, filesystem, workload mix, or iocost model. A capacity decision needs repeated randomized arms, synchronized starts, a calibrated model, and the storage stack where the workload will run.

## `io.max` is a second lever

`io.max` handles a different policy. It takes a backing device major:minor and BPS or IOPS limits:

```shell
$ printf '253:16 wiops=200\n' > .../low-priority.../io.max
$ cat .../low-priority.../io.max
253:16 rbps=max wbps=max riops=max wiops=200
```

`253:16` is the device identifier from this node. Derive it from the workload storage path before applying a cap on another node.

This is useful for a known offender, such as a bulk job whose write rate must be contained. It can be combined with `io.weight`: use a rate limit to contain the bulk job, then use weights to arbitrate remaining contention. Neither one creates a latency guarantee.

## Scope and production requirements

This prototype runs on one `kind` node, one virtual block device, and two Pods. It does not coordinate Pods across nodes. It does not establish what a network filesystem, NFS share, or iSCSI volume will do. Tier assignment is annotation-driven and targets two named Pods, not a cluster-wide policy API.

Turning this into a platform feature requires more than moving the code into a DaemonSet:

- Verify cgroup v2 and `io` delegation on every node type.
- Resolve the actual Pod cgroup layout instead of assuming this systemd path.
- Map the workload storage path to its backing device.
- Check that iocost is active for that device where proportional enforcement is required.
- Select the control by policy: arbitration (`io.weight`), rate limit (`io.max`), or latency target (`io.latency`).
- Validate the chosen policy against the real storage stack and workload mix.

The useful conclusion is narrow. Kubernetes does not provide disk-I/O QoS as a Pod resource, but a node agent can use cgroup v2 `io.weight` to impose explicit proportional priority on live Pod cgroups. Use it for arbitration. Use `io.max` when a workload needs a rate limit. Use `io.latency` only on nodes where that latency control exists.