Disk I/O QoS for Kubernetes with cgroup v2 io.weight

Kubernetes can request and limit CPU, memory, and ephemeral-storage capacity. It has no native Pod-level resource semantics for runtime disk-I/O bandwidth, IOPS, or latency QoS.

CPU has requests and limits. Memory has requests, limits, and OOM eviction. ResourceQuota can account for requests.ephemeral-storage and limits.ephemeral-storage, but that is disk capacity, not disk performance. A backup job can stay inside its storage quota and still consume service from the same device as a database. Neither the scheduler nor ResourceQuota arbitrates that contention.

The Linux cgroup v2 I/O controller can. This prototype uses io.weight to give two Pods an explicit, node-local priority policy: low tier gets 10; high tier gets 500. It answers a narrow question: when device service is constrained, which workload should be favored? It does not reserve bandwidth or guarantee a latency target.

The control I started with was not available

The first mechanism I checked was io.latency. It accepts a target completion latency for a cgroup and protects it by throttling peer cgroups whose own target is higher. That is closer to the usual database-versus-backup problem: protect a latency-sensitive workload from bulk I/O. It is still not a hard SLA, but it is a latency-oriented control.

It was absent from this node hierarchy. find /sys/fs/cgroup -name io.latency returned no result anywhere on the kind node, so this implementation could not use it. io.latency and iocost are separate kernel facilities; enabling iocost does not make io.latency appear.

That left io.weight. This article validates its control path and active iocost accounting. The conclusion is deliberately narrower than a benchmark claim: io.weight expresses relative I/O priority, not a reservation.

What io.weight actually is

io.weight is a relative share. Pod cgroups expose it only when their parent enables io in cgroup.subtree_control. The default is 100; valid values are 1 through 10000. The interface accepts a cgroup-wide default value and reports the effective value on readback:

1
$ cat .../pod-<uid>.slice/io.weight
2
default 100
3
$ printf 'default 500\n' > .../pod-<uid>.slice/io.weight
4
$ cat .../pod-<uid>.slice/io.weight
5
default 500

A 500:10 policy favors the high-tier cgroup over the low-tier cgroup when both compete through the same active I/O controller. It does not promise 500 IOPS, a bandwidth floor, or a completion-latency bound. The workload, request mix, backing device, and device model still determine the observed result.

The controller has separate mechanisms for separate policies:

• io.weight is proportional arbitration.
• io.max is a per-device rate limit. Temporary bursts are allowed.
• io.latency is the latency-target mechanism.

Those distinctions are operational, not semantic trivia. A 50:1 weight ratio changes relative priority. It does not give a bulk writer a fixed latency bound. Check io.latency on the actual node image before designing around it; it is not universally available.

For iocost-backed weighting to operate on a device, the root cgroup must report that device in io.cost.qos with enable=1. Writing io.weight proves configuration. Reading io.cost.qos for enable=1 and cgroup io.stat for cost fields verifies active accounting; inspect io.cost.model to understand the model it is using.

The cgroup path is configuration, not an API

The common examples use /sys/fs/cgroup/kubepods.slice/.... That was not the path on this node. The kind node used the systemd cgroup driver with kubelet cgroupRoot: /kubelet, producing:

1
/sys/fs/cgroup/kubelet.slice/
2
  kubelet-kubepods.slice/
3
    kubelet-kubepods-burstable.slice/
4
      kubelet-kubepods-burstable-pod<uid_with_underscores>.slice

Guaranteed Pods are direct children of kubelet-kubepods.slice. Burstable and BestEffort Pods have an extra QoS-class slice. The Pod UID is part of the leaf name, with dashes replaced by underscores.

The exact path is not portable. It depends on the cgroup driver, kubelet configuration, and Pod QoS class. A cgroupfs-driver cluster uses a different directory layout. That is why the daemon accepts cgroupRoot as configuration instead of hardcoding a path.

This helper builds the target path for the systemd layout:

1
func PodSlicePath(cgroupRoot, qosClass, uid string) (string, error) {
2
  if uid == "" {
3
    return "", fmt.Errorf("cgroup: empty pod UID")
4
  }
5
  base, err := kubepodsBase(cgroupRoot)
6
  if err != nil {
7
    return "", err
8
  }
9

10
  normalizedUID := strings.ReplaceAll(uid, "-", "_")
11
  lowerQOS := strings.ToLower(qosClass)
12

13
  switch lowerQOS {
14
  case "guaranteed":
15
    return fmt.Sprintf("%s/kubelet-kubepods-pod%s.slice", base, normalizedUID), nil
16
  case "burstable", "besteffort":
17
    qosSlice := fmt.Sprintf("kubelet-kubepods-%s.slice", lowerQOS)
18
    leaf := fmt.Sprintf("kubelet-kubepods-%s-pod%s.slice", lowerQOS, normalizedUID)
19
    return base + "/" + qosSlice + "/" + leaf, nil
20
  default:
21
    return "", fmt.Errorf("cgroup: unknown QOS class %q", qosClass)
22
  }
23
}

A per-device io.weight write was rejected

The first write form I tried was the per-device form documented by the interface:

1
$ printf '253:16 200\n' > .../low-priority.../io.weight
2
sh: 7: printf: I/O error
3
$ cat .../low-priority.../io.weight
4
default 10

The write failed and the file retained its prior value. The cgroup-wide form succeeded on the same cgroup:

1
$ printf 'default 150\n' > .../pod-<uid>.slice/io.weight
2
$ cat .../pod-<uid>.slice/io.weight
3
default 150

The daemon therefore writes only the cgroup-wide form. The node used mq-deadline, not BFQ, but that alone does not explain why the kernel returned EIO. The useful fact is simpler: on this node and device, per-device syntax was rejected while the cgroup-wide syntax applied and read back correctly.

The daemon

The prototype polls two named Pods. It reads an io-qos.demo/tier annotation (high or low), maps it to a weight, resolves the cgroup path, writes the value, and reads it back.

1
func (d *Daemon) reconcileOne(ctx context.Context, t Target) error {
2
  pod, err := d.Client.CoreV1().Pods(t.Namespace).Get(ctx, t.Name, metav1.GetOptions{})
3
  if err != nil {
4
    return fmt.Errorf("get pod: %w", err)
5
  }
6

7
  if pod.Status.Phase != corev1.PodRunning {
8
    log.Printf("qosd: %s/%s: phase=%s, waiting", t.Namespace, t.Name, pod.Status.Phase)
9
    return nil
10
  }
11

12
  tier, ok := pod.Annotations[TierAnnotation]
13
  if !ok {
14
    return fmt.Errorf("missing annotation %s", TierAnnotation)
15
  }
16
  weight, ok := TierWeight[tier]
17
  if !ok {
18
    return fmt.Errorf("unknown tier %q", tier)
19
  }
20
  if applied, ok := d.applied[string(pod.UID)]; ok && applied == weight {
21
    return nil
22
  }
23

24
  slicePath, err := cgroup.PodSlicePath(d.CgroupRoot, string(pod.Status.QOSClass), string(pod.UID))
25
  if err != nil {
26
    return fmt.Errorf("resolve cgroup path: %w", err)
27
  }
28
  written, readback, err := cgroup.WriteWeight(slicePath, weight)
29
  if err != nil {
30
    return fmt.Errorf("write weight: %w", err)
31
  }
32
  log.Printf("qosd: %s/%s uid=%s tier=%s qos=%s path=%s wrote=%q readback=%q",
33
    t.Namespace, t.Name, pod.UID, tier, pod.Status.QOSClass, slicePath+"/io.weight",
34
    trimNL(written), trimNL(readback))
35
  d.applied[string(pod.UID)] = weight
36
  return nil
37
}

The write helper is intentionally boring. The readback matters because a successful-looking control loop is useless if the kernel rejects the requested form.

1
func WriteWeight(slicePath string, weight int) (written string, readback string, err error) {
2
  line, err := WeightLine(weight)
3
  if err != nil {
4
    return "", "", err
5
  }
6

7
  weightFile := slicePath + "/io.weight"
8
  if err := os.WriteFile(weightFile, []byte(line), 0644); err != nil {
9
    return line, "", fmt.Errorf("cgroup: write %s: %w", weightFile, err)
10
  }
11

12
  data, err := os.ReadFile(weightFile)
13
  if err != nil {
14
    return line, "", fmt.Errorf("cgroup: read back %s: %w", weightFile, err)
15
  }
16
  return line, string(data), nil
17
}

The live daemon runs inside the kind control-plane node. That placement gives it access to the node cgroup hierarchy and the Kubernetes API credentials it needs to resolve Pods. It is not a DaemonSet or a general cluster controller; this is a focused one-node prototype.

Validating the policy path

The test used two Burstable Pods with direct-I/O fio jobs against a shared hostPath on the kind node. Low tier wrote sequential 1 MiB requests with four jobs. High tier issued 4 KiB random writes with one job. The request sizes differ by 256:1 , so their IOPS and bandwidth are not directly comparable. They are different workload shapes sharing the same target.

The test sequence was equal weights, weighted policy, equal weights again, then weighted policy again. Before and after each arm it captured the two io.weight files and io.stat. The test VM root cgroup reported iocost enabled for the device.

Here, charged-cost share means the high-tier cgroup’s cost.usage delta divided by the combined high-tier and low-tier deltas. I use it because this test validates controller behavior, not storage performance. It is controller accounting, not bandwidth, IOPS, or latency.

Across the two ordered pairs, the high-tier share moved upward after the 500:10 setting. That is consistent with the configured priority policy. It is not a throughput or latency result.

The test found the live Pod cgroups, applied each weight, read it back, and captured iocost accounting before and after the workload. The daemon log separately records the same resolution and write/readback path. That distinction matters: the workload capture shows the cgroup control state; the daemon log shows the daemon’s own reconciliation path.

The diagram summarizes configured policy, not measured throughput:

sequenceDiagram
    participant Low as low-tier Pod<br/>(1 MiB sequential write)
    participant Dev as shared block device
    participant High as high-tier Pod<br/>(4 KiB random write)

    Note over Low,High: equal weights: 100 / 100
    Low->>Dev: bulk I/O
    High->>Dev: competing I/O

    Note over Low,High: weighted policy: 10 / 500
    Low->>Dev: lower configured share
    High->>Dev: proportionally favored when constrained

The point of the test is the control boundary. io.weight changes relative priority. It does not establish a throughput number that transfers to another device, filesystem, workload mix, or iocost model. A capacity decision needs repeated randomized arms, synchronized starts, a calibrated model, and the storage stack where the workload will run.

io.max is a second lever

io.max handles a different policy. It takes a backing device major:minor and BPS or IOPS limits:

1
$ printf '253:16 wiops=200\n' > .../low-priority.../io.max
2
$ cat .../low-priority.../io.max
3
253:16 rbps=max wbps=max riops=max wiops=200

253:16 is the device identifier from this node. Derive it from the workload storage path before applying a cap on another node.

This is useful for a known offender, such as a bulk job whose write rate must be contained. It can be combined with io.weight: use a rate limit to contain the bulk job, then use weights to arbitrate remaining contention. Neither one creates a latency guarantee.

Scope and production requirements

This prototype runs on one kind node, one virtual block device, and two Pods. It does not coordinate Pods across nodes. It does not establish what a network filesystem, NFS share, or iSCSI volume will do. Tier assignment is annotation-driven and targets two named Pods, not a cluster-wide policy API.

Turning this into a platform feature requires more than moving the code into a DaemonSet:

• Verify cgroup v2 and io delegation on every node type.
• Resolve the actual Pod cgroup layout instead of assuming this systemd path.
• Map the workload storage path to its backing device.
• Check that iocost is active for that device where proportional enforcement is required.
• Select the control by policy: arbitration (io.weight), rate limit (io.max), or latency target (io.latency).
• Validate the chosen policy against the real storage stack and workload mix.

The useful conclusion is narrow. Kubernetes does not provide disk-I/O QoS as a Pod resource, but a node agent can use cgroup v2 io.weight to impose explicit proportional priority on live Pod cgroups. Use it for arbitration. Use io.max when a workload needs a rate limit. Use io.latency only on nodes where that latency control exists.

References

1. cgroup v2 io controller docs
2. Kubernetes resource management