A circuit breaker for Kubernetes admission webhooks

A slow webhook can block API requests even when its mutation is optional. With failurePolicy: Fail, Kubernetes rejects a matching request when the webhook times out.

failurePolicy: Ignore changes that result. Kubernetes still calls the webhook and waits for the call to fail, but it lets the request continue afterward. That is the important limitation: Ignore restores admission availability, not webhook latency.

I built a small controller that changes one approved webhook from Fail to Ignore after repeated failed health checks. It changes the policy back after the webhook recovers. The local run completed that cycle. The recovery code is not safe enough to deploy.

When is it safe to fail open?

failurePolicy applies when a webhook call fails, times out, or cannot be completed. It does not override a webhook that replies with allowed: false. A healthy webhook can still reject a request while its failure policy is Ignore. Kubernetes documents that behavior in Dynamic Admission Control.

If it is always safe to admit an object without the mutation, use Ignore from the start. Kubernetes recommends that design, with separate validation for rules that must still be enforced.

This controller is for the narrower case: the webhook is required in normal operation, but an operator has approved admitting objects without its mutation during an outage. That means accepting two risks: some requests fail before the controller opens the circuit, and some objects are admitted without the mutation afterward.

If an operator accepts those risks, the target should say so directly:

1
allowAdmissionWithoutMutation: true

The controller can check that a target is explicitly listed. It cannot decide whether skipping a mutation is safe. That decision belongs to the operator. A mutation can be required for security, routing, storage, or another operational reason.

The delayed-webhook test

I ran the controller and a TLS webhook on a local kind cluster. The webhook matched ConfigMap creates:

1
webhooks:
2
  - name: slow-mutator.webhook-circuit-breaker.test
3
    failurePolicy: Fail
4
    timeoutSeconds: 1
5
    clientConfig:
6
      service:
7
        namespace: webhook-circuit-breaker
8
        name: slow-mutator
9
        path: /mutate
10
    rules:
11
      - apiGroups: [""]
12
        apiVersions: ["v1"]
13
        operations: ["CREATE"]
14
        resources: ["configmaps"]

The test webhook does not change the ConfigMap. For the API-server request used here, it waits for a configured delay and then returns Allowed: true. I used a two-second delay, which is longer than the one-second admission timeout. This isolates what Kubernetes does when a webhook call fails under Fail and Ignore; it does not show what a missing real mutation would do to an application.

In this run, the controller sends an HTTPS GET to /healthz once a second, with a 250 ms deadline. Three failed checks open the circuit. Five successful checks close it. The test applies the same delay to /healthz and /mutate, so the probe sees the injected fault. That does not make /healthz a general test of admission health: the API server can reach and exercise a webhook differently.

Ignore admits requests, but still waits

The four ConfigMap creates produced these client-side times:

The delayed request succeeded under Ignore, but it still took about one second. Kubernetes called the webhook, waited for the configured timeout, then continued because the policy was Ignore. The controller did not remove the webhook from the request path.

These are single wall-clock measurements around kubectl create. They include client and API-server work. They show the sequence, not a latency distribution, throughput limit, or the exact time required to open the circuit.

The registration changed from Fail to Ignore, then back to Fail. The request and registration captures show those three policy states. They do not show the controller’s saved recovery record.

How the controller opens the circuit

The controller needs the old policy so that it can put it back later. It saves that value in a ConfigMap, then patches the webhook registration:

1
base := cur.DeepCopy()
2
original := cur.Webhooks[i].FailurePolicy
3
rec := Record{Original: original, OriginalWasNil: original == nil,
4
  OpenedAt: r.Now(), ResourceVersion: cur.ResourceVersion}
5

6
if err := r.Store.Put(ctx, t.Key(), rec); err != nil {
7
  return err
8
}
9

10
cur.Webhooks[i].FailurePolicy = failPolicy(admissionv1.Ignore)
11
if err := r.Client.Patch(ctx, &cur, client.MergeFrom(base)); err != nil {
12
  return err
13
}

That ordering creates the first serious problem. If the ConfigMap write succeeds but the webhook patch fails, the controller has a record saying the circuit is open even though the webhook is still Fail. On the next check, it finds the record and stops trying to open the circuit. Requests can keep failing.

The record is useful after a restart, but it is not proof that the policy patch succeeded. The controller needs an explicit state for “record saved, policy not yet changed”, or it needs a different design.

How recovery can get stuck

When the webhook appears healthy, the controller reads the current policy. If it is still Ignore, it restores the saved value and then deletes the record:

1
if policy(cur.Webhooks[i].FailurePolicy) != "Ignore" {
2
  return nil // another writer changed the policy
3
}
4

5
base := cur.DeepCopy()
6
if rec.OriginalWasNil {
7
  cur.Webhooks[i].FailurePolicy = nil
8
} else {
9
  cur.Webhooks[i].FailurePolicy = rec.Original
10
}
11
if err := r.Client.Patch(ctx, &cur, client.MergeFrom(base)); err != nil {
12
  return err
13
}
14

15
return r.Store.Delete(ctx, t.Key())

This has the reverse failure. The patch can restore Fail, then the ConfigMap delete can fail. The controller restarts with a stale record. It sees Fail, treats it as an external change, and keeps the record instead of clearing it.

There is a second ownership problem. The controller only checks whether the current policy is Ignore. It cannot tell its own Ignore from an administrator’s Ignore. It also does not save the webhook configuration UID, so deleting and recreating a configuration with the same name creates ambiguity.

The patch itself can also overwrite another writer’s change. client.MergeFrom(base) can replace the complete webhook list. The fresh read helps, but it does not protect an update made after that read. controller-runtime’s optimistic-lock merge patch is one option: it turns a concurrent update into a conflict, which the controller can retry from a fresh read.

Why I would not deploy this

The controller’s ClusterRole can patch any mutating webhook configuration and update ConfigMaps across the cluster. The target list limits the program, not its Kubernetes permissions. A real deployment should restrict the allowed webhook configurations and keep state access inside the controller namespace.

The probe accepts only Service-backed webhooks with a PEM-encoded caBundle set explicitly. Kubernetes permits other valid configurations, including an omitted CA bundle that uses the API server’s system roots. This controller counts those unsupported configurations as probe failures. It could open the circuit because it cannot probe the target, not because the webhook is down.

The probe also ignores clientConfig.service.port and always uses 443. A webhook on another port looks unhealthy even when it is working.

One replica holds the probe counters. Adding replicas is not safe without leader election or shared counters. If the controller opens the circuit and then stops without restarting, nothing restores Fail. Restoring the policy also does not fix objects that were created while the mutation was skipped.

The unit tests use a fake Kubernetes client and a fake prober. They cover the counter logic, but not TLS, API-server defaults, or the two-write failures above.

Changing the policy is easy; recovering safely is hard

For an always-optional mutation, static Ignore is simpler. For a temporary exception, the request and registration captures show Fail, then Ignore, then Fail. They do not show the saved record being written or deleted. The policy change did not remove the one-second webhook wait.

Changing Fail to Ignore is easy. Safely owning that change, recovering from an interrupted update, and knowing when to undo it are the hard parts.

References

1. Dynamic Admission Control
2. Admission Webhook Good Practices
3. MutatingWebhookConfiguration
4. controller-runtime client