# Pod Startup Forensics: The Problem

Suppose a pod takes 40 seconds to start instead of 4. It doesn't look like an incident. Nothing pages. `kubectl get pods` eventually shows `Running`. The deploy finishes. The only trace is a vague sense that something felt slow, and by the time anyone goes looking, the pod that was actually slow is usually gone.

Kubernetes has a real metric for this. `kubelet` exposes `kubelet_pod_start_sli_duration_seconds`, an alpha histogram for Kubernetes' narrower startup SLI: it excludes image pulls and init-container execution. SIG-Scalability's official stateless-pod SLO uses that signal: the 99th percentile, measured over the last five minutes, must stay at or below 5 seconds for a cluster-day. So the instinct is to assume the problem is already solved.

It isn't. That metric tells you the cluster-wide p99 crept up this week. It can't tell you which pod, which phase, or why.

## Autoscaling only works if the scale-up is fast

The entire value of a Horizontal Pod Autoscaler or Cluster Autoscaler reacting to load is that new capacity shows up before the old capacity falls over. Consider a scale-up that completes after demand has already outrun the existing pods. The autoscaler technically did its job. It decided to add capacity. But the capacity arrived too late to protect the service: existing pods absorb the overflow, latency degrades, and the new pod can arrive after the pressure has already subsided or triggered a different alarm.

Pod startup latency isn't one number. It's schedule delay, image pull, PVC attach and mount as two separate operations, init containers running in sequence, sidecars starting alongside or before the main container, and the main container itself becoming ready. Every one of those phases is a place autoscaling responsiveness can quietly evaporate, and none of them show up as a single knob to tune. Picture a cluster that scales in a few seconds on a warm node and takes a minute or more on a node that has to pull a fresh image: that cluster doesn't have one autoscaling latency. It has a distribution, and the tail of that distribution is exactly the case autoscaling exists to handle: sudden load, no warm capacity, has to happen fast.

## CI/CD pipelines can spend more time starting pods than doing short work

Tekton, and any CI system built the same way, one pod per unit of work, pays the pod startup cost on every `Task` in a pipeline. A `PipelineRun` with five sequential `Task`s doesn't pay pod startup once. It pays it five times, because each `Task` runs as its own pod with its own schedule, pull, init, and ready sequence. The individual `Steps` inside a `Task` run as containers sequenced within that one pod, so they share the pod's startup cost rather than each paying it separately.

This matters more than it looks like it should, because the actual work inside a Step can be fast. The captured `TaskRun` pod shows its `step-hello` container starting at 16:35:55Z and finishing at 16:35:57Z—under two seconds of recorded wall time. That single short-task example illustrates the dynamic: when the startup overhead around a pod runs for several seconds, it is not a rounding error next to the work inside it. A pipeline that feels slow can be one where the build and test commands run fine but the pods wrapping each `Task` are the bottleneck, and a typical CI dashboard does not separate those two costs.

## The on-call cost: Guessing with kubectl describe pod

When a pod is slow to start in production, not failing, not crash-looping, just slow, the standard move is `kubectl describe pod` followed by reading the `Events` section top to bottom. That gives a chronological list: `Scheduled`, `Pulling`, `Pulled`, `Created`, `Started`, maybe a `FailedMount` or a repeated `Unhealthy` if something is actually wrong. What it doesn't give is any indication of why a phase took as long as it did.

So the on-call engineer does arithmetic by hand: subtracting displayed event times—or, when querying the API directly, Event timestamps—to estimate an image-pull interval, while knowing Events are best-effort records rather than a precise phase clock. If the answer isn't obvious from the event list, there's nowhere else to look. In this project's induced `connect()` stall, the Event record captured lifecycle transitions but did not say that the init container was blocked in `connect()`. Events record controller-observed transitions, not the syscall that occupied the interval between them. The engineer either already knows this failure mode from having seen it before, or starts guessing: the CNI, DNS, the CSI driver, the registry. That guessing has a real cost: time spent per incident, and because nothing captures the answer, the same ambiguity can get re-diagnosed from scratch the next time it happens.

A fair fraction of the pods worth interrogating are already gone by the time anyone looks. Kubernetes garbage-collects completed pods, CI systems delete `TaskRun` pods after a retention window, and a pod that failed to schedule and got evicted leaves no long-lived object to describe. `kubectl describe pod` only works on a pod that still exists. Kubernetes Event objects have their own bounded retention window—one hour by default at the API server—and can outlive the Pod they refer to. Once the Pod is deleted, `kubectl describe pod` no longer works; once the Event TTL expires, the associated chronology is gone too. "Why was that slow" becomes permanently unanswerable through the tools most people reach for first, and for a Tekton `Task` pod whose steps finish in a couple of seconds, that clock starts running almost immediately.

## Why kubectl get events isn't a phase breakdown

`kubectl get events`, or `describe pod`'s embedded events section, gives the raw material a phase breakdown would be built from, not the breakdown itself. It's a flat, chronological list of whatever the kubelet and the various controllers happened to report, in the order they happened to report it. There's no concept of a phase in that stream. No line says "image pull took 10.4 seconds" or "this was the slowest part of startup." `Pulling` shows up at one timestamp and `Pulled` at another, and it's on the reader to subtract them, notice which containers were init containers versus sidecars versus the main container (which, depending on Kubernetes version, can report state in genuinely different shapes: a native sidecar with `restartPolicy: Always` shows up as `running` in the same status array where an ordinary init container shows `terminated`), and reconstruct the ordering by hand. There's no root cause in that stream by construction, either. An Event is a report that something happened, not an explanation of what the process was doing while it was blocked. If a container spent eight seconds inside a `connect()` call, no Event says so, because Events cover what Kubernetes' own controllers observed, and no controller is watching syscalls.

## Why kubelet's own latency metric doesn't answer this either

`kubelet_pod_start_sli_duration_seconds` is a genuinely real, useful signal for a specific question, which makes it tempting to assume it solves this one too. The gap is structural, not a matter of granularity.

Each kubelet exposes an unlabeled histogram for the pods it observes. A monitoring system can aggregate those per-node series into a cluster-wide p99, but that roll-up still has no per-pod label, phase attribution, or trace back to the pod that populated a bucket. No timestamp points at a specific second and says where the time went. A cluster-wide p99 can trend upward on a dashboard for a week straight with no way to go from that trend to a single pod name worth investigating.

`kubectl get events` gives raw, unstructured chronology for one pod at a time, with no phase model and no root cause. kubelet's own metric gives an aggregate trend across the whole cluster, with no per-pod resolution at all. Between a single pod's raw chronology and the cluster's aggregate trend sits the actual question: which pod, which second, why. Neither stock interface covers that middle ground on its own. That gap is the one this series sets out to investigate.