# Pod Startup Forensics: Surviving Deletion

A pod that started slowly is often gone by the time anyone asks why. `kubectl describe` returns `NotFound`; its Events may linger briefly, but on a retention schedule unrelated to the pod itself. The evidence needed to explain the startup has become a race against deletion and garbage collection.

To answer after deletion, this exporter has to capture the evidence while the pod exists and reconstruct it from durable storage later. This part shows the storage design behind that reconstruction and the `--timeline` command that reads it back.

## Capture it before deletion

The exporter writes the observations it needs to reconstruct a startup to a database. Postgres is the deployment choice for shared, durable history; SQLite offers the same `--timeline` behavior in one local file for development or a single-node demo. With either storage option, the owner, child, pod, phase, and event records form one hierarchy, so a deleted pod can still be found under its original `PipelineRun`, `Job`, or `CronJob`.

## What actually gets stored

For the deletion reconstruction in this part, two stored record types carry the core answer: phase records for duration and an ordered log of Kubernetes Events for explanation. Phase records retain timings that Events can't provide, especially container start and finish times. Events retain the verbatim reasons, messages, and retry history that explain delays outside those phase boundaries. `--timeline` joins both when it reconstructs a pod.

The output preserves the precision it actually captured. A populated Event `eventTime` can carry microseconds, while the older `firstTimestamp` and `lastTimestamp` fields are whole-second values; `--timeline` doesn't invent fractional precision for the latter.

## Reconstructing a PipelineRun after its pods are deleted

The payoff is a timeline that remains readable after the pods are gone. A three-`TaskRun` Tekton `PipelineRun` produced three pods on a kind cluster. One deliberately blocked for about thirteen seconds against a non-routable address before giving up, giving the timeline a known slow phase to find. All three pods were then deleted directly:

```bash
$ kubectl delete pod -l tekton.dev/pipelineRun=multi-taskrun-run-slow
$ kubectl get pods -l tekton.dev/pipelineRun=multi-taskrun-run-slow
No resources found in default namespace.
```

`kubectl` has nothing left to say about any of these three pods. A direct query against Postgres confirms the same thing from the storage side: all three pod rows carry a non-null `deleted_at`, with their phases and events still attached:

```bash
         owner          |                   child                    |                      pod                       | deleted | phases | events 
------------------------+--------------------------------------------+------------------------------------------------+---------+--------+--------
 multi-taskrun-run-slow | multi-taskrun-run-slow-first-task-run      | multi-taskrun-run-slow-first-task-run-pod      | t       |      3 |      7
 multi-taskrun-run-slow | multi-taskrun-run-slow-second-task-run     | multi-taskrun-run-slow-second-task-run-pod     | t       |      4 |     18
 multi-taskrun-run-slow | multi-taskrun-run-slow-third-task-run-slow | multi-taskrun-run-slow-third-task-run-slow-pod | t       |      3 |      7
(3 rows)
```

With no matching live pods in the PipelineRun's namespace, `--timeline` reconstructs the stored history alone:

```bash
$ profiler-cli lookup --kind tekton.dev/v1.pipelinerun \
    --name multi-taskrun-run-slow --namespace default --timeline \
    --exporter-url http://127.0.0.1:19090

Owner historical timeline: default/multi-taskrun-run-slow
(reconstructed from durable storage — works even if every pod below is now deleted)
══════════════════════════════════════════════════════════════════════

Child: multi-taskrun-run-slow-third-task-run-slow
  Pod: multi-taskrun-run-slow-third-task-run-slow-pod  [deleted]  node=pod-startup-profiler-control-plane
  ──────────────────────────────────────────────────────────────────
    schedule                            0.335s
    init_container (prepare)            0.000s
    main_container (step-blocking-step)   13.000s <-- SLOWEST PHASE IN PIPELINE
  ──────────────────────────────────────────────────────────────────
    TOTAL                              14.000s
    ...

══════════════════════════════════════════════════════════════════════
FLAGGED SLOW: main_container (step-blocking-step) in Child multi-taskrun-run-slow-third-task-run-slow / pod multi-taskrun-run-slow-third-task-run-slow-pod took 13.000s (longest phase across the whole owner chain)
```

All three `TaskRun`s reconstruct with their pods labeled `[deleted]`; the excerpt shows the deliberately blocked 13-second step as the owner chain's longest phase. A second read returned the same bytes, and a fresh three-pod run reproduced the same slow-phase flag after deletion.

## When no container ever starts

The deleted PipelineRun shows that completed startup history survives deletion. A pod blocked on an unbound `PersistentVolumeClaim` never records a container phase, so there is no `started`/`finished` pair to time. The exporter used to drop that shape entirely. Now `--timeline` renders `no phases recorded` and shows the captured Event log.

No container started, so this timeline has no syscall or probe records. The CLI keeps its standard merged-timeline heading, but only captured Events follow:

```bash
Pod: pvc-failure-demo  [exists]  node=?
──────────────────────────────────────────────────────────────────
  no phases recorded — this pod never reached container start; see the event log below for why
──────────────────────────────────────────────────────────────────
  TOTAL                               0.000s

  MERGED TIMELINE (K8s Events + eBPF syscalls/probes, interleaved by real timestamp)
  2026-08-30T04:13:26Z  T+  0.000s  [K8S EVENT]  ExternalProvisioning   PersistentVolumeClaim/rook-ceph-stuck-pvc (x2 repeats)
      Waiting for a volume to be created either by the external provisioner
      'rook-ceph.rbd.csi.ceph.com' or manually by the system administrator. If volume creation
      is delayed, please verify that the provisioner is running and correctly registered.
  2026-08-30T04:13:26.572306Z  T+  0.572s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
  2026-08-30T04:13:26.578279Z  T+  0.578s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
```

That output names the provisioner handling the claim and records the scheduler's immediate blocker: an unbound immediate PersistentVolumeClaim. It identifies the next component to inspect and explains why the pod can't be scheduled. A phase-durations-only view has no row for this pod, regardless of how long it waits.

This capture stores only two repeats. A stored count can understate a persistent failure because the exporter refreshes the Event set only when it observes the pod; later Event changes can be missed if no further pod observation occurs. This run alone doesn't show that happened here.

The previous case retained phase data; this pod never reached container start. The same `lookup --timeline` interface covers both shapes.

## Open questions

This test doesn't answer whether the database keeps up under sustained production load.

[Part 6](/blog/pod-startup-forensics-closing-the-gap) follows a flagged slow phase into eBPF syscall evidence, then fixes a bug in assigning those syscalls to the right pod.