Running macOS builds from Tekton with a CustomRun

A passing xcodebuild command does not tell Tekton whether its disposable VM has been deleted. The build outcome and VM cleanup are separate facts. On the normal completion path, this controller keeps the CustomRun running until the Mac agent confirms deletion. Cancellation and lost-agent recovery need separate rules.

The Mac agent connected in Part 0 can receive build requests, but a TaskRun still needs a Pod. Its steps depend on Tekton’s injected entrypoint and files shared inside that Pod. The agent runs guest commands through tart exec. Moving execution outside the Pod also moves control of the remote process and its cleanup to the integration.

The macOS build therefore needs an external job with its own status and cleanup owner, rather than a TaskRun step executing on another machine.

CustomRun gives the controller an external job to track

I chose CustomRun, Tekton’s extension point for custom tasks, so a dedicated controller could track the external build and its cleanup while the Mac remained an outbound API client. A Go controller watches the CustomRun and creates a MacBuild describing the build. The Rust agent watches that object and executes the requested build inside a fresh Tart VM. It records the build outcome and cleanup status on the MacBuild.

The controller has to define status, respond to cancellation and enforce deadlines. The agent has to execute the build and confirm cleanup. Disposable VMs could also be managed from an ordinary remote-execution step. CustomRun makes the external job’s status visible to its own controller.

Part 0 showed the later EKS deployment. These local cases established the earlier execution contract, with the Go controller running on the Mac with a kubeconfig for kind. The diagram uses an illustrative cluster name:

flowchart LR
  subgraph cluster["kind cluster mac-ci-local (Linux node)"]
    PR[PipelineRun] --> TR[TaskRun: Linux Pod]
    PR --> CR[CustomRun]
    MB[(MacBuild)]
  end
  subgraph mac["Mac host (external API clients)"]
    CTRL[Go controller]
    AG[Rust agent, LaunchAgent]
    VM[disposable Tart VM]
  end
  CTRL -- watches --> CR
  CTRL -- creates --> MB
  MB -- watched by --> AG
  AG -- "tart exec: xcodebuild" --> VM
  AG -- "status, VMDeleted" --> MB
  MB -- status --> CTRL
  CTRL -- "condition and results" --> CR

A TaskRun expects more than a command

In this one-step Pipeline, the PipelineRun references a TaskRun, which owns the Pod. Tekton replaces the step’s command with its entrypoint binary. A prepare init container copies that binary into a shared volume. This excerpt is abbreviated, and the image reference is shortened:

1
initContainer: prepare
2
  image:   <Tekton entrypoint image, v1.15.3>
3
  command: /ko-app/entrypoint init /ko-app/entrypoint /tekton/bin/entrypoint step-uname
4
container: step-uname
5
  image:   alpine:3.22
6
  command: /tekton/bin/entrypoint
7
  args:    -wait_file /tekton/downward/ready -wait_file_content
8
           -post_file /tekton/run/0/out -termination_path /tekton/termination
9
           -step_metadata_dir /tekton/run/0/status -results arch -entrypoint /bin/sh -- -c "uname -sm; ..."

The original /bin/sh -c ... is now an argument to Tekton’s binary. Tekton’s container contract describes this replacement. The paths and flags above are what v1.15.3 generated on this Linux cluster.

Step ordering uses files in shared volumes and a file mounted from a Pod annotation through the downward API. Tekton reads the step result from its termination message and copies it to the TaskRun as arch=aarch64. tart exec runs the guest command without supplying the init container, shared Pod files or termination-message path.

A node selector needs a matching worker

The first failure was placement. The test TaskRun specified xcodebuild -version, a macOS node selector and a 120s timeout:

1
spec:
2
  timeout: 2m
3
  podTemplate:
4
    nodeSelector:
5
      kubernetes.io/os: darwin

The cluster’s only node was linux on arm64. The Pod stayed Pending, and the scheduler recorded the mismatch:

1
Warning  FailedScheduling  default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. ...

After 120s the TaskRun reported False with reason TaskRunTimeout, and the Pod was gone. The step never ran.

The kubelet sets kubernetes.io/os from Go’s GOOS. The Kubernetes worker-node documentation covers Linux and Windows. EKS Hybrid Nodes lists supported Linux distributions. The selector matches an existing worker label. It does not register the external Mac as a worker.

A Darwin binary would solve only one problem

A node advertising darwin could clear this selector mismatch. The next dependency is the injected binary. Tekton’s v1.15.3 entrypoint image lists linux/amd64, linux/arm64, linux/ppc64le, linux/s390x and two windows/amd64 builds, with no darwin entry.

The extracted linux/arm64 binary had the same SHA-256 on the host and in a fresh macOS 27.0 guest. Executing it in the guest produced these errors:

Both are arm64, but the file is a Linux ELF executable. macOS loads Mach-O, so execve returns ENOEXEC.

These invocations used Tekton’s published Linux entrypoint binary. A build of its Go source targeting darwin was not tested. A compatible binary would solve only the executable-format problem. A provider would still need to implement the Pod interfaces for TaskRun execution.

The controller returns test results to Tekton

A PipelineRun ran a Linux prepare TaskRun, then delegated the macOS task through a CustomRun referencing the MacBuild kind. The agent ran the tests in the VM, and the controller returned their counts as string results on the CustomRun.

The CustomRun reported TestsPassed and two passing tests. This excerpt is abbreviated and uses an illustrative API group:

1
spec:
2
  customRef:
3
    apiVersion: mac-ci.example.com/v1alpha1
4
    kind: MacBuild
5
  timeout: 10m0s
6
status:
7
  conditions:
8
    - type: Succeeded
9
      status: "True"
10
      reason: TestsPassed
11
      message: 2/2 tests passed
12
  results:
13
    - {name: testsTotal, value: "2"}
14
    - {name: testsPassed, value: "2"}
15
    - {name: testsFailed, value: "0"}

The PipelineRun finished Succeeded in 58s. The MacBuild showed VMDeleted True. Deleting the PipelineRun afterwards removed the CustomRun and MacBuild. The agent released its finalizer.

A build verdict does not mean the VM is gone

Tekton v1.15.3 reads the CustomRun’s Succeeded condition and named string results. It leaves CustomRun status to the custom controller. On the normal completion path, the controller maps the agent’s build verdict, but holds the CustomRun at Unknown until the agent confirms VMDeleted=True.

Part 0’s EKS trace recorded the PipelineRun as Cancelled before VM cleanup completed. Those timestamps cross cluster and Mac clocks whose offsets were not measured. A terminal PipelineRun outcome alone does not confirm external cleanup.

Tekton requests cancellation by setting spec.status: RunCancelled. The controller’s cancellation path deletes the MacBuild and keeps the CustomRun at Unknown with reason Cancelling while the child still exists. The agent’s finalizer keeps that deleting object present while it stops and deletes the VM. On a subsequent reconcile, a NotFound response for the child lets the controller report False with reason CustomRunCancelled. Tekton requires that exact reason to count the custom task as cancelled.

The controller interprets the child’s absence in the context of the cancellation request. It relies on the agent releasing its finalizer only after cleanup, rather than independently checking the VM. Absence alone cannot distinguish completed cleanup from a child whose finalizer was removed outside that protocol.

These local cases tested the agent’s half of that path by deleting a MacBuild during execution. The controller’s PipelineRun cancellation path was reviewed in source, but was not exercised here.

Waiting for cleanup also needs a limit when the agent stops responding. Part 4 extends the controller to end a timed-out or cancelled CustomRun when the agent’s lease has expired. It reports vmRetirement=unconfirmed because it cannot confirm that the VM is gone.

The VM was gone after success, failure, and deletion

The successful integration above exercised the PipelineRun path. The following four cases exercised MacBuild directly, using a pinned commit of a small macOS app with two XCTest tests and one VM slot on kind. They did not test failure or cancellation through a PipelineRun.

The agent ran as a launchd LaunchAgent in the user’s GUI session. In this guest, tart exec used the auto-logged-in user’s GUI session for xcodebuild test. Running without that session was not tested.

After each case, tart list showed no build clone. VMDeleted was True for the pass, failing-test and bad-commit cases. For the bad commit, the VMDeleted condition’s message reads “no VM was created”. The failure happened during source fetch, before VM allocation.

In the deletion case, stopping and deleting the VM took 6.39s. The object became NotFound 9.10s after deletion was requested. The agent released its finalizer after the VM was gone.

Each row describes one run on one Mac and one sample app. They establish the observed outcomes and cleanup ordering. Part 5 measures timing across repeated runs.

A timeout field still needs someone to enforce it

Tekton’s CustomRun contract distinguishes fields it passes to the controller from behavior the controller must implement. The division below follows the v1.15.3 source:

In v1.15.3, CustomRun.GetTimeout() defaults an unset timeout to 60m. This implementation has an agent deadline and a controller backstop, but neither was reached in these runs.

This baseline did not implement retries or customSpec handling. Part 4 later adds bounded agent retries. Tekton permits custom-task controllers to ignore retries. Callers should rely on retries only when the controller explicitly supports them. Unsupported customSpec requests need an error in status.

Other routes still need a cleanup owner

CustomRun is one way to give the external job a cleanup owner. A Linux step can connect to the Mac, a custom controller can own the job, or a virtual-node provider can represent the VM as a Pod. The choice depends on which component will track the remote process and confirm cleanup.

Only the CustomRun integration was exercised here. The alternatives were reviewed in documentation and source; their cancellation behavior was not tested.¹

orka-tekton uses an exit trap to delete the VM when delete-vm is true, with a sample finally teardown task. Konflux’s controller uses a TaskRun finalizer to deallocate the host. Neither mechanism alone confirms remote cleanup.

For plain SSH, OpenSSH bug 396 describes a remote command left running after the client exits when no PTY is allocated. A remote-execution step therefore needs an explicit cleanup protocol.

Konflux removes its finalizer even if deallocation fails. The orka-tekton source has no explicit cancel handling beyond its teardown mechanisms.

As I read Agoda’s post, the provider suits CI systems that drive a running VM from outside the Pod. Agoda describes running GitLab’s Kubernetes executor this way, with the VM as the build environment and helper containers in Docker.

That does not supply the TaskRun contract traced above. Agoda’s provider omits init containers. In the v1.4.1 source I reviewed, it does not run the first container’s command and args, so this TaskRun would need Tekton-specific changes to run there. Part 6 compares the execution models and other ways to run Mac builds.

The agent must complete the cleanup

The macOS stage exposes parameters and string results. Steps, sidecars and shared Pod volumes remain features of ordinary TaskRuns. Linux stages can still run ordinary TaskRuns in the same Pipeline, as the prepare task did here. The external path needs controller-enforced deadlines, explicit retry behavior, scoped API access and recovery when the agent or controller disappears.

Part 2 follows how source and artifacts cross that boundary without a shared workspace. Part 3 explains how the agent admits a build, allocates its VM and stops and deletes it before releasing the job.

CustomRun gives Tekton an external job to track. Reliable execution and confirmed cleanup still depend on the protocol between the controller and the Mac agent.

References

1. Container Contract, Tekton Pipelines
2. Windows in Kubernetes: nodes and worker OS support
3. Well-known labels, annotations and taints, Kubernetes
4. Prepare operating system for hybrid nodes, Amazon EKS
5. Custom Tasks and CustomRuns, Tekton Pipelines
6. orka-tekton, MacStadium
7. multi-platform-controller, Konflux
8. macOS-vz-kubelet, Agoda
9. How We Integrated Native macOS Workloads with Kubernetes, Agoda Engineering