Connecting a Mac build host to a private EKS cluster

A Tekton pipeline needs a Mac to build and test a macOS app, but the EKS API is private and the Mac has no route into its VPC. With inbound access to the Mac also ruled out, the Mac has to initiate the connection.

I used an outbound AWS Systems Manager (SSM) port-forward through an EC2 relay in a private subnet. The Mac stays outside the cluster, watching for build requests through the API server; macOS is not a supported worker operating system for EKS Hybrid Nodes. Through that tunnel, an agent on the Mac receives requests, runs each build in a disposable macOS VM, and writes the result back.

In a 20-call batch with no failures, a fresh kubectl call had a median of about 1.35s through the tunnel, against 35ms on local kind. Those timings include different network and authentication paths, so they do not isolate tunnel overhead. Build requests, source and artifacts cross that path, while VM startup and Xcode execution stay on the Mac.

Tekton owns the pipeline; the Mac runs the build

Tekton and the build controller run in the cluster. When a pipeline reaches its macOS step, the controller creates a MacBuild, a custom resource that describes one build. An agent on the Mac watches for MacBuild objects and uses Tart to clone a macOS VM. It runs xcodebuild inside the guest and deletes the VM when the build ends.

Tekton tracks the external stage with a CustomRun, its custom-task object. The diagram follows a completed build. After the tunnel is established, the API arrows show logical exchanges through it. Failure and cancellation follow different paths.

sequenceDiagram
    participant P as Tekton + controller (EKS)
    participant A as API server (private endpoint)
    participant R as SSM relay (private subnet)
    participant M as agent on the Mac
    participant V as Tart VM

    Note over R,M: Mac and relay connect outbound to SSM
    M->>R: establish SSM port-forward
    R->>A: forward TLS to the endpoint, port 443
    Note over A,M: subsequent API traffic uses this tunnel
    P->>A: create MacBuild
    A-->>M: watch event
    M->>A: fetch source (service proxy to registry)
    M->>V: clone, boot, xcodebuild build + test
    M->>A: push artifact, record build outcome
    M->>V: stop and delete
    M->>A: write cleanup-complete status
    A-->>P: final CustomRun result

This part follows the Mac agent’s connection to the private API server. The rest of the series develops the execution contract and VM lifecycle behind it.

How do you reach an endpoint you can’t expose?

Two settings on the cluster make the endpoint private:

1
vpc_config {
2
  endpoint_private_access = true
3
  endpoint_public_access  = false
4
}

The VPC has two private subnets and no public ones. A regional NAT gateway lets the private nodes pull container images. It provides no route for unsolicited inbound connections from the Mac to those private resources.

The EKS documentation lists connected networks and access hosts inside the VPC as routes to a private endpoint. For one Mac connecting to a short-lived cluster, I chose an EC2 relay managed by SSM. It needs no inbound SSH access, and the network and relay can be created and removed with the cluster.

The Mac opens an outbound SSM tunnel

The relay is a t4g.nano instance with no public IP and no inbound rules. Its only egress is HTTPS and DNS. The SSM agent on it keeps an outbound connection to Systems Manager, and that connection is how the Mac reaches it.

The Mac starts a port-forward to a remote host through the relay:

1
aws ssm start-session --target <relay-instance-id> \
2
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
3
  --parameters '{"host":["<endpoint-host>"],"portNumber":["443"],"localPortNumber":["8443"]}'

AWS-StartPortForwardingSessionToRemoteHost forwards a local port on the Mac to a host the relay can reach. The remote host does not have to be managed by SSM, so it can be the EKS endpoint. Port 8443 on the Mac now reaches port 443 on the API server. A small shell loop restarts the session if it ends.

The kubeconfig points at the local port and preserves the endpoint hostname for TLS verification. This excerpt is abbreviated. The full kubeconfig retains the generated CA data, context and exec API version. mac-ci is an example name:

1
clusters:
2
  - name: mac-ci
3
    cluster:
4
      server: https://127.0.0.1:8443
5
      tls-server-name: <endpoint-host>
6
users:
7
  - name: mac-ci
8
    user:
9
      exec:
10
        command: aws
11
        args: [eks, get-token, --cluster-name, mac-ci, --output, json]
12
        interactiveMode: Never

tls-server-name handles the hostname mismatch introduced by the tunnel. The API server’s certificate is issued for the cluster’s hostname, not for 127.0.0.1. The field tells the client which name to verify, so TLS still checks the real certificate while the connection goes to a local port.

TLS verifies the server’s identity. The cluster still has to know who is calling. Credentials come from aws eks get-token, and an EKS access entry attaches Kubernetes permissions to an IAM identity. Here, one entry gives my IAM user the AmazonEKSClusterAdminPolicy. The agent shares that identity and therefore has cluster-admin rights.

Source and artifacts take the same tunnel

A build needs source going in and an artifact coming out. Both live in a small registry inside the cluster. The agent doesn’t connect to the registry directly. It goes through the API server’s service proxy, using the same tunnel and the same token. The URL below uses illustrative mac-ci names:

1
https://127.0.0.1:8443/api/v1/namespaces/mac-ci/services/http:mac-ci-registry:5000/proxy/v2/...

Source and artifact transfer need no separate network path into the Mac. The registry logged the relay’s VPC address for the captured source and artifact requests. Requests from cluster pods carried their pod IPs. I used the service proxy to keep source and artifact requests on one authenticated API route. That puts artifact traffic through the control plane. For a runner fleet, I would keep artifact transfer off the Kubernetes API server and evaluate direct artifact-store access with its own network path and permissions.

A fresh API call took 1.35s through the tunnel

The Mac reached an EKS cluster in us-west-2 over the public internet through SSM; kind ran locally on the same Mac. I timed kubectl get --raw /readyz as a fresh process against EKS and kind on the same Mac. The EKS row below summarizes a 20-call batch with no failures. A separate 20-call EKS batch had four TLS handshake timeouts; the table characterizes request timing in the failure-free batch, not connection availability.

I also timed curl with a pre-fetched token, using new and reused connections to see how much reuse changed the request time. The reused row excludes the first request, which established the connection. The p90 uses nearest-rank: the 18th value in each sorted sample of 20 or 19 requests.

The fresh-process comparison includes different authentication paths: EKS runs aws eks get-token, while kind uses a client certificate. Token generation had a median of about 230ms in five calls.

For the small /readyz response, reusing a connection reduced curl’s median from 1,124ms to 643ms. Reuse saved setup work, but the request still took hundreds of milliseconds. These calls measure kubectl and curl. They do not establish the agent’s connection reuse or transfer throughput.

Slower API calls, the same build outcomes

The same four pipeline cases ran on EKS as on kind. All four produced the expected outcome:

In the pass case, the built app was copied out of the VM, pushed to the registry, and its digest checked again from a Linux pod in the cluster. After every case, tart list showed no clone, only the base images. The Mac used the same agent binary with a different kubeconfig.

The PipelineRun read Cancelled roughly 7s before the VM was gone in the captured timestamps. That comparison crosses cluster and Mac clocks whose offsets were not measured. CustomRunCancelled confirmed that cleanup had finished. Part 3 follows that distinction.

The outcomes match. The next question is where the remote path adds elapsed time. Watch delivery, source fetch and artifact publication use the remote API path. VM boot, compilation and cleanup run on the Mac. The table shows selected intervals from one run per case per cluster, for test-fail, cancel and compile-error. Ranges span those three cases.

The total starts at PipelineRun creation and ends when the controller writes final CustomRun status, so it includes cleanup after cancellation. The selected rows do not sum to that total:

Watch delivery crosses the Mac and cluster clocks, whose offsets were not measured. Those timestamp differences do not establish watch latency. The totals start from whole-second Kubernetes timestamps, so they are rounded here.

Source fetch, artifact publication and cancellation took longer on the EKS path. The local phases did not change in a consistent direction. Matched case totals differed by roughly 0s for test-fail, +4s for cancel and -2s for compile-error. The cancellation case interrupts execution, so its total is not a completed build time.

The successful case took about 79s to final CustomRun status on EKS versus 66s on kind. The pass trace shows an additional delay before MacBuild creation. Its cause was not isolated, so the roughly 12s difference cannot be attributed to the tunnel.

These are single paired samples on one Mac, not a latency benchmark. Variation in VM readiness is comparable to the observed end-to-end differences. These timings do not isolate the tunnel’s overhead from the other differences between EKS and kind.

What would it take to rely on this?

Ongoing builds would need a dedicated agent identity, tested recovery and persistent artifact storage.

The watch delivers events without polling for each change. Status writes, lease renewals and source and artifact transfers still use the remote API path. Before choosing this topology for a runner fleet, I would measure concurrent watches, status-write and lease-renewal frequency, service-proxy throughput, and API-server load at the expected build rate.

I would give the agent a dedicated IAM role mapped to a Kubernetes group, with permissions limited to the required MacBuild operations, status updates, finalizer changes, its Lease, and the registry’s service proxy. That scoped identity still needs validation.

All API traffic depends on one relay. An alternate path could be a second relay with tested session failover or a VPN. Tunnel reconnection and expired credentials were not exercised on EKS. The deliberate recovery fault-injection cases in Part 4 ran on kind. The registry used emptyDir storage. Ongoing use would need persistent artifact storage.

When a tunnel isn’t the right fit

If a private-only endpoint is not required, public access with a CIDR allowlist is another option. Public access is the EKS default, and the allowlist limits it to known addresses. It fits when the Mac has a stable egress IP and a public control-plane address is acceptable.

A client or site-to-site VPN into the VPC fits when many machines or people need the endpoint and a managed network already exists. Connecting one Mac that way adds networking infrastructure to manage. Neither the public-endpoint route nor a VPN was measured here. A closer host or a different network path would need its own latency measurements.

A connection is not an execution contract

The Mac remains an external API client. Moving the control plane to EKS changed how build requests reached the agent; execution stayed inside the macOS VM. The agent owns the VM lifecycle, and the controller reports the external job’s completion to Tekton.

The tunnel carries build requests and status updates, but it does not define what counts as a finished job. Part 1 asks the harder question: what can Tekton promise about a macOS build that runs outside a Pod, and when is that external work actually finished?

References

1. Start a session (port forwarding to remote host), AWS Systems Manager
2. Cluster API server endpoint, Amazon EKS
3. Grant IAM users access to Kubernetes with EKS access entries
4. Prepare operating system for hybrid nodes, Amazon EKS