Tekton on Mac

A Holiday Hack Project Trip Report

The Idea

The ability to build and test on Mac is an important part of any CI story, and I’ve been wanting to figure out how we might support this in Tekton for awhile. Kubernetes doesn’t support Mac at all, so I wasn’t really sure how this would work. Then I remembered that MacStadium announced a product that used Kubernetes to orchestrate Macs, called Orka. I wanted to give this a spin and see if Orka might be useful for us, but had trouble finding the time over the last few months. This made it a perfect project for the holidays.

The awesome people at MacStadium set me up with a free cluster and gave me a walkthrough on how to get started using their API and CLI. Unfortunately, Orka didn’t quite work the way I was hoping in my head. I was imagining something like Virtual Kubelet, where you could use some kind of special annotation on a pod to have it scheduled to run on a Mac node. And some service would take care of unpacking the Pod containers on a Mac VM, proxying network connections, executing processes, mounting volumes, etc. If that sounds pretty hand-wavy it is. And there’s probably a reason Orka doesn’t work that way.

There is a Kubernetes cluster involved in Orka, but it’s used more as an implementation detail. You can’t use it directly to run Mac jobs, like Virtual Kubelet or something might work. The user experience of Orka is pretty awesome, even without Kubernetes involved. Once you get setup, you can create a Mac VM from an image with SSH access in under a minute. You can also modify these images and save them, in case you need to add dependencies. It feels very similar to building docker images using docker run and docker commit.

The Plan

So, back to the drawing board. If we can’t schedule containers and pods directly on Macs, what can we do? The next idea I has was inspired by my good friend James Strachan. If you’re stuck on anything in software, the answer is to wrap it! So, that’s what I decided to try. What if instead of running a pod on a Mac, we run a pod on a normal Linux node. Then that pod creates a Mac VM, ssh’s into it and runs the commands there? That didn’t seem like it would be too hard to prototype.

A Tekton Task itself is a set of steps, specified as containers, that run in order inside of a Kubernetes Pod. Tekton takes care of setting up a set of directories under /workspace that contains files representing the input PipelineResources, and locations for the Task to write output PipelineResources. Under the covers, this is implemented as another set of containers that run before and after the steps the user specifies, but this doesn’t matter for our case. All we need to know is that we can expect a bunch of files to exist, and some magic will take care of dealing with the output files we write. To make all of this work on a remote Mac, all we need to do is copy these files over before we run our script, and back after it finishes.

The first part of the prototype was just to get it all working manually. I gave that a try on my laptop. You need to use a VPN to get access to your Orka endpoint and to SSH into the VMs, so I got that running and tried everything out by hand. Creating a Mac VM, SCP’ing a directory from my laptop to the node, running a command, and SCP’ing the directory back all seemed pretty straightforward and worked.

Then it came time to move this into a GKE cluster. The directions for setting up a VPN on GCP took me quite some time to get through. But I got that setup and healthy, then hit my first roadblock. The VPN was working from GCE instances, but not from inside my GKE cluster. I googled around for awhile until I stumbled on IP masquerading. It seemed like I needed to install that agent in my cluster (via a DaemonSet), and configure it to masquerade the IPs for my MacStadium cluster. So, with that setup I could now run all those same commands inside a Pod on GKE, using kubectl run. The config I used looked like this:

Next came the process of translating this into a script that could be embedded inside a Tekton Task. Remember, we’re wrapping the actual user script here. So instead of the Task having a step specified by the user, we have a Task with a hardcoded step and a script parameter. That hardcoded step sets up connectivity with MacStadium, creates a VM, copies over our /workspace directory, runs the user-specified script parameter, copies things back and cleans up after itself.

This is where I ran into trouble. The Orka cli was not designed with scripting in mind. It has a nice wizard mode to help you enter any missing parameters for all of your commands, but it has trouble when run outside a terminal. It took me a long time to get those same commands working in non-interactive mode, and I had to look up a few hacks along the way. But, here’s the working Tekton Task!

Feel free to give it a try, just don’t look at the script too closely :)

Thanks again to the helpful folks at MacStadium for helping me get setup with Orka, and for the free cluster.

What Next?

It would be really nice to find out a way to make Mac tasks feel more native in Tekton. I could imagine some kind of controller that manages a pool of Mac VMs in Orka (or something else), and is responsible for executing Tasks on those, similar to this mechanism. We don’t really have a way to do that in Tekton extensibly today, but maybe we need it.

Windows Tasks are also something I’d like to try out. Although since Kubernetes supports Windows now, it should be roughly straightforward. We’ll need to cross-compile all of our containers and tools to run on Windows, then support scheduling Task Pods onto Windows nodes.