This took me awhile to figure out, so I thought I would write it down in the hopes it saves someone else time later.

I have a Golang app that runs on Kubernetes (in my case GKE) that I want to add some basic monitoring to. I was hoping for a solution that roughly meets these requirements:

  • No vendor-specific code in my application
  • Easy support for exposing custom metrics
  • Since I’m running on GKE, it would be nice for the metrics to show up in the StackDriver console

Searching and reading examples led me down a two routes: OpenTelemetry/OpenCensus and…


Making “many eyes” a reality with automation

Many advocates of open source software claim it is easier to make secure because it has “many eyes” looking at it. It is probably true that in the limit enough eyes can catch all bugs, but there’s no evidence we’ll ever hit that limit. Thankfully, we can help add more eyes to OSS with automation. This post explains how I built a platform to hunt for malicious behavior hiding in plain sight.

Image for post
Image for post
Photo by Victor Freitas on Unsplash

The specific type of malware I’m looking in this post embeds itself in open source packages, and exploits the fact that…


Not really. But Kind of?

Did you know that you probably already have a working PKI system for signing artifacts on your laptop today, with no keyservers, web-of-trust, or configuration? You can use it to sign files, and to find the public keys for other people and use them to verify files they signed.

So why aren’t more people using this? I think it’s just gone overlooked because it’s a relatively new feature in apretty old piece of software. I’ve only been able to find one other blog post explaining how it works (outside of the man pages).

Sign on the dotted line.
Sign on the dotted line.
Photo by Lewis Keegan on Unsplash

Signing and Verifying

Since early…


More haunted dependencies in the jungle

This is Part Two of my Dependency Jungle series. (Part 1 here). I’m trying to better understand how large networks of projects manage dependencies, vulnerabilities and upgrades, so I decided to spend some time doing it myself. By documenting and sharing this experience, I’m hoping we can come up with ideas to systematically improve this process, making it faster and easier for real-world projects to stay up to date and secure.

From time-to-time I live-tweet these adventures, so consider following me on Twitter for some bite-sized versions of archaeology like this.

In the last…


Or, how to deploy 25+ CVEs to prod in one command!

WARNING: This is very long. I tried to trim it down, but decided against splitting it. Showing all steps required and how long this took was the point of the exercise. I expect most people not to finish reading this, and that’s OK — I didn’t even finish analyzing everything. At least scroll to the bottom so you can see how much there is, and consider following me on Twitter for some bite-sized versions of archaeology like this.

When the first railroad train was built over Niagara Falls, engineers…


The Dependency Tree is Actually More of a Jungle. And it’s Haunted.

I was looking through the Kubernetes go.mod file, and noticed something weird. A few strange-looking dependencies that didn’t seem to belong. I’m still not quite sure what caught my attention about these specific modules — there are over 300 direct and indirect dependencies required to build Kubernetes — but these particular ones really didn’t make sense to me.

On a normal day I might have just gotten distracted and moved on, but I decided to really dig in here. I wanted to better understand the state of Go modules, the tooling around them, and what is going on in the…


And the connected web of open source.

This isn’t another post talking about how many CVEs are in open source code (a lot), or how hard to use the CVE system is (very). It’s also not about how under-paid and overworked maintainers are (depressingly so), although it does touch on these topics. This is a post about another reason to update dependencies, one that I just realized.

Image for post
Image for post
Why not? It’ll only take a minute…

I was playing around with the snyk CLI to see the CVEs present in a few of my codebases. I had some spare time and wanted to see if I could fix a few. I started with Tekton, a project…


…and other real world alternatives to TDD

Image for post
Image for post
Photo by Pedro da Silva on Unsplash

We've all heard of Test Driven Development, and maybe even tried it. It's not my favorite way to write code - it feels like navigating along a highway by bouncing back and forth between guide rails you're rapidly installing right before you hit them.

In this post, I'm going to explain some of my actual favorite ways to write software.

Procrastination Driven Development

Also known as procrasti-coding, this is when you write software to avoid doing a less desirable task. Performance reviews, expense reports, documentation, and that conference presentation coming up next week are all great choices. …


Does Leadership Have to Be A Balancing Act?

I’ve been in the field of software engineering for a decade now. I’ve had the chance to work with many leaders, and to be one myself. This post contains some anti-patterns I’ve seen and at times fallen into in my roles.

Image for post
Image for post
Photo by Jon Flobrant on Unsplash

At MIT, in the Mechanical Engineering Department, Course 2.003 is Dynamics and Controls. Students in this class, at least when I studied there, learned how to model and simulate the motion of complex systems. The final project in the class was to design a control system to keep an inverted double pendulum upright and stable. The control system was…


Testing in production, or the next step to safe releases?

Image for post
Image for post
Testing in production, or the next step to safe releases?

Progressive Delivery is the technique of rolling out software to users incrementally to improve the safety and reliability of the deployment process. Relying on automated testing to catch every issue before it hits production is like abstinence-only sex education — both are doomed for failure in the real world. Progressive Delivery acknowledges that mistakes will happen no matter how hard we try to prevent them. If we’re going to accidentally release bad software, let’s at least do it carefully.

When done right, Progressive Delivery can reduce the risk of a software deployment while keeping release velocity high. When done poorly…

Dan Lorenc

Software Engineer at Google

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store