Pitfalls of Progressive Delivery

Testing in production, or the next step to safe releases?

Image for post
Image for post
Testing in production, or the next step to safe releases?

Pace Yourself

Testing in production poses real risks to your infrastructure and users, so the typical strategy is to do it slowly on a small percentage of traffic, then increase this gradually as confidence builds. Having a few requests fail is much better than going down for everyone.

An alternate dimension

When monitoring a progressive rollout, you typically monitor metrics that fall into three categories:

Be patient!

Image for post
Image for post
Good things take time. So does Progressive Delivery.

Stay off the hammock

One very common mistake teams make after successfully adopting Progressive Delivery is to rely too heavily on it. Teams realize that their release process is capable of detecting issues before they hit too many customers, so they get lazy. Integration tests are hard to write and boring, let’s just try it out on a few users and see if it works.

Walk before you run

Progressive delivery is complicated and not for everyone. If your production environment and release process are not already in great shape, trying to implement progressive delivery might cause more harm than good.

  • Can you trace a request through your entire production environment?
  • Do you have defined SLOs? Do you know if you’re meeting them?
  • Are you confident in your rollback mechanisms? Do you prefer to rollback then debug to rolling forward?

Get creative!

Image for post
Image for post
Software Delivery can be an art and a science.

Wrapping Up

I hope this post covered some of the benefits of progressive delivery, while also providing realistic guidance on how to implement it successfully. If you pay attention and plan for the pitfalls outlined above, your organization should be able to keep release velocity high while improving your SLOs by catching issues before they hit all users.

Acknowledgements

I would like to think Kim Lewandowski and Rajeev Dayal for reading drafts of this.

Software Engineer at Google

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store