DevOps Benchmarking: You Can’t Manage What You Can’t Measure

At some point along the way, somewhere between tiny-little-projecty-idea-thing and commercially-viable-business-with-real-customers, things get complex. While our business teams are celebrating “scaling,” your development teams are acutely aware of the challenges that arise from growth -- exponential increases in infrastructure and system complexity, code volume, and technical debt.

So, we go into “Management Mode.” And we think, “Hey, all of these services we’re using, they provide event streams -- this will be fine, I can get the data! I can measure it." Then you realize capturing and interpreting all the data streams is really time-consuming. What may start as a single script using the Github API can quickly grow into event streams from hundreds of developers. Suddenly, you’re building a full-blown internal tool for development team measurement.

At Allstacks, we’ve spent years understanding the software development lifecycle and how it often resembles a winding trail through the woods rather than an assembly line. As we baseline a customer’s dev lifecycle, we typically find an immediate opportunity to improve CI/CD. So many teams are hamstrung by slow builds and manual deploys!

First, we start monitoring build times. If you decrease build time, cycle time is reduced overall -- meaning your developers can get back to coding faster. But build times don’t tell the whole story. What if build times are relatively short, but builds don’t actually begin for quite a while? It turns out a better metric to monitor may be what we call Build Delay.

Shorter build times lead to faster iterations in a vacuum, but Build Delay is a nuanced metric providing an answer to what engineers actually care about: “When is this actually going to be done?” If someone kicks off a 2-minute build, but it takes 30 minutes to start, that’s 32 minutes before your developer can iterate on their work. Shortening that Delay positively impacts the developer and everyone else downstream -- including your customers.

Evaluating the context of the metric and understanding the business impact of the output leads to healthier, more informative measurements. At Allstacks, we’ve seen customers recaptured dozens of hours per month of productive time by focusing on driving improvements to Build Delay. It’s amazing how valuable a little clarity can be when things have gotten complicated.

So, yeah -- you can’t manage what you can’t measure... but you also can’t improve what you don’t understand.