How to Develop Outcome-Based Engineering KPIs

Most people are familiar with KPIs (Key Performance Indicators). Yet many of my colleagues struggle to nail down which KPIs matter to their engineering team and their organization as a whole. This blog breaks down how to develop outcome-based engineering KPIs most effectively and focus on the outcomes after the fact. 

The KPI process involves many moving parts. It can sometimes feel like single indicators are either measurable and not truly indicative of performance or well aligned with business goals but not measurable.   As engineering leaders, we strive to translate our engineering activities into business outcomes—a challenge for many that I hope to offer practical advice in overcoming. 

The trick is determining which KPIs matter to you, your teams, and your organization. This process starts with understanding why your company even has an engineering team to begin with. 

(Want to skip to the good stuff? Get engineering OKR tracking with Allstacks.)

Understanding Engineering Team Roles

Company leaders determine core objectives that all departments work toward. As an engineering leader, it’s crucial to figure out how your KPIs directly relate to those greater goals. 

At a high level, company goals seek to:

• Gain profitability
• Drive growth
• Deliver customer value
• Offer great customer experience

Determining KPIs is about taking those significant, if a little vague, measures of success and pinpointing the steps to reach them. These outputs need to be stated clearly, mapped out, and measured.

Types of Engineering KPIs

Let’s discuss what KPIs should measure and how they relate to overarching organizational goals and outcomes.  

1. Continuous attainment targets 

Engineering activity KPIs measure things like:

• SLAs
• Uptime
• Compliance
• Headcount growth or reduction
• Bug rates
• Code metrics

You should continuously check and monitor Target KPIs. These KPIs usually fall entirely within engineering’s control but can sometimes slip through the cracks.

A great example would be your escaped defect rate. Most people know about bugs and their impact on customers but don’t dig deeper to track where those bugs originate. It’s a common practice to differentiate between bugs found in live code and bugs that get caught during internal QA. When you start tracking this type of target, you establish a new KPI. Your internal QA process may be built around reducing or eliminating the rate of escaped defects.

2. Externally tracked targets 

Examples of externally tracked target KPIs are traditionally business outcomes that include:

• NPS score
• Feature utilization

These items measure your application’s perception. Most recommend using externally tracked targets as guiding metrics to validate when the application is running correctly and helping you do a better job when it’s not. 

External targets are typically beyond the direct control of your team. Short of asking your engineers to fill out NPS surveys, it’s impossible to ship a feature that will increase NPS by 1 point directly. However, these targets should drive other KPIs or goals in your team. If survey participants overwhelmingly respond that the app is hard to use, you may set a target to spend 10% of your engineering time on UX improvements. This would be an activity that supports the outcome. 

3. Destination targets 

These longer-term shared business and engineering goals aim to meet business objectives internally or externally. For example, marketing could build a campaign supporting a release, and sales may need a new feature added to break into a new industry. Engineers typically view these goals as the breakpoints in the roadmap. 

Regardless of the time or energy required, your team works to build or deliver something. The source of the work may differ depending on the type of goal, but the general tracking process remains consistent.

Most of us will recognize feature delivery, but this includes significant pure-engineering efforts. Goals like paying off tech debts, upgrading to the next major library version, and migrating a production database may not appear on the typical product roadmap. 

Customer success happens when all departments track outputs toward those larger company goals. But how should engineering set up KPIs to make sure customer success happens? Keep reading...

Want to create and visualize the progress of your engineering KPIs in one place? Let us show you how!  

Best Practices for Building Your KPI Tree and Focusing on Outcomes

If you define your engineering KPIs as “what you need to do and when,” then meeting KPIs can quickly become a swirl of urgent but unimportant tasks. To avoid the swirl, set your KPIs with a purpose. Follow these steps to match outputs with the overall desired outcomes of your organization: 

Know the lineage, all the way back to the original company outcome or goal.

Regardless of what you measure, from running costs to average downtime to the number of comments per pull request, it needs to track back to the company goal. How do costs affect profitability? Does rest get shorter? Does customer experience improve all the time? KPIs need to be specific and linked to overall goals. 

Be clear about how each engineering KPI impacts and supports the overarching company KPIs.

You should develop each engineering KPI with the same rigor as the organizational and company KPIs. For instance, it should be clear that measuring “mean time to recovery (MTTR)” helps meet your SLA target. Further, you should be able to explain each KPI by its derivative KPIs fully. For example, your target outcome will be lost if you only track MTTR but not SLAs for the discovered defects. 

Ensure each KPI is measurable or fed by measurable KPIs.

While your KPIs may not have well-defined numerical values or concrete actions, they should be at the next level down. For example, if a goal is to communicate information about new features and their delivery date for marketing to plan promotions, the KPIs below should have measurable metrics. The associated trackable activities for engineering would be to document releases in Jira, outline what was included in the Epic, and create a Confluence page with feature screenshots and full descriptions. Your marketing team can then validate that it meets their requirements.

Make sure teams can control their targets.

A KPI is not a KPI if your team can’t control it. Keep specificity in mind here. You can’t pin broad goals like reducing hosting costs to a single-task user. Keeping things general means no one takes ownership. 

Instead, set narrow, achievable goals. Let’s say your data pipeline team thinks they can make things 30% more efficient. Your database team thinks that caching can save you 20% in disk IOP (Input/Output Operations Per Second) spend, translating to a 10% reduction with 80% confidence. Set your commitment to a 5% reduction in 3 months for a solid KPI, and start delegating the lower targets to the teams and individuals that implement the changes. 

You can now act on this broad goal within specific groups and produce defined outcomes. As a result, your teams will likely set internal targets to reach while understanding why and what outcome to expect.

Final Thoughts on Metrics, KPIs, and Targets in Engineering

KPIs are less about perfection and more about continuous improvement. Implementing an outcomes-based KPI strategy can help provide the gravitational field that keeps engineering aligned with the rest of the company instead of getting caught up in the vague pressures of daily operations. 

What are your thoughts on building outcome-based KPIs? How are you tracking against your desired outcomes? If you need a helping hand in improving your engineering operations, reach out to us to see how we can help or schedule a demo to see our solution in action.  

Our industry is ever-growing, and there’s never a shortage of thought-leaders and forward-thinking entrepreneurs to follow and draw inspiration from. Check out our list of 5 top technology leaders to follow in 2022.