4 Common Pitfalls to Avoid When Tracking Engineering KPIs

In the previous article, I walked through the process of establishing outcomes-based engineering KPIs through:

• Identifying the role the engineering function plays in achieving overall company goals
• Understanding the kinds of outputs that will move the needle in achieving those goals
• Framing out the steps to get there

Sounds simple, but things can get complicated quickly when you’re juggling multiple projects, remote teams, and constantly evolving processes. This article in the KPI Best Practices Series will outline a few of the most common pitfalls in establishing an outcomes-based KPI framework and how to avoid them.

Pitfall #1: Tracking activity instead of outcomes

Engineering KPIs need to focus on measuring outcomes, not activities, no matter how vital those activities may be. This is a straightforward trap to fall into. You can capture data on what your team is doing and create a compelling story of why this data shows good progress, but without tying those activities to outcomes, you can spend time and energy in the wrong place. For instance, You may track commit activity or details about a code review process when you’re not even sure there are issues present. Delivering more code seems optimistic, but moving faster doesn’t always mean moving in the right direction. 

Instead, compile outcomes-based data in a centralized KPI dashboard, and map them back to the goals your leadership wants to accomplish. Track outcomes like shipping releases and bug rates that directly support company-wide objectives. 

Pitfall #2: Misalignment in priority

Now that outcomes—and not just activities—are tracked in your KPI framework, another mistake to avoid is around your priorities. Teams work hard to produce the best deliverables possible, but this is one instance where perfection can be the enemy. If you focus your energy on a single KPI or outcome, you can meet that goal, but at the expense of other outcomes. For example, your team fixes rare bugs that customers don’t often experience in an attempt to get your bug rate to zero; however, your delivery dates from your product roadmap begin to slip as a result. You are doing great work in this instance, but you’re not working on what is most important right now.

There are two flavors of this issue. First, you discover a problem and pivot way too hard in that direction. Second, you’re off the balance in where you’re focusing but making progress in all things. While you’re trying to meet all of your KPIs all of the time, it’s sometimes unrealistic, and you have to prioritize. It’s possible that a critical, priority one (P1) issue emerges and causes you to miss a deliverable. At that point, you’re making the call to prioritize your application quality over your roadmap delivery. In this example, your organization prioritizes functional software over releasing new features.

All KPIs are not created equal. Having a solid sense of the priority of your KPIs is essential to keep your team focused on what matters, help you make decisions about tradeoffs, and get the proper buy-in to take action accordingly. Over the medium term, revisit how much focus you place on each KPI to ensure you have a healthy culture and know when to increase resources (i.e., add staff). 

KPIs should prioritize your company’s overall goals instead of chasing either a mythical perfect score or just bouncing from fire to fire. Focusing on the proper outcomes can help you maintain balance, expose the real needs in your organization, and protect your employees from unnecessary stress.

Want to create and visualize the progress of your KPIs in one place? Let us show you how! 

Pitfall #3: Existing processes hinder KPI attainment

Of course, engineering does not operate in a vacuum. Your team’s efforts towards meeting KPIs fall into some form of process. If one team doesn’t understand how their work aligns with that of others, there is a high risk of duplicate work, or worse, work that doesn’t get done.

Here’s just one potential scenario: A defect escalation process is put in place, where customer issues get reported to engineering as bugs. Part of that process is to assign severity and priority. Customer support assumes engineering will handle it, and engineering expects customer support to take it. As a result, many bugs get stuck in some “waiting for more information” stage of the process, and customers are not happy. The company is likely to miss its customer satisfaction goals, yet your team still claims victory because your bug SLA/ bug rate is very low, as the bugs were not taken into account. Part of defining your KPIs and tracking them back to the value they deliver will help you spot deficiencies in the KPI flow.

By planning ahead and keeping engineering KPIs aligned with goals across teams, you’re able to set expectations, establish ownership, and track the correct KPIs that support processes working smoothly.

Pitfall #4: Important goals get lost

A corporate mission statement is a good starting point for identifying the outcomes your organization wants to achieve. However, mission statements are notoriously broad and challenging to track against. How your company responds to pressures, shifts in the market, and unforeseen crises should align with the values outlined in your mission statement. Typically, the goals embedded in those values are not well-tracked. 

Tracking KPIs that exhibit your core values can help set the tone of the organization. If one of your core values is, “We treat customers with respect and vow to make their lives easier through the use of our product,” you should identify KPIs to track how you are making their lives easier. Engineering can develop deeper testing plans to improve product quality and alleviate frustrations. Without the accountability and measurability that good KPIs provide, tying testing plans to customer satisfaction outcomes is nearly impossible. Even if you hit your goals, you wouldn’t have the data to support that you’re meeting the company’s mission. 

I suggest you have a simple, agreed statement that acts as a guiding light for your organization. It doesn’t have to be perfect; you just have to start somewhere. When tracking KPIs back to business outcomes, this statement can help fill in the gaps and find where the critical goals got lost.

Final Thoughts

Building well-thought-out KPIs is complex and unique to your business. It’s not once and done, but something that evolves over time. You will need to revisit your KPIs often, especially when shifts in your engineering structure stop aligning with the existing KPIs. I hope these suggestions can help you avoid some of the common pitfalls engineering teams experience when starting their KPI journey.

What KPIs do you have in place? How are you tracking against your desired outcomes? If you need a helping hand in improving your engineering operations, reach out to us to see how we can help.