How did we get here? The Complex Journey to Value Stream Intelligence

More often than not, these resulted in mostly well-intentioned consultants losing sight of the intent of our task - building high-quality software more efficiently (read: faster) and more enjoyably while delivering additional customer value.

A few distinct phases stick out. I’ll describe the logical progression I see in the story because while the early phases of agile and DevOps trend toward delivering customer value, I think there’s a lot left to be desired. It’s also why I am so excited about what we’re doing here at Allstacks with Value Stream Intelligence (VSI).

Here’s the short version of the story arc as it has played out:

• Phase 1: Agile: Wow, building software is actually quite hard, and it’s not OK to take years to ship customer value… we have to find a better way.”
  
  (Methodologies, Principles, Platforms, & Consultants Emerge)
• Phase 2: DevOps: "These agile concepts work! Oh, shoot, we don’t control our own destiny… how do we work with the Ops/IT teams to deploy to customers quickly and reliably?”
  
  (Tools, Pipelines, Automations, Orchestration Layers, & Consultants Flood the Market)
  
  
• Phase 3: Value Stream Management (VSM): “All these processes… so many tools and teams and stakeholders…how do I gain visibility into all of these moving pieces?"
  
  (More consultants, fancy charts, layers of process and long change management engagements) 
  
  
• Phase 4: Value Stream Intelligence (VSI): “How do I take advantage of these data streams to actually shape my business outcomes?”
  
  (Introducing: Allstacks)

Follow me through a bit more of the history of this and you’ll understand why Value Stream Intelligence is about to earn significant mindshare in the boardroom.

The Digitalization (and complication) of all the things 

Everything became digital, and now “every company is a software company.” As organizations grow, so does complexity -- more stakeholders for you to collaborate with, more people you need to hire, and more new technology you must implement. 

Tools explosion #1:

In the past, companies became more reliant on technology teams to create products and value. Still, each advancement or new category that promised to get software teams closer to said value only got us further away, which led to uncertainty around software delivery and lost alignment with the rest of the business. 

Insert process explosion:

Agile (especially at scale) creates more work 

By the early 2000s, agile had already won the hearts of many software teams. Between 2009-2011, agile transformations really took off in mid-size to large enterprise companies. The message was, “It’s a new world. The cloud is coming.” The sense was that the way work gets done is infinitely more complex and ripe for implementing processes to better orchestrate the work.

We’ve learned that agile takes a lot of resources, primarily an investment in time to estimate work, participate in ceremonies like stand-ups and retrospectives, etc. Which naturally led us to ask ourselves, “What tools can we use to automate and better traffic all this work?” 

Insert tools explosion #2:

DevOps scales work, but it doesn’t connect business outcomes

In 2013, DevOps was introduced as a logical extension (some say enhancement, some say competitor) of Agile as we realized it was difficult to deploy at scale. There is no debating the value DevOps has brought to software development by enabling (and sometimes automating) deployments at scale and pioneering entirely new concepts to orchestrate delivery. A simple search of “DevOps Director” job postings will give you a quick sense of this method and culture’s foothold in the enterprise.

Between agile and DevOps, teams are dramatically better at trafficking the work items through the process and out via the pipelines. The missing piece is the ability to translate all this work into customer value across the organization, and perhaps most crucially, to know when something is going wrong ahead of time.

Value Stream Management looks in the rear-view mirror

VSM involves people in the organization examining workflows and other processes to ensure they derive the maximum value from their efforts while eliminating waste — resources, time, and assets. 

The problem with VSM? It’s essentially a flavor of DevOps with backward visualization (BI.) It takes a long time to develop and implement since it requires things to be unnaturally strung together. When you embark on a VSM journey, buckle up and prepare for the herculean and expensive effort of change management to unwind the years of poorly integrated (or perhaps not integrated at all) tools, processes, and teams. 

Can you imagine spending all that time and money on implementing VSM only to still have these outcomes:

• 51 percent of features delivered late
• 47 percent of projects go over budget
• 38 percent of projects failed to meet goals
• $100M lost every year due to failed software delivery 

Teams use agile, DevOps, VSM, and the best technology in the hopes to eliminate these surprises and misalignment in the business -- yet it keeps happening. 

We really need the concept of looking at workstreams in terms of how they drive customer value that VSM promises but in a way that proactively de-risks your outcomes rather than merely responding to your risks as they emerge. We also need to get smarter about aggregating all the data available to us and analyzing it in a meaningful way without the overhead that comes with VSM. 

Value Stream intelligence builds trust, improves reliability, and delivers value 

When things are more predictable, outcomes flourish. Rather than looking into the rearview mirror, software companies need to proactively de-risk their outcomes. 

VSM requires a thorough and complete implementation to be effective, which means it takes a long time to see the value. The intelligence layer provided by Value Stream Intelligence (VSI) expects data to be imperfect and can fill in the gaps between systems without the implementation overhead. With automatic, intelligent analysis of a team’s historical data, software organizations can more predictably ship software that aligns customer value with forecasting and risk alerts to proactively respond to changing delivery efforts. 

With VSI, you aren’t adding in a new methodology or tool in your stack so there is no additional investment in process change other than turning to real-time dashboards to understand:

• What work is happening?
• When will work be completed?
• Where are the bottlenecks that we need to unblock to reach our target delivery dates? 
• If we are getting better over time?

By tapping into the historical data that already exists in your SDLC and using more intelligent technology to make sense of that data, you can connect engineering output with business outcomes and customer value easier than ever before. 

The functionality you can expect from a VSI Platform:

• No process change or manipulation of data to receive value.
• Proactive risk alerts that deliver pivotal information to the team at the most impactful moments.
• Contextual metrics across the SDLC that improve the team and individual productivity.
• Intelligent forecasting insights from across your SDLC to surface roadblocks and estimates software delivery dates. 
• Shareable dashboards that align all levels of management and roles. 

With VSI, your org can march forward into a new paradigm of delivering value to the customer. One where software is a change agent for the business, enabling your business to be driving towards results, not looking back, wondering what transpired to land you where you are today. 

If you are interested in learning more about what a VSI platform or Allstacks can do you for, here are some additional resources: 

• Value Stream Intelligence (by David Rubenstein at SD Times) 
• Forecasting software delivery. Engineering's new best friend
• Allstacks Interactive Product Tour 
• Allstacks SDLC Integrations 
• Schedule a live demo of Allstacks