How to Measure Throughput and Win the Business’ Trust

Every application development leader I have spoken to at Info-Tech is asked to deliver more, faster by their business partners. Our research also shows that the single most important factor for business satisfaction with IT is your project delivery and capacity. Specifically, your ability to rapidly turn around development requests and deliver projects on time and on budget is more important than any other factor (The Moneyball CIO).

When I led IT development teams in financial services our mission was simple:

We Deliver

The devil was in the execution. We committed to deliver useful products on time, on budget, and it worked. Our CEO told us he needed us to do more than just deliver. He needed IT to be a partner that helped expand the business. In Info-Tech’s business satisfaction scale, we were invited to a seat on the execution of key business transformations. Our award-winning digital transformation of our products and web presence followed.

It took a collaborative, disciplined approach to estimation, how we managed our resources, and the portfolio of work we had committed to deliver. It was a massive effort. Resource management alone took over 6,500 hours a year. We more than doubled the benchmark for project management effort. We were effective, but not efficient. We suffered from a problem shared by the many of the IT organizations I have advised since: we measured work from our own point of view, not a business perspective.

IT Measures Activities, Not Business Outcomes

Good project management practices from the PMBOK “create a unique product...to fulfil objectives.” Structured IT delivery methods such as waterfall quickly lose sight of the objective. The priority is an estimate of the time required by role to create the product, which backs into a delivery date. In project execution, the focus shifts to role-based activities and deliverables such as a charters, business requirements documents, designs, and so forth. Even quantitative measures adopted to manage efficiency still focus on role-based activity and deliverable completion. For example:

These artifacts help IT organizations improve their effectiveness. Even a small project has so many artifacts that the objective is lost like the forest among all the trees. Nor do they predict the answer to the most important question on the business’ mind…When will you be done? (Ambler)

IT Benchmarks Do Not Help…or Matter

Time-based estimates have led consultants to benchmark the proportion of time by practice to peak performance and outcomes:

Source: Project Management: A Systems Approach to Planning, Scheduling, and Controlling, QSM, Seilevel

The benchmarks are useful tool for selling practice improvement services. They also reinforce IT’s role fixation. You need to do more than spend the right amount of time on the right things. You need to do them right as well. Nor does a role dictate who does the work. Even if you avoid these pitfalls, a perfect activity-based time distribution does not tell the business when you will deliver.

The Agile Revolution

Alistair Cockburn introduced the concept of a User Story as a “promise for a conversation.” That conversation became a revolution when it focused IT’s grab bag of the practice-level artifacts, time benchmarks, and their related metrics into a single package.

"As a < type of user >, I want < some goal > so that < some reason >."

Combined with Story Points, they allow you to measure the velocity of a team or how much work they can deliver in a fixed time box. Ron Jeffries, their inventor, has said Story Points are, “thinly obfuscated TIME.” It is, however, a very useful obfuscation. Velocity is one of the few brilliant Agile concepts Bertrand Meyer identified in his critical analysis, “Agile: The Good, the Hype, and the Ugly,” because we now have a “directly useful technique” to “provide visible, constantly updated evidence of progress.” It is the missing link required to measure and improve the productivity of teams.

Where Agile Falls Short

The Agile Manifesto declares, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Unfortunately, in practice this seems to be lip service only. While 80% of organizations believe Agile and DevOps practices should drive business value, “that value goes unmeasured and unrecognized.” Agile Velocity and Burndown metrics are stuck in the classic IT activity-focused model, namely writing working code. Some practices go as far as identifying the value to an end user in the User Story. Few, if any, make the connection to the business outcomes that drove the case for change. With the chain broken, as Goldratt’s Law says, “If you produce something, but don’t sell it, it’s not throughput.” However, if you ask most Agilistas when you can sell or use a product their answer is likely to be, “It doesn’t work that way in Scrum.”

The Non-Linear Problem Is Real, Now Get Over It

It is easy to blame this behavior on Agile idealism disconnected from the pragmatic reality of business. However, avoiding commitment is a classic IT syndrome based on a real problem. Unlike manufacturers who create identical widgets, each piece of new IT work is a black box with a non-linear size and complexity compared to the business value. There is also a deep discomfort among IT people with uncertainty. Developers are particularly susceptible and it leads them to overblow how complex a work item might be. However, manufacturers also do custom work and the ability to measure throughput is just as relevant. Ted Hodson is a Senior Mechanical Designer at Siemens who manages the delivery of custom manufacturing orders. Despite the inherent uncertainty he says, “It is really important for the customer to get an accurate delivery date as soon as possible.”

Estimation: The Black Sheep of IT

Breaking down the work into discrete chunks represented by artifacts such as a User Story is essential for good quality software development. The practice and the non-linear relationship to business value also disconnects the subsequent User Stories. Agile solved this with a fudge and a wink. Treat velocity (or productivity) as throughput, don’t commit to an estimate for a whole solution, and leave the Business to worry about the rest. Sadly, the wink and the fudge contributes to the classic disconnect that has led to IT being called, “The Strangers on the Train”. The Business does not care how many Story Points you completed last spring. As Scott Ambler of Disciplined Agile has found, the Business cares most about when you will deliver.

Enter the Minimum Viable Product

Once you decide that when you deliver matters, the most important question becomes what you will deliver. Frank Robinson introduced the concept of the Minimum Viable Product a year before the Agile process was born in a chalet on the slopes of Snowbird. While warts have shown up in execution, the fundamental concept is the foundation for a useful measure of throughput: an estimated set of features/user stories that make up a product you can use or sell. Agile practices have begun to embrace the need to estimate size and delivery. For example, in the Scaled Agile Framework (SAFe) you create estimates of the whole package of work in Iteration Planning.

Recommendation: First Measure It, Then Manage It

With a velocity and an estimated product, you have a manageable metric of throughput:

Throughput is just the first step. You must manage future changes to the solution or workload of the team as well. A jumbled bag of user stories is not enough. Good practices for software engineering and Agile require you to manage the design of the product. As Jeff Hemmet, the Senior Director or R&D Systems Innovation at Elekta Medical Systems says, “A User Story is not a requirement.” You must use the work you do to estimate and design the solution to improve throughput as well. You must ensure every aspect of the solution is traceable to:

• Use business requirements to ensure you realize the benefits you expected.
• Connect individual stakeholder needs to the solution to balance competing priorities.
• Identify and manage the impact of changes.
• Audit-proof your IT delivery process.

Bottom Line

Stephen Covey wrote, “One of the fastest ways to restore trust is to make and keep commitments.” Software development leaders are challenged to make commitments every day. When they avoid commitment, the Business sees them as slow and out of touch. When they commit and fail to deliver, the Business views them as ineffectual. Develop the discipline to measure and manage throughput and you can make realistic commitments to the Business and keep them. This will build a trusting partnership based on repeatable, measureable results.

Want to Know More?

The Moneyball CIO – Please contact Info-Tech for a personalized presentation

Create a Horizontally Optimized SDLC to Better Meet Business Demands

Enable Organization-Wide Collaboration by Scaling Agile

Structure Your DevOps Adoption Using a Metrics-Driven Approach