Agile Tips the Balance Toward Success for Most Projects (but It Takes Time and Effort to Become Agile)

Image courtesy of The Standish Group International, Inc., 2015

In 1994 the Standish Group published its first CHAOS Report, which famously reported that only about 16% of software projects were completed on time and on budget. Over the years, this staggeringly low success rate has thankfully more than doubled (albeit with a modified definition of success). It’s important to know that Agile projects have carried the lion’s share of this improvement, which is why many organizations are making the move to Agile (but be warned that an Agile transformation takes time and effort).

The 2015 edition of the report examined success rates based on whether projects used an Agile versus Waterfall methodology, and the results showed an almost four-fold improvement in probability of success (39% compared to 11%) when the project used Agile instead of Waterfall. And this multifold improvement continues to be supported by other sources. So it’s no surprise that Info-Tech member organizations are increasingly adopting Agile, which has gone from being a rarity to becoming a necessity.

But becoming Agile is not as simple as just training your teams in Scrum, SAFe, DAD, or any number of other established methodologies. This is true because becoming Agile (first and foremost) is a change management undertaking, and organizational change is hard. It’s therefore not surprising to see many first-time adopters of Agile struggle to achieve its promised benefits. This is because the first tendency of any organization is often to “do Agile” rather than “be Agile,” meaning they focus on doggedly following prescribed process steps without actually focusing on the intended purpose of those steps. A common example is sprint retrospectives that become blame sessions because organizational culture has not shifted toward collective ownership of both success and failure by the entire delivery team (not surprisingly, CollabNet’s 13 State of Agile Report says that “Once again, the survey responses indicate that organization cultural issues remain the leading impediments to adopting and scaling agile”).

Our Take

Successful adoption of Agile is about organizational change, but many mistakenly see it as simply a process change. Following a process, like Scrum, XP, Kanban, or some hybrid, is an important first step in becoming Agile, but it doesn’t end there. Your organization will also need to change the way it thinks and operates to achieve the full potential of Agile. Some of the more difficult changes that need to take place include:

• Abandoning command-and-control in favor of self-directed-teams: Agile relies on the collective skills and motivation of the delivery team to make good choices for the organization no matter what twists and turns they encounter along the way. A culture that relies on top-down control and decision making will interfere with your ability to succeed with Agile. Your organization needs to properly empower delivery teams to make critical decisions quickly, which means that not only do you need to support and trust your teams, but you need to help them understand what good decisions look like.
• Becoming more willing to fail: Every organization suffers from “The Determinism Fallacy,” which is the mistaken belief that, by thinking long and hard enough about any problem you can determine the correct set of steps required to solve it. Unfortunately, software development often requires a trial and error approach to be successful, which can be extremely difficult for organizations to accept. Agile rejects the determinism fallacy and instead asks the organization to try, fail, and adjust in short bursts so that the impact of failure is minimized. Think of each sprint as a short experiment to see if something works. If it does, great! If it doesn’t, use the learning from that sprint as a course correction that will reduce the risk of future failures.
• Blaming the problem, not the person: Agile embraces the fact that things can and will go wrong. Fundamentally, each failure is an opportunity to learn and adjust behavior to reduce future failures. But this can only happen in an environment where failure does not lead to blame. Delivery teams must feel safe enough to admit what isn’t working and collaboratively take steps to eliminate the problem for the good of the organization. A culture that blames the person, instead of the problem, will fail at Agile.
• Adopting collective ownership of both success and failure: Traditional command and control organizations often make people responsible for “one small piece of the puzzle” (for example, making Business Analysts solely responsible for gathering and documenting requirements). Agile instead, requires each member of the delivery team to be collectively responsible for every aspect of the deliverable. They all either succeed or fail together, and this means doing work which might not be part of your traditional job description. This shift requires the organization to empower the delivery team, and equally, it requires the delivery team to step up to the challenge of collective ownership. An organization which hangs on to an “it’s not my job” culture will fail at Agile.
• Seeing change as good: Many organizations have been habitually trained to believe that mid-project changes (e.g. new or different requirements) are bad and need to be avoided. Agile embraces change as not only normal but good, especially when it is driven by learnings from earlier sprints. A delivery team needs to adapt its way of working to accommodate this need for change (e.g. don’t over invest in a design that may need to change once people start to see what the final product will look like). A delivery team that is resistant to changes will fail at Agile.
• Seeing baby steps as great progress: Agile advocates frequent demonstration of progress to stakeholders (e.g. sprint demos every few weeks), both to show progress and, more importantly, to gather feedback that is used for continual course corrections toward the final product. At the beginning of the project, the amount of progress that can be demonstrated will be naturally quite small and often will look very different from what the end product will eventually become. This can be a challenge to many organizations that are used to only seeing demos when a team has something substantial (and finely polished) to show. An organization who sees small incremental steps as disappointing, rather than comforting proof that progress is being made, will fail at Agile

These fundamental changes in organizational thinking and behavior will be difficult for your organization to accomplish, and so it will take patience, perseverance, and strategic thinking to help Agile take root in your organization over time. Here are some tips to help your organization reduce the risk of failing at Agile:

1. Avoid big bang: Moving to Agile is difficult (and risky) because of the amount of change required by the organization. Don’t make the mistake of doing a “big bang” introduction of Agile to your organization. Start small, expect to make mistakes, learn from those mistakes, and adjust your approach as you go. We often call it an Agile journey because it is just that: an organizational journey that starts slow and improves incrementally over time. And just like any journey, what it looks like will be different for every organization.
2. Pick the right pilot: Piloting Agile on a project is a great way to avoid a big-bang approach. Use the experience and learning from a pilot to increase your chances of successfully adopting Agile across the organization. However, it is important to pick your pilot carefully to avoid failures that can hinder broader adoption. Any Agile pilot will be risky simply because of the amount of organizational change required, so pick a low risk project. For example:
   1. Don’t pick an in-flight project and switch to Agile in the middle. This is a guaranteed recipe for failure.
   2. Pick a small project that will be relatively easy to accomplish in a short period of time (the delivery team needs to focus on getting good at Agile practices without being overwhelmed by a long and difficult project to boot).
   3. Choose a delivery team that is open and eager to try Agile. Ideally, you should seed this delivery team with people who have used Agile before (perhaps in a previous job) and are willing and able to share their learnings on the pilot with other teams.
   4. Pick your Product Owner and Scrum Master carefully. It is said that the most valuable contribution Scrum has made to Agile is creating the Product Owner and Scrum Master roles. Poor choices for either role will be detrimental to your pilot’s success. Both should be filled with willing, capable, and respected individuals who can dedicate the time needed to the pilot. Your Product Owner needs to be empowered to make decisions quickly and have easy access to subject matter experts who can validate decisions when needed. Your Scrum Master should have some technical expertise, but more importantly must be a good facilitator, mentor, coach, and effective leader.
3. Train your teams: Be sure to provide your delivery team with high-quality training on the Agile methodology they will be using. Don’t assume that “reading up a bit” on the process will be enough.
4. Understand that training is not enough: Training is an important first step, but the truth is that your delivery team will only get good at Agile by practicing it. And in the early days, they will need support, guidance, and patience to allow them to make mistakes and learn. Expect things to be a little rough at first as the delivery team works to put theory into practice.
5. Leave yourself a buffer: Don’t assume your delivery team will be working at peak efficiency from the get-go. Leave some buffer in your Agile pilot schedule to account for learning and mistakes your team will almost certainly make. Your first Agile project will NOT be your most efficiently run project.
6. Use an Agile coach: Bringing in an experienced and capable Agile Coach is a great way to increase your chances of success. Many of the problems, questions, and concerns that new adopters get stuck on can be easily overcome with the help of a good Agile coach. Pick someone who comes highly recommended by references and be sure to have one or more members of your organization work closely with the coach to learn their practices so that you maintain this skill set within the team once they leave.
7. Automate as much as possible: Adopting Agile is far easier and more likely to succeed when you incorporate use of automated tools to carry some of the load. For example:
   1. Use one of the many available software development management tools that support Agile processes (like Jira, VersionOne, or Helix) to make your delivery team’s job easier and make it possible to automatically gather highly valuable metrics (like sprint velocity) without additional effort by the team.
   2. Use software test automation tools (like Parasoft) to make regression testing less labor intensive.
   3. Use team collaboration tools (like MS Teams or Slack) to improve team communication, particularly for remote team members.

Take advantage of Info-Tech software reviews to see how individual tools rate based on unbiased customer feedback.

8. Get organizational buy in for the changes needed to support Agile: It is critical that your organization’s leadership both understand and accept the changes required to successfully adopt Agile. Many leaders are unaware of the true challenges associated with the shift to Agile and the role they must play to help change culture, thinking, and behavior. Similarly, if they are cannot accept that Agile is something that requires significant time and effort to adopt, your organization is less likely to be successful.
9. Use Info-Tech’s expertise and offerings: Agile adoption is a complex journey with plenty of obstacles that can get in your way. Info-Tech offers blueprints, expertise via analyst calls, and workshops that can help make this journey easier and more likely to succeed. Consider using Implement Agile Practices That Work or the soon-to-be published Agile Transformation Accelerator to help you with your journey.