This is a question that matters greatly for any organization that is dogmatic and decides it must be 100% Agile to operate efficiently and effectively. But I must admit, this question is a red herring. Here is the question that should be asked:
What should a well-functioning team bear in mind when deciding how much to plan and estimate in advance?
First, it’s worth noting that this varies not only by the organization, but also by the project. So any organization that adopts a single methodology for all of its work and expects every project to run equally well within it is in for a surprise.
While there are a lot of organization-level factors that should drive development methodology, I’d like to focus on one in particular because of the outsized influence I believe it has on the success or failure of development teams.
Organizations that cater to consumers are not usually bound by external commitments. In addition, their customers are their buyers, which makes the process of learning and iterating very quick. On the other end of the spectrum, organizations that cater to businesses typically have a myriad of external commitments they have to make - some big enough to make or break an entire quarter or year. And because their buyers are senior-level folks often multiple levels removed from the actual users of the software, it is not always quick or easy to discern if rapid changes being made are a net positive for both users and purchasers.
An entire book can be written on the difference between enterprise vs. consumer software organizations, but when it comes to development methodology, understand this - for organizations selling enterprise software, there tend to be far more projects that require somewhat firm scope with hard deadlines.
Which brings me to the individual project. Unless every project a team works on is similar in nature, then the development process used to accomplish any given project should vary. What are some of the most important project-level factors?
The level at which the work is being done
Whether you are developing against a monolith or microservices architecture, you inevitably have some code that is very “core” to the rest of the code and some which is more “ancillary” in nature. You intuitively know the difference. Hopefully you’ve also documented it. This should drive your process as well. If you are making changes to your architecture or your backend platform, you should in all likelihood be applying more care, planning, and estimation to the process than if you are making some minor UI tweaks to a React component. It’s not to say that both aren’t important in their own way. But the complexity of the work and the consequences of getting something wrong tend to amplify exponentially the closer you get to your product’s “core”.
The likelihood of changes to requirements
Are you shipping a beta or early feature meant to be learned from and iterated on over time? Do all relevant stakeholders want to just prioritize getting something - anything - out over having a reasonable level of certainty over scope or timeline? Great. Plan and estimate as little as possible.
Are you shipping a feature because a commitment was made to a client for a specific set of requirements by a specific date? Also fine. Plan and estimate liberally.
The amount of coordination required with other business units
Building on the previous factor, work that is being done for the purpose of beta testing will generally require less coordination with other departments and stakeholders than work that is being done imminently for specific campaigns or contracts. For example, if you have agreed to integrate your product with a partner’s in a very specific way, invest mutually in promotional materials around the integration, and then make a joint announcement about all of it at a scheduled tradeshow, you have a lot of people and dollars relying on the fact that you do a very specific set of work by a very specific date. Plan and estimate liberally.
It comes down to cost/benefit
As you can see, the amount of and the approach for planning and estimation should be relative to the costs vs. benefits of those activities. This will vary by organization and also by the work being done at any given time. Always ask yourself and your team: Is the time and grief it will take to think through this project upfront significantly greater or less than the time or grief it will cost our organization if we miss a requirement, assumption, or deadline? All of the dogma in the world around development methodology should not get in the way of practicing that basic litmus test at the outset of every project.
But can you ever really plan or estimate your way out of failure?
No! Failure should be part of your culture, regardless of the nature of your organization or projects. If you never fail, you’re probably not moving quickly enough. But undoubtedly, the more planning and estimation you do, the less you will fail - especially if you ensure the purpose of doing it is to identify project complexity and uncertainty. In this way, it is all about appetite for risk. You need to know when risk surrounding a project is high and when it is low. Place your bets properly. Plan and estimate accordingly.