Prashad is the project manager for a new product. After reviewing the project scope, Prashad’s team gives it an initial estimate of about three to four months to complete, which is good enough for management. The project kicks off, and using Scrum along with a little bit of project management, the project gets underway. The first month goes quickly, there is great progress, and everyone loves what they see. But in the second month, things start to slow down — so, Prashad adds more resources to the project to compensate. But by the third month, there are real problems and it looks like the project will be late. Prashad adds even more resources. Now the team is almost double the original size. He adds a lot more quality assurance and testing support for the team, too, because a lot of problems are cropping up. The team cranks away, and they finish the project in the fifth month, and claim “success” because they almost hit the deadline (it was, after all, just an “estimate“).
But what about all those additional resources? Is it really success if the project cost twice as much as originally estimated? The problem is, that cost was never formally stated. It was never tracked as a metric, so the fact that the project went horribly over budget is quietly swept under the rug — at least this time. There are plenty of cases where the budget isn’t there, or the resources aren’t there, or the product just can’t ship late.
So, if so many projects are missing their mark, what’s causing the problem?
The fallacy of so called Agile methods
According to studies conducted by KPMG, as much as 70% of projects fail to meet their goals. In this case, “goals” mean quality, schedule, and function (or a combination of those). Clearly, being able to accurately estimate a project’s delivery date is important. Without knowing when a project is done, you can’t predict cost, plan business operations, or dovetail releases with marketing, training, and customer service.
Agile methods like Extreme Programming and Scrum make promises that, in my opinion, they can’t keep. For well over a decade we’ve seen an increasing trend in project failures, and a large number of those failures are the result of “unending projects.” Projects that go on and on, continuing to deliver improvement but slowly creeping over budget and never reaching an end state. Part of the problem is the lack of formal estimation and metrics.
Both Scrum and Extreme Programming dodge the entire issue of project life cycle estimation. They focus on the near term, providing estimates for the next one or two sprints. This works great if you don’t have budget or time constraints, but in the real world that’s rarely the case.
Burndown just doesn’t work
Most Agile methods don’t explicitly define how you estimate progress, but the most common method is burndown — the measurement of completed effort against the planned sprint goals. This is fine for a quick-and-dirty project but as a strategy it utterly fails to pinpoint problems with budget and timeline. For anything larger than a few months effort, it doesn’t do the job.
And the problem is, few of today’s engineers seem to be getting the formal education and training needed to use proper estimation methods.
There are better solutions than burndown
Earned Value Management, or EVM, is nearing it’s 40th year of practice. Throughout this 40 year history, EVM has repeatedly evolved to meet new demands as technology and innovation made new leaps forward. It’s a proven method for measuring progress and it’s been proven time and again.
Of the dozen or so software engineers I spoke with about this article, none of them had even heard of EVM.
Earned Value Management, as well as a variation known as Performance Based Earned Value® (PBEV), are my favorite choices for estimating a project. The PBEV approach provides an incredibly robust method for measuring progress and staying on top of your project. It’s only marginally more difficult than other techniques of measurement. And most project managers are already capturing the metrics needed to make it work.
PBEV is an improvement over Earned Value Management Systems’ (EVMS) national standard. It supplements EVM with guidelines for the integration of project cost, schedule, and technical performance — thus created a comprehensive and highly accurate method for measuring true progress in a project. The key here is integration of effort with schedule and performance.
Had Prashad been using PBEV, he would have had a very different experience. The early estimates would have been more accurate, since PBEV estimates the entire project. It would have measured not only resources and cost, but also the actual performance — the rate at which work is being done. Had the project gone off track, Prashad would have known much more quickly, and would have had a more precise picture of how to correct the project’s course.
There will definitely be situations in which PBEV is not warranted, and a skilled project manager will need to know when to use other methods, and why. EVMS, for instance, addresses only the quantity of work completed, ignoring cost and schedule metrics. The implication here is that a quick burndown or EVMS-based evaluation of a project might look A-OK, when in fact the only reason it’s on track is because resources are being thrown at it — and costs are skyrocketing.
Burndown versus critical path thinking
Burndown is a very simple method for estimating work effort over a short term. It explicitly avoids timeline, cost and performance metrics. That’s fine for projects that are not constrained by time, budget, or resources. The problem is, burndown is used as a defacto standard too often, on the promise that using an Agile method will deliver a project faster or cheaper than its alternatives. I don’t disagree on this front. Agile methods generally are faster and cheaper — but the question is, faster and cheaper than what?
Most business need to be concerned with time, money, and staffing. Projects that have those constraints need to think about the critical path — the sum of all activities to reach the end of the project. Having a solid understanding of the fundamentals of critical path management is important to managing any project schedule. Depending on the nature of your project, you may be able to rely on casual effort estimates — or, you may be required to carefully identify, analyze, and attack critical paths to shorten your project lifecycle and stay within stringent guidelines. Without understanding the principles, and knowing where to turn for more information, neither is possible.
I’ll revisit the topic of PBEV in a future article and provide more depth into how it can be applied to a project. If you’re eager for more information, check out Performance-Based Earned Value®, Paul J. Solomon, Ralph R. Young, IEEE Computer Society, Wiley Interscience, 2007.