Software cost estimation: Where’s the silver bullet?

Recently Kirk Gray wrote a piece — more of a plea really — titled Software Estimation is Hard. The problem at hand is that there doesn’t seem to be a silver bullet that delivers accurate software project cost estimation. Software cost estimation (and here, I mean “cost” in the sense of effort, time and money) is hard, as Kirk points out:

“The problem is that, especially in a small shop, we:

a) Don’t have comparable projects that can be used to provide historical data, especially for partner projects that are often more one-off than core product development. I can say with some confidence how long it takes to add a new Struts 2 action and mapping – that part is easy, but what about the elements on the page? What about the logic?

b) Don’t have the infrastructure in place to really record time spent on any one task, and often find that if we did, it would be inaccurate because we wear so many hats, each hat being taken on and off randomly throughout the day/week.

c) Can’t afford just yet to spend a lot of time making exhaustive estimates, because that’s time that won’t be spent in construction and testing.”

This is probably the most common problem facing software teams today and, quite simply, the root cause of most project failures. Even projects that are regarded as ultimately successful suffer from this issue, as evidenced by frequent project overruns, software delays and adjusted delivery goals. And yet, we have the tools at hand to solve the problem: Function Point AnalysisProxy-Based EstimatingEvidence-Based Scheduling, and similar tools offer the basic software estimation processes needed to accurately predict the overall cost of a project.

The problem is not with lack of process: It’s with lack of proper adoption. As Kirk points out, in his own situation a host of pressures prevent dedicating the necessary effort, time and resources to do the job right. Put bluntly, relatively unseasoned teams often lack the experience and wherewithal to set up the right processes and make the right investments. These “investments” typically translate into very real costs to the business, longer development schedules being among the most obvious. The pressure applied by a business to set a delivery date now, before enough data has been gathered, can be daunting. It often takes a combination of two things to successfully pull off software cost estimation: First, a team that has the ability to make accurate statements regarding a software project, and second, a business that is willing to listen to that team.

An experienced team

During my years performing project auditing I’ve noticed a trend throughout Silicon Valley, particularly among startup companies. This trend involves creating teams from remarkably bright individuals that have very little formal training or experience in software development. Other sectors, including large scale enterprise and the military, take a different approach: Assembling teams that have the senior leadership necessary to create an effective team. This means a formal understanding of development process as well as experience — because, quite simply, technical knowledge without experience generally results in missteps. Those experienced, senior dinosaurs are actually quite valuable in providing guidance and mentoring to the development team as a whole. But, again being quite blunt, these senior experts (SME’s or Subject Matter Experts) support their teams in a very effective way by providing the voice of experience and authority to the business. This is the bit about “standing up to the business,” and is critical when the business tries to step in and force decisions that just shouldn’t be made yet.

A listening business

Perhaps equally important as building a team with experienced, authoritative leadership is having a business that recognizes the value — and expertise — of its team. That has to extend to a willingness to listen to the team, and take the team’s recommendations at face value. I’ve personally walked away from more jobs than I’ve taken where it was apparent that the business believed in a purely “top down” management style, not giving the team the resources or time it needed to make the right decisions to deliver a quality software product.

Let’s make this concrete: What does “listening to the team” mean? More than anything it means listening to the needs of the team and addressing them. Kirk points out that they have a lack of expertise in software cost estimation: Hire a consultant to work with the team for three to six months to address this need. He points out a lack of infrastructure: Buy the right tools and give the team time to put them in place (they aren’t that expensive). Most appalling to me, he points out a lack of time to invest in estimation because that means the team isn’t coding: Give them the time to do their job right. The long term benefits will be realized ten-fold through accurate software estimation, greater reproducibility and improved software quality (you know how the software is always late or ships with too many bugs in it? This is the root cause).

Strategies for success

For projects that don’t have a body of work product to use for historical data, look outside the project. Unless the team consists of entirely junior programmers with no experience under their belt this won’t be hard to do (the formal nomenclature for this is wideband delphi, an easy-to-use and surprisingly accurate method for discovering historical data where none seemed to exist). The basic process is simple: A moderator and an estimation team with three to seven members is selected for two meetings run by the moderator. In the first meeting the estimation team creates a work breakdown structure (WBS) and discusses assumptions. After the meeting, each team member creates an effort estimate for each task. The second meeting is the estimation session, in which the team revises the estimates as a group and achieves consensus. I’ve had excellent success using this process as a basis for arriving at reasonably reliable estimates, but it’s important to add one more step: Make sure that the programmer that takes up the given task agrees with the estimate, and understands the task in full. Sometimes the group estimate will be more aggressive than an inexperienced programmer is willing to take on; likewise, sometimes an experienced programmer will know it can be done more quickly. Be conservative, and defer to the individual programmer when appropriate.

In regard to making time for the underlying effort in software cost estimation, one of the most common excuses I hear is “we don’t have the tools to record time, our team wouldn’t agree to it, and if we did, it would be inaccurate because we wear so many hats.” Frankly, this is just procrastination. First of all, read Joel Spolsky’s article on Evidence Based Scheduling. It’s an excellent introduction that clearly explains why wearing a lot of different hats, being distracted by unplanned tasks, and having a chaotic environment just doesn’t matter. A good system for measuring velocity is going to understand and accommodate those realities — in fact, the entire purpose for estimating velocity is because of all these unexpected, unplanned interruptions. If we had none, our estimates would be perfect, wouldn’t they? So, where does this excuse lead you but down the same, inevitable path: Lots of misunderstandings with the business, failed deliveries, slipped ship dates, poor software quality and an awful lot of late nights trying to get incomplete software out the door. Stop the addiction: Do the right thing and spend $200 on FogBugz or, if you want to go industrial choose my favorite, Atlassian JIRA (it’ll cost you a couple thousand dollars, but it’s probably the best investment your department will make, period). If you’re looking into JIRA, check out the Greenhopper plugin.

And finally, if you feel you just can’t take the time to estimate and manage your project, keep in mind this is the same as saying “we just can’t take the time to do it right — so we’ll hope for the best.” This is a classical problem for many Agile organizations that focus entirely on the short-term while sacrificing long-term strategy. It’s a situation that leads to a perpetual, self-feeding problem, and it gets worse not better. It’s time to break the pattern and make the decision to fix it.

Probably the most important piece of advice I offer team managers is simply this: “I don’t know” is OK. The worst thing you can do is commit too early. When the business challenges you for a timeline and presses for a hard and fast ship date, the best thing you can say is “I don’t know.” The natural follow on question is going to be “fine, what do you need to find the answer?” Stand firm and insist that you need the time, tools and resources to accurately estimate the job. Cite past performance — or past lack of performance, failed deliveries and customer service problems, if need be. Ask the business “do you want more of the same, or do you want to improve?” Ultimately, it’s up to you as the team lead, Scrum Master or project manager to make this happen.

4 thoughts on “Software cost estimation: Where’s the silver bullet?

  1. I do agree with most of what you say. I think that there *absolutely are* a lot of things that we could do. I think that in addition to what I mentioned as issues in my post, a big issue is this:

    Some web startups are software companies, and some web startups are ‘businesses’.

    Now, the distinction shouldn’t necessarily exist – all startups are (obviously) businesses, and generally web startups are software companies (technology drives most of their value proposition). The thing is, if I requested a budget line item to bring in a consultant or to implement JIRA (wouldn’t that be nice, though we do use Unfuddle which is better than nothing), I would possibly be laughed out of the room because that money would be better spent on direct marketing or traffic sharing or business development.

    I guess it depends who the leader is and what their background is, to decide whether buy-in at that level is feasible or not. Nobody is wrong, just a perspective thing. I will take a look at some of the suggestions, and some of the many posts on your blog and maybe try to take some baby steps =)

    Like

  2. It’s unfortunate that you’re in a situation where the people relying on you would laugh you out of the room. I’ve been there a few times — and as I mentioned above, quite frankly I’d be looking for a new employer that places a higher value on your expertise. However, that said, I’ll add that the best weapon you can use against this is to present your boss with choices: Tell him outright, “we need a consultant to help us improve our process; without one, there’s no point in setting a release date because we simply don’t have the ability to make it reliable.” Redirect the decision to the business, with clear emphasis on the consequences. That way, the business is the one that must decide, consciously, to go down a sub-optimal path. Most often, it at least opens up a dialogue about other options.

    Like

  3. Fortunately I am not in that situation now – but I have been. It’s harder when economics are as they are to convince people to make investments in less-tangible endeavours, but I understand what you are saying.

    Like

  4. Software estimation is hard and there is no silver bullet. I build software estimation models and quite frankly, if I tried to sell you a silver bullet you would laugh me out of the room exactly because you know software estimation is hard. The reasons it is hard are many, some of which you have mentioned above and some of which I have detailed in my article “Agile Developement and Software Estimation” (found at http://blog.pricesystems.com/blog/arlene-on-accurate-estimating).
    But as you note, software estimation is crucial to project success. And as the provider of tools I would certainly advocate the inclusion of a good tool into the software estimation process. But don’t expect the tool to be a silver bullet – rather it isa way to enforce a methodology not only for estimation but for proceeding forward with successful data collection and future project estimation success. It doesn’t matter how good the tool is – without good data it will not be ‘right’ out of the box But it will, out of the box, help create that infrastructure necessary for data collection by intimating what things are important to collect and what buckets are best for collection of effort and schedule information. Additionally, as noted in the article of my colleague, “The Silver Bullet – Practice Makes Perfect” (found at http://blog.pricesystems.com/blog/price-systems) the only way to move from immature estimating practices to more mature estimating practices is through practice.
    Certainly I am an advocate for the value of tools in this process – they help speed the adoption of good estimating practices and ultimately speed up the estimation process itslef. But even in a small start up – where tools may not be deemed a worthwhile investment – establishing a rudimentary process that feeds itself through data collection and comparison of estimates to actuals – will – in relatively short term – pay for itself in greater confidence in the outcomes. It will be easier to identify ‘comparable’ projects, data collection will be facilitated with knowledge of what data to collect , and confidence in the value of estimation will be engendered up and down the chain.

    Like

Comments are closed.