Why are projects so frequently late/over budget?

I am thinking of projects of every type, from civil projects (The Big Dig) to software (take your pick) to buildings. Performance, at least in my unscientific view, is skewed towards late/over budget. If a software project is within 10% of target it is generally considered a success, and it is a rare project that beats all expectations.

So why is this? I don’t know if there’s a factual answer. Maybe some business PhD has done a thesis on it. But I wonder about these factors:

  1. Human nature includes a desire to give good news to gain approval. People want to give optimistic estimates because it gets them a pat on the back. Therefore they subconsciously ignore things that could make the news bad.

  2. Human nature includes a desire to deceive people with false good news to gain approval. Therefore they consciously hide things that could make the news bad. The cynic in me has seen system development projects that we call “priced to win.”

  3. Human nature includes a complete inability to estimate accurately, regardless of intent. People just suck at estimating.

  4. Project estimation involves so much complexity that it is beyond the range of human ability. There are just too many variables.

  5. Human nature includes a complete inability to manage a project. I think this is actually the least likely explanation. Managing a project isn’t that hard. What’s hard is anticipating everything during the estimating stage that could possibly cost you time and money.

WAG: initial estimates on projects are rarely able to take into account all of the complications which arise during the project. If a particular task is more challenging to overcome than originally expected, or requires a significant (and unexpected) reworking of the project as a whole, there goes your estimate.

For a project of any complexity in order to foresee every complication that arises, you have to basically do the project. And if you cannot foresee them and just budget “+30%” or something, then it’s pretty definite you will be overbidding.

A friend of mine became the manager of a project in this manner:

  1. Company president asks all key personnel how long the project will take.
  2. All but my friend answer 9 months to a year
  3. My friend answered 6 to 9 months

Two years later the project was done.

There’s probably a thousand answers to this, but I’ll boil it down to three:

(1) Feature Creep. The project initiator doesn’t know exactly what he wants, and continually adds in new features. Or they don’t like an implementation and constantly change it.

(2) Poor estimating/Lack of ability. Simple enough. The winning bid underestimated the time it would take, or overestimated their abilities. As the saying goes Shit Happens, and experienced people tend to account for this in their bid, and avoid the worst shit.

(3) Unexpected difficulty in developing new technology. If you have to use a new technique, method, or product, these all might take longer than expected, or simply not work.

(4) Acts of god. People die, natural disasters happen, etc.

I’ve now moved into the Infrastructure & Planning area of my company and I’ve learned a few things. Firstly, most projects seem to come in either on budget or slightly under budget, at least at my organization - I was shocked, to say the least.

As far as delays - to give you an example 5 weeks ago I sent in an order to have an electrical project scoped and a quote generated for a fairly simple project. To date I have nothing because, oops, we confused this project with another project, then we were busy so we had to farm it out to contractors then they were busy and didn’t get here very fast, then someone unrelated, but overly involved in the project intercepted the contractor and started talking to him about a bunch of other crap, then I’m a woman so the contractor wouldn’t talk to me about the project and instead talked to the other interested, but not involved guy, and then our head of electrical was out sick for dental surgery.

TLDR version - typical SNAFU - 5 weeks later and I still have no idea what the cost of this project is going to be, when they’re going to start it and how long its going to take. Suffice it to say, it’s going to go long. :rolleyes:

This is what management is supposed to do, and what very few managers do well. The real job of management isn’t protecting their territory and schmoozing in the office, but that’s what an awful lot of them do instead of doing their damn jobs.

Engineers, as a breed, are optimistic and occasionally arrogant. They nearly always overestimate their ability to produce. Many of the good ones also have a streak of perfectionism and passion for an elegant solution that leads them to spend a lot of time finding a perfect way of doing something, instead of doing something that works. There needs to be a balance between an ugly hack that can’t be maintained and throwing away time on redoing the project over and over because you find that part of the foundation is flawed or you find another way that will probably work better in the long run, but will take much longer to implement.

And then there’s the distraction of shiny new features. Duke Nuke ‘Em’s nearly fatal path was the, “but we can do it better now, so let’s throw out most of the stuff we did before and add feature X, and graphics engine Y, with new procedural path Z” treadmill.

Again, that’s a management failure. Management is supposed to handle schedules, targets, features, and scope. If they let the focus get blurry, the schedule slips, necessary features suffer from time spend on implementing new bling that may or may not offer any real value, and bloat becomes a real problem along with everything else.

Two Steve Jobs quotes are very appropriate here: “People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. Focus is about saying no.” (bolding mine)

And, “Real artists ship.”

I think both quotes demonstrate why Apple did a lot of the things they did. The iPhone didn’t have copy and paste for, what, two years? But then they implemented an approach that was regarded as about the best way possible to do C&P on a dedicated touchscreen device.

Why? Focus: they didn’t include something that they felt they couldn’t do properly. They worked on the stuff that needed doing now, and implemented the best possible approach later when they could devote more time and resources to that specific problem. And partially because of that tight focus, they generally hit their ship dates.

I’m not privy to any internal workings at Apple, but I’d be willing to bet that there was an official priority list of features at different tiers, and ship status was predicated on getting all tier 1 stuff to an acceptable level of doneness before implementing any extraneous projects.

Something that probably also helps is that Apple didn’t pre-announce things, so while I’m dead certain there were hard internal deadlines set, they weren’t beholden to an outside time schedule. That cuts some of the complexity, potential time and energy wasted on managing the interface between an outside company or companies, and also means that the right hand knew what the left was doing.

So if you want to keep a project on budget and on time you:
[ul]
[li]hire competent managers who[/li][li]create a realistic schedule and budget and who[/li][li]keep the engineers on task,[/li][li]say no to features that weren’t part of the original project unless something absolutely necessary for the actual functions was overlooked, in which case someone responsible for planning the mess fucked up and should be held accountable, and who[/li][li]deal with any of the outside concerns that suck up time and energy in project management, like multi-company interfaces, external schedules, and the like.[/li][/ul]

The only things engineers should have to deal with are the problems involved in building shit that works, and preferably works well. As much as is actually feasible they shouldn’t have to deal with people, budgets, schedules, or anything outside of their area of strength.

On the project end, you need to:
[ul][li]decide on a feature set[/li][li]stick to it[/li][li]limit complexity from the start, and prune as much as possible while building; simple code is code you can write more quickly, and is more maintainable[/li][li]set priorities and targets for the steps you need to do to implement a feature[/li][li]ship[/li][/ul]

If you know what steps you need to take, and have target dates or metrics to evaluate progress, you can find out pretty damn early on if there’s going to be a problem with meeting your schedule. If you don’t, then “it’ll be done when it gets done” becomes the prevailing mantra, and that leads to huge problems. Your managers should be riding herd anyway, but coders should also know to set aside ideas for features that come up along the way for future releases, not this one. Planning is also important. Figuring out both what you want to do and how you’re going to do it is far preferable to diving in enthusiastically like you’re building a treehouse with no plans, and then figuring out that you need more wood, some nails, and that brace you thought would work at the time is a little too rickety for comfort…so let’s pull this apart and re-use the wood for a new start.

On the other hand, I’m an industry outsider who has only gleaned this second-hand, from people who do work creating software, and in reading about the many, many failures of tech companies, so I could be full of shit. The same basic problems seem to crop up in multiple industries, though, and it doesn’t seem to me to be a unique set of problems in tech.

Can’t add much to what Treis and Sleel said- it’s all of those things, plus one important failure - to define, in advance, what will be done if something goes out of tolerance/budget. A properly managed project will have decided.in advance how much deviation from plan is tolerable and what happens when that is exceeded.

To add… people tend to measure the success of a project in terms of the time, cost and quality of the end product - ‘Delivered on time, to budget and to specification’ is a common phrase, but that view leads to auction-style barganing and nibbling, ultimately delivering something that wasn’t what it set out to be.

A project can be considered successful if it concludes, at the earliest possible stage, that the product cannot be delivered on time/to budget/specification, then closes down.

Back when I was running machine shops, salesmen and contractors learned the hard way to talk to me. Best example, some brand new salesman on the route didn’t read the notes and rolled in and said roughly ‘Hey honey, get me a coffee and where’s the boss.’ so I put my rude panties on and told him to screw off, go back and tell his boss to send a real salesman because I was the boss and I wasn’t his honey. Lost him the commission on a several million dollar order. 26 machine shops run through consumables like water in a busy summer. Sucked to be him, but I would be willing to bet he was a lot more respectful to secretaries after that.

Next time the contractor won’t talk to you, tell them that if they don’t they don;t get paid. If they have a boss, call their boss in front of them and ask if he is giving up on the contract and if he can suggest an alternate project manager. If he doesn’t have a boss, call your boss and tell them that the contractor is refusing to work with you so you are going to replace them with a new vendor. See if that shakes them loose.

This is what’s done way too often by external contractors. First they do a Best Case Projection, then they tell the customer it’s going to take 3 months less than what the projection says.

When you actually build expectable delays and a certain amount of unexpectables (1), things actually follow the map. When you plan a route from Porto to Denver where one of the steps is “walk under the sea from Lisbon to New Yor City”, you’ve got a problem - but often, the people who need to approve the proposal don’t know there’s an ocean there (I mean, that’s what they hire experts for, to tell them - but too often those experts lie).

1: just based on previous experience - you can’t know whether someone will get sick or what with, but you know that if your project involves 100 people and a year, several of them will; if it involves 3 people and 3 weeks, most likely nobody will. The amount of unexpectables you have to allot will vary with the nature and size of the project and team.

three months late, 35% overbudget, and half a world away from the intended target.

Exactly - which wasn’t ahat anyone asked for in the first place - and therefore the project, although it delivered something, has failed to do what it was created to do.

Stages of a Project:

  1. Euphoria and Excitement

  2. Disenchantment

  3. Panic

  4. Search for the Guilty

  5. Punishment of the Innocent

  6. Reward for the Uninvolved

(Hey, someone had to do it…)

Side note:

Years ago, I closely watched the progress of a culvert replacement job in western MA (it was on a route I traveled frequently). It took over a year to complete. No hidden gas lines, electric cables to re-route, floods, nothing unusual. Just a larg-ish culvert under a secondary road.

Seriously? Almost 15 months for a frikkin’ culvert?

Small wonder infrastructure=stimulus, I suppose…

It sure seems to be standard in IT. Nearly every project is over budget, over time or both. But IT professionals simply quote the figues as an excuse.

“Well, 80 percent of projects are over budget, over time or both.”

Imagine if other professionals tried that.

“Well, 80 percent of my diagnoses are wrong.”

“Well, 80 percent of my investment advice loses money.”

I’m on one now. I estimated 18 months after we got everything signed and approved. My boss told his boss twelve months starting “today.”

It took six months to get things signed and approved. It will take two years from “today.”

Now I know why you’re considered dangerous: you actually provide realistic estimates!

(So do I. Don’t tell anybody)

For software, it’s almost always because people don’t want to spend the money/take the time to create specifications that are in-depth enough to create accurate estimates from. It’s a bit of a catch 22; to really estimate a software project correctly, you have to do a huge amount of work up front. You have to figure out flow, create screen mock-ups, map out a database schema, etc. For complex projects, that can take fully 50% of the project time & budget.

Now, most places don’t want to sign off on a project without an estimate. But an estimate can take several weeks (or months, if it’s a big project). So a lot of companies are in the position where they simply have to guess. If the project is similar to other projects they’ve done, it might be somewhat accurate. If it’s not, who the heck knows.

I’m in the midst of one of those now; we’re doing a project unlike any other I’ve done before. My client asked me to do the estimate, and I did one. We all knew it was a shot in the dark, because they didn’t want me to take the time to do all the stuff I listed above. I did maybe half of what I think was necessary to do a good estimate. And, to no one’s surprise, it’s taking longer than we thought.

This used to bug me a lot when I was a wee code monkey. Nowadays, I just shrug, and say “it’ll be done when it’s done.”

For the political projects, the answer is easy. The politicians lied their ass off to get buy in. This happens a lot in big bureaucracies also.

Also I’ve found that the guys that have to do the work have zero input on the schedule. Some marketing guy says they need this new product to announce at a show in 6 months and that is your schedule. Most people figure out that if they insist the schedule is impossible they will be fired so they keep their head down and send out their resumes or try to transfer before the chickens come home to roost. In any case, the consequences of failure are often less than those of insubordination.