I’m about as far from a compulsive as you can get. However, there are also a lot of teams that do incredibly stupid things and use “Agile” as the magic word to explain their problems. There are also teams that somehow manage to configure Agile to exacerbate their issues rather than resolve them. (The teams that move to agile because they were spending so much time in status meetings and somehow turned every standup into an additional status meeting on top of the ones they already have. The teams that know things aren’t working well, but never do retrospectives to examine why that might be, because that would take time away from working dysfunctionally.)
Businesses are into fads just like people are. Agile seems like the latest fad for CS projects. In a few years something else will replace it.
So, our company started doing Scrum in the past couple of years and I can see some advantages to it.
I can also see that there are some misconceptions here… at least according to the person who taught us how to Scrum
First, Agile is a philosophy, not a methodology. Scrum is a methodology that fits well with the Agile philosophy. So is Kanban. There are probably a couple of other methodologies out there, but we didn’t hear about them.
Agile was a reaction to over-proliferation of documentation that many groups were doing in the late 90s, early 2000s, and to some extent before that. Some of the documentation methods were getting so complex that for one document type, I was given a week’s training on how to create and interpret the document. (IMO, a document should be self-explanatory, or at least understandable with a key)
I think it was also driven by a factoid that I heard in the late 90s - that people can only successfully estimate if they are going to be finished or behind in a project that is <= 3 weeks long. So, if you create a project that you estimate is going to take 6 months long, and you don’t know that you’re going to miss that estimate until you’re at the 5 month mark… you’re screwed and it’s way too late to do anything about it. But if you break it down into 2-3 week mini-projects, then you know halfway into your first project if you are good or need more time and/or resources.
The point of Scrum and of Kanban is not to reduce meeting time, or to increase the amount of code produced. It’s to increase predictability over time, so you can successfully convey to the business that if they want this particular feature, it will be 1 month or 6 months, and be reasonably close to correct… after you’ve had a lot of experience with the estimation portion of the cycle. And assuming that your team doing the estimation has a decent time sense, something I suspect is in short supply in developers.
It’s also to increase the amount of time spent on working on the right thing. It can do this by getting iterations of the code to the testers and the people who might actually be using it earlier in the process. So, instead of spending 6 months designing and another 3 months coding a big whomp-a-deedoo project, and then showing it to the users who then say WTF, leaving you with 9 months of work down the drain because you can’t do what the users say they needed without starting over, you spend a couple of days working on a small portion of the process, write just enough code that you can show it to the users, and then if they have feedback, it goes straight into the next iteration of the design process.
Of course, all this is somewhat theoretical, but if you get to the coding phase without having to generate a dead tree’s worth of documents (that are ignored the minute they’re printed out), you’re still better off than with some of the projects I’ve worked on.
Not really. First, because see above as to what Agile is.
Second, Waterfall is more of a description of how things actually tended to happen, with a touch of OCD, rather than a methodology. All projects need a design, documentation, coding, testing, deployment and support. We’re not going to get away from that. Waterfall happens when TPTB decide that we can’t start one step until we’ve finished the previous one.
Scrum can end up as mini-waterfalls, and the Scrum purists will decry that. And it helps to have the testers reviewing the design while the developers are starting on the coding. But you still should spend some time thinking about how you’re going to approach a problem before you actually tackle it. (And this is from the queen of “Let me just start writing code TYVM”)
Of course, we are still going to end up with some users going WTF?
Agile also addresses a fundamental problem with Waterfall in that it’s almost impossible to come up with a perfect design before you start building the software, becaause you can almost never gather all the requirements and dependencies with enough detail. But going back and changing the design once you are into the build phase can become prohibitively expensive. So typically what you would end up with in waterfall projects is a “long tail” of bug fixes dragging out beyond the end of the project. Agile tries to reduce that risk by developing self contained modules of completed code.
Nothing can replace experience and brains. The real trick is having enough experience to know which methodology or approach is appropriate for a given situation/problem/project/task etc.
For most larger projects I am part of we use a mixture of approaches because a project is rarely a collection of tasks that all have the same nature.
Some factors that influence the approach for specific tasks and possibly the overall project:
Geography and time zones of resources
Required methodologies of different teams due to being part of different organizations
Is the solution to the problem known in advance (design up front), or is it unknown and requires iterative exploration with the business?
Is it possible to go live with a partial product, or is that not really possible?
Skills and experience of resources available (this is a big one that requires significant adapting of approach to the people available, a team of 15 year domain veterans can be handled very differently than a team of 5 year domain rookies)
Exactly.
Now, lest y’all think I consider Agile to be the greatest thing ever…
One of the concepts of Agile is “self-organizing teams”. This is the idea that the work gets done best if your team as a group decides what to work on, and who does the work.
Now, the original people who came up with agile were really really good programmers, and if your team is composed of really good programmers who don’t just know what they are doing, but are willing to admit that other people on the team are better at some things, and willing to figure out what everyone is good at, a self-organizing team will work well.
But if your team has someone who thinks they know everything, and/or aren’t really that good at thinking through what they are doing, and, when things go poorly, you’re included in the recipient of the shit storm…
FML
One of the things that I feel that Agile proponents frequently fail to do is discuss the prerequisites for successfully implementing an Agile methodology. This can lead to serious project failures due to incompatibilities between the methodology and the available tools and people to implement it. For example, a big part of Agile is performing frequent releases. A hidden assumption here is that performing a release is so cheap it might as well be free. That’s not always true, though. Sure, building and packaging your software is basically free, but what does it cost you to test it? You had better have a a serious automated test suite, with little-to-no manual testing required to validate the release. You’re also going to need to ensure that you have enough computing resources to run tests at a higher cadence. If you don’t ensure that you have these two things, you’re either going to bottleneck at the testing phase and strangle your process, or lower your quality standards for your release.
Many software projects do have a very good automated suite, and can afford to have enough hardware/VM time/whatever to run those tests. But if you don’t, Agile isn’t for you until you address that.
Also, I just said that building your software is free, but what if it’s not software? I used to work at a company that produced hardware for internet service providers. My VP once told me a story about a time he went to a software conference, and at some discussion with a bunch of Agile advocates that when developing a new hardware platform, we should produce a new board every 2 weeks so we could do the hardware side as an Agile project. Never mind that there was a minimum 6 week lead time to do a new board revision at the manufacturer. Never mind that spinning just one revision cost the company a not-insignificant fraction of its yearly revenue. These people were so stuck in their bubble that they just couldn’t comprehend that any of this could possibly be a bigger problem than not following an Agile methodology.
There are a lot of other hidden assumptions. Another example: one of the big benefits of frequent releases is that you can get frequent feedback from the customer to guide your development. That’s a great thing to have, but the hidden assumption here is that it’s possible to get that feedback in the first place! If you’re developing a tool for internal use at your employer, which I’ll admit comprises a big amount of code written today, that’s easy (assuming upper management is on-board). But when you’re producing hardware (and the associated software) for ISPs? Good luck. We would have been thrilled if our customers upgraded their software once a year rather than us having to support old, buggy software all the time. Getting customers to upgrade frequently and give feedback on the features just wasn’t in the cards. The point that it’s very difficult to properly design a whole feature up-front is well taken, but in a number of markets there’s really no other way.
On the flip side, one of the things that the software group at that company had to produce was tools that the hardware designers could use to validate the operation of their design. This was always a big problem because new hardware required new software tools, and the tools were usually being written in parallel with the design being validated. When the software team switched to an Agile-style delivery mechanism for these tools during a design project it eliminated a whole host of problems.
The moral is that you can’t just choose your development methodology in a vacuum. It impacts basically everything related to your software, so you have to understand what constraints your development is under first.
Good programmers can be good in almost any environment.
Bad programmers will always be bad.
Mediocre programmers have a wider range, and are the ones who can really benefit from some environments over others.
Therefore, the goal of a methodology is to allow the mediocre programmers to be as good as they can be, without handicapping your good programmers too severely.
Because your good programmers can leave, leaving you with mediocre programmers who will be traumatized by any shift in the environment.
Waterfall does handicap good programmers if you’re too anal about setting the previous phase in stone once you’ve moved on. This is, as has been mentioned, especially true with requirements, which are subject to change in almost all kinds of software development. Missile guidance computers can have requirements set in stone, automated defibrillator machines can have requirements so set in stone it would literally be against Federal law to modify them past a certain point, but your company’s product almost certainly has fluid requirements. You might not know it, but the people using it certainly do, and your competition certainly does.
So we have to have some idea of how to work on software while acknowledging that fact. To take an extreme example, it can be as simple as One-Person Agile or Gymnast, both names I made up for when there’s one programmer going through the whole process of making a program, often for one other person. It’s Agile in that the programmer gets an idea of what the other person wants, goes off to code something, and then uses that to give the other person a basis for making suggestions. Iterate that, and you end up with something the other person’s happy with, by an evolutionary process. Real Agile is more complicated, because managing multiple people is inherently complicated, but it’s spiritually similar and, as Rysto says, makes similar assumptions about how cheap development is and how available feedback is.