I have been goaded into this.

When people talk about Agile as compared to Traditional software development, it generally translates as Scrum as compared to Waterfall. Not all the time, but generally. There is the impression that Agile is tied to a particular set of to-do’s and structures and that it’s arch enemy, that which has proven inadequate and totally unrealistic, is the Waterfall a.k.a. Traditional approach.

This is a fallacy. On several levels. As Public Enemy said, don’t believe the hype.

For one, Agile is simply a school of thought. It is not Scrum any more than it is XP, or a combination of the two, or something you make up that follows a core set of principles. Like there are the 10 Commandments and various ways to interpret them, there is the Agile Manifesto and various ways to put them in place.

For an Agile method to be valid, it does not need to have a recognized acronym or have a book written about it. It does NOT need to be branded. This is my main beef with Agilists. I like Agile approaches. They are common sense, really. You do your morning routine in an Agile way. Time to brush your teeth? Cant find your toothbrush? Use your finger, or your wife’s (sorry sweetheart). It is adaptation, at the core of the Agile Manifesto:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Taking these one at a time:

Individuals and interactions over processes and tools says to ME that the people in your team are more important than the fact that you want them to Scrum. What will work for them? Stand-ups? A backlog? Both? XP? Just being in the same room talking all the time and having a terrific dynamic? These are all good, if they help deliver good software and/or software as contracted. You cannot forget about the contract, although it is not the end-all, be-all of the project. If an internal effort, there is an implied contract between development and the stakeholders. Look at the main statement again. Process is Scrum. Tools are backlogs and burndown charts. Again, the people are more important than any artifact, process, or tool.They come first, then their methods of work. This is just work, folks.

Working software over comprehensive documentation says that it is more important that you produce something that works than you can demonstrate your immediate impression of it on paper at any given point. I have heard it said that the functional requirements for software is the final codebase. The idea of comprehensive documentation is, I believe, a throwback to Big Requirements Up Front methods of analysis. But still, if the Client demands this, and they will, people come before process. If you make a Scope Document or a Class Diagram, or do some UML, this does not mean you are not Agile. In many cases, you have to – particularly where there will be integration points. You kinda want to be aware of this up front, and it might even be in everyone’s best interest to have the high level – HIGH LEVEL - architecture documented. Almost everyone who has had any experience in software knows that if you make a GANTT chart at the beginning of a project and compare it to what happened, it will almost be apples and oranges (with the difference increasing exponentially with the time, scope, and complexity involved).

Customer collaboration over contract negotiation says that the Client is part of the team, and if you are going to be Agile in some fashion, they should be aware of it and on board, ready to take part. This winds up being more important for the development team to remember (and the salespeople as well) than anything else. Instead of worrying too much about having everything signed off on before you begin (death trap), you want a dialog to be ongoing and indeed, iterative refinement (an MSF term), progressive elaboration (a *gasp* PMI term), and any other term (continuous integration, etc.) that describes the dynamic allowing for change to occur with a degree of transparency should be of foremost concern. It is also, I think, another way of saying that people come before process and that working software comes before a giant stack of documentation.

Responding to change over following a plan means that sometimes, even your Scrum team might have to break into “Traditional” mode. It might. It also means the obvious: expecting change will be more useful than  reacting to it. The former manifestation of this statement is the important one, I think, since we all know what “Agile” means and it is tiresome at this point to reiterate. I don’t think it is impossible that an Agile team would find itself forced to map against dot notated requirements at the request of a CFO at a Client organization. This has happened to me. We did it, but as people fought it, it strained the team and doubt snuck into the Client’s minds. Your Client is probably not totally hip to what you are doing, even if you describe it, and especially if their boss was not in on the conversation and is asking, “how much and when?!”. For the team to say, “this is not Agile” is not something that goes along with responding to change. It says you had a plan, and this isn’t part of it. It is hypocritial, really. It is a pain, sure, but are you going to complain about work being hard sometimes? I am sure you will find endless lakes of empathy.

This is why I take issue with companies that will teach yours to be Agile, yet show up onsite with books on Scrum and Scrum only, just touching on the Manifesto and common sense reality behind things. It isn’t about Scrum. It is about realizing that software is like anything else and does not always go according to plan, that people and relationships make projects sucessful and not contracts, that your Client has to like what you deliver and be aware of what you are doing to feel as though they have not thrown money into a time machine, and to know that there is no cookie cutter solution to *anything*, much less software.

Software Engineering and the Development process is a Complex Adaptive System, and like any other CAS, (this is an excerpt from the preceding link, which is not about software and software only, but about CAS) there are things we can look for:

· Simple Rules: Complex adaptive systems are not complicated. The emerging patterns may have a rich variety, but like a kaleidoscope the rules governing the function of the system are quite simple. A classic example is that all the water systems in the world, all the streams, rivers, lakes, oceans, waterfalls etc with their infinite beauty, power and variety are governed by the simple principle that water finds its own level.

· Iteration: Small changes in the initial conditions of the system can have significant effects after they have passed through the emergence – feedback loop a few times (often referred to as the butterfly effect). A rolling snowball for example gains on each roll much more snow than it did on the previous roll and very soon a fist sized snowball becomes a giant one.

· Self Organising: There is no hierarchy of command and control in a complex adaptive system. There is no planning or managing, but there is a constant re-organising to find the best fit with the environment. A classic example is that if one were to take any western town and add up all the food in the shops and divide by the number of people in the town there will be near enough two weeks supply of food, but there is no food plan, food manager or any other formal controlling process. The system is continually self organising through the process of emergence and feedback.

This is my issue with proselytizing Agilists. They have their way, and it is THE way, and if you challenge them they say, “well, how is your Waterfall project going?” and give you a smart ass grin.

Like I said, I really dig the Agile Manifesto. I even like Scrum and XP, when they work. Try forcing a company who has been used to producing software under a Fountain methodology or a Spiral methodology to do Scrum and take away their coffee, force them to stand up at 8AM when they are used to their routine. They will resist if they have seen any success from what they have been doing, and believe me there really are companies that produce good software under models that are not branded Agile.

As I always say, take the good, leave the bad, and approach every project with an open mind and open toolbox.

And yeah, I will also state that Agile Certification is a sham. I will certify you. Send me $500 (sale!) and a photo of you next to a nice burndown chart or sitting next to another developer and you will get a piece of paper that says you are a bona-fide Agile Professional. It will mean as much as any other Agile Certification does. You probably know people who studied for the MCSE, passed the test, and then looked for work in IT with the certification in hand but no experience. Not much difference here, except that Agile Certification classes dont even require you to take a test. If they just called it “class” or “education” I’d have no issue with it.

If you want to learn and be good at this stuff, read, talk, and use it. In other words, follow the Agile Manifesto.

Is that a “duh” or what?

Thank you,

Josh Milane

MIT Consultants, LLC

PS – I like UML. Visio is my friend. Not only that, I like COMPLIANT UML. Activity Diagrams especially, with pins and all. When there are complex business rules, it helps to see them in front of you and (imagine this?) improve the processes that are in place. I also like scoping up front, in a broad and shallow way. And I am an agile proponent. However, I am foremost a proponent of the Client.

I sort of wonder if this post won’t irritate people, including people I really respect and like. In fact, I sort of expect it will. That is not my intention and I feel like I have to proffer something along the lines of a disclaimer. I do not in any way intend to disrespect the discipline of software development or software project management. If anything, this should be seen as a tribute to the celerity of those involved in establishing Lean, Agile, other principles within a remarkably dynamic arena. Also, at the end, I plan on closing with something to bring this all together. That might not happen, because I try to post without editing in some attempt to exercise my lucidity.

Still, in the process of communicating what is on my mind, some people may be offended and I only ask that you understand I am one of you and if I ever stop asking questions – even about those things I think I know inside and out, I should be slapped around (lightly, please).

So, this morning I stumbled down the hall, aching as usual, like a bear with arthritis, and Flip This House or something just like it was on the television. I rarely watch TV, but the channel was already on, my wrist hurt, I did not know where the clicker was, and I got sucked in. Very quickly, the Construction Project Manager said  that he needed to visit the house that they were going to “flip” to define his scope of work.

I am thinking that the next time I work on a refactoring project, I will call it “Flipping this Software”.

As much as I had to offer an apology to IT PMs and Developers, I have to apologize to my Father as well. He owned a construction company for 20 years and built it into a nice success through blood and broken bones. This “realization” is a little embarrassing to me but I am glad that my Dad doesn’t use the computer except to look at pictures of cars and stuff like that. Anyhow, I worked for his sprinkler fitting company from about the ages of 11 to 21. I was Manager at one point, towards the end, when we closed to company due to reasons that are not appropriate to address here and would launch a tirade of who knows what nonsense I would spew about life at that point. The company closed as I was graduating college with a Philosophy degree, before my abrupt transition to IT, and I think Dad’s making such a big deal out of me getting a Colgate degree and not having to break my back every day resounded within me as profound. Indeed, the effort and gesture surely was. Somehow, it made me think I was doing something *entirely* different than he did. Until this morning, that seemed largely the case except for things like how to deal with Clients and other soft skills where he could still teach me a lot and where we now see eye to eye.

There seemed to be two kinds of projects; there were projects where you went home covered in dirt and grease and there were projects where you went home with a sore ear from your headset and sore fingers from typing. One was low tech and one was high tech – as though “tech” was a measuring stick of some sort. I am feeling a little dumb right now. On the plus side, I think my CV has to say that I have 15 years of PM experience and 12 years of IT PM experience now.

I remember few things from when I was young, but one of the things I remember was waking up at 4:20 in the morning and meeting the rest of the crew at the shop. The shop was where we did fabrication, preparation, loaded the trucks, and talked about how yesterday went, what was going on today (as well as who was going to do it), and what we were running out of in terms of supplies, what permits we didn’t have, or what other things were impeding progress. There were buckets we could have sat on, but no chairs, and still everyone stood without being told that there was a reason to stand.

This should start to sound familiar.

In construction, there is no talk of Scrum or XP. Someone might say, “huddle up!” or “get your @sses over here!” but stand-up /scrum was not in the vernacular. We had stand-up meetings every morning. How else will they know what’s going on? And more often than not, people work together on the same feature (hanging a 15′ 10″ pipe 30″ in the air takes more than one person, trust me and my compression fractures) and people need to know what is going on so they know what they can do or can’t do or might want to think about doing.

Also, people who were hungover and didn’t make that meeting did not last long. Dad was tough like that. I rode in with him so I couldn’t be late, even if I wanted to… and I DID want to.

The argument I have heard, among others, is that software development is not widget manufacturing. This is true, but widget manufacturing is not software development either. Let’s look at this a bit. Gather ‘roud.

Each process involves tasks, people, and steps (yes – budget, scope, time, cost, customer satisfaction, etc., etc., et cetera too). The kicker – the x factor – is always the unexpected, and Agile “expects the unexpected” as opposed to it’s mortal enemy: The Waterfall Method. As we should say in the shop (because we talked this way), “shit happens”. Things fell on your head. The Inspector was a stickler. The carpet crew came in a week early. Whatever. Dad didn’t want to hear about what the problem was as much as he wanted to figure out how to fix it or to make people aware of it and incorporate problem solving into the process of construction (oddly, the phase of MSF where you code is called Construction, but maybe it is not all that odd). ScrumMasters are trained (hopefully) to do this, but in other industries, where the deliverable is something you feel and see, it is perhaps a much more obvious question because (and I am sure there are other reasons as well) it is not something you can let slide, even for a day. You can’t just walk away and assume it will resolve itself and hope/assume a specialist of some sort will handle it because it *will* be brought up the next day and some ambiguous excuse like “it just takes longer than we thought… I need more time” won’t fly. I am not saying that a poor excuse will fly in software development, but because it is a more amorphous process, I have seen it accepted at face value and passed over much more often than I have seen it glossed over in construction. Screwing in a pipe fitting is a simpler process than coding a web service, but there is a lot more involved than the actual performance of a task. Try coding a web service with the keyboard over your head, on a 18′ ladder, with greasy boots and people running around like crazy with heavy equipment and sharp things that can cut you underneath.

Software development is not harder than manual labor. It may be more cerebral, but hey, you made that choice; you chose to work your mind harder than your body. You need to live with it and be accountable, responsible, and professional.I have my scars from working with pipe, and I have scars from software too. Most cause me to write. They aren’t as cool to the chicks, as they say.

Expect the unexpected. Be Agile and adaptable. This makes sense to my readers when it comes to building software. For a great example of how this holds true in other vacations, please read this account of a software professional building a fort. All the examples I could give of the unexpected happening, or of dependencies changing form, requirements shifting, and the like are there. Still, I think the following is interesting, which means I force you to read it if you want to read this top to bottom:

• While I worked with pipe and grease and smart people with largely non-formal education:
  • Scrums happened every morning because it was required. What needed to go on the truck? What happened with that standpipe yesterday? Did the ceiling guys come? Did the fabrication shop drop off the pipe this morning at the job site? Etc., etc., and et cetera.
  • XP happened every time there was >1 humans required to accomplish the task effectively. Try having two people hammer a nail… it is like XP in a lot of places I have seen XP explored. IMHO. This is not to say that it is always a total and complete utter waste of time when you have competant people. Not at all.
  • Lean Principles were mandated. Waste? Forget about it. If we had a 12′ piece of scrap in the back and could cut a foot off of it, thread it, and use it, that was one less giant piece of metal in the shop and it only made “duh” sense. Waste was visible, and smelled a lot of the time. It was physically dangerous, too. And there was no way in hell I was going to drag the stuff to drop 50 sprinkler heads to the site before it was needed unless it involved no overhead or risk.
  • The Waterfall approach was understood to be “sunny day” at best, and was a good thing to think about at project onset, but only to see where it wouldn’t work or be dependable. Even most of the strict Agilists will admit that before you start coding, you need a User Story of some sort.

The methodology that reigned was “what makes sense for this project?” Implied was these conditions, these circumstances, with this stuff going on simultaneously, with this person out sick, etc., etc., et cetera. Every job was custom, although the good old toolbox was always nearby (see previous metaphors about toolboxes for yet another metaphor/pun).

There is no way someone could walk out of school, even (especially?) with an MBA and be an effective construction project manager. And we were a subcontractor. We had our challenges and the General Contractor got to inherit them, or us, in addition to his/her own: OSHA coming (watch everyone run for their hardhats), budget cuts, etc., etc., et cetera.

So my major beef is that we have Agilists and the like bickering back and forth about how novel their approach is while we have others saying that it (agile development) has been around 15 whole years. Neither is true. It has been around for a long time. Software development in OO enterprises has not been around a long time. People are funny. They will find a way to adapt to new challenges. People are agile. It is not that these people bother me in their declarations of novelty, but more so in the fact that they seem to have ad hoc eliminated the possibility from learning from other disciplines, with Toyota being the most notable exception. Maybe the cool Japanese words like muda make papers on Agile seem almost enlightened and Eastern. I don’t know. Maybe success speaks to everyone. I don’t know. Both postulations probably hold some water. I like the idea of people selling a process as an aincent art by using an exotic work much like I like the absurdity of people who have Chinese characters framed in their house without knowing what they mean for sure. (I always kind of hope they really mean “Whomever hangs this is an @sshole”). 

“Interesting painting you have there… or, I guess that’s a word or a letter or something?”

“Yes, that means ‘peace’.”

“Ah. Peace. Nice. So it’s framed, and in Chinese instead of English, why?”

“Well, there was a space there on the wall, and me and Mrs. Simmons have strong dislike for muda.”

(No offense to those of you with Chinese character tattoos. That was a smart move. No, really. Looks exotic and edgy, just like the tribal band around your arm.)

Seems it isn’t enough to have something smart or good. You have to be fashionable and have a cool name, decoration, or what have you. Read about the success of  Yellow Tail wine for more on that. Don’t get into the Six Sigma Blue Ocean theory. It’s theory.

We are all on the same page. We are just looking at it from different angles, which is GOOD. But now, let’s look at more than that one page and try to gain some perspective? I understand about the productization of methodologies. I really do. I can’t say I do not hope my book does not sell like crazy. Still, you gotta be in this because you love it. If you love it, and even if it is a love/hate relationship like mine, you involve more than your brain. Your heart is tugged, and you go home wishing you had done something better even on your best day. I would like to see more of that. Honest work towards refining the process of developing software will make the world a better place. More focus on Semantic Web technologies will make the world a better place.

And you will make a pretty good living if you work at it, I think. Especially if you are talented and ask the tough questions, learn the hard lessons, can offer calloused hands to the effort. I have nothing against smooth hands. I just don’t want my carpenter to have them. I lift weights, so I have hands like an elephant’s heel, if they have heels.

Additional Questions:

• How is developing custom software for an unsure Client truly different than being an interior designer for the stars, or a wedding dress designer for a new bride? Besides coding. Don’t be a wiseguy, please. Leave that to me, unless you’re really funny.
• How is doing iterative development for a large and sexy Client (using RIP, for instance) truly different from a home architect working with a wealthy and irritable dot com couple circa 1999?
• You get the idea.

Thank you for reading. Meanwhile, please try not to muda any Yellow Tail.

Josh Milane