I forget who said it, but I will find out I am sure, that when a Client asked them how to have fewer bugs in their code and how to create less defects, the person responded “Build less stuff.” Pace is going to be a common theme in any development effort with Scope. Pace is subject to all kinds of theories, including my own (which is really a mishmash of what other people have said and what other people have did – I cannot pretend to have walked down the mountain with stone tablets dictating the Commandments of Building Software and I think the Agile Manifesto – the first page at least – is common sense).
Yes, a lot of this stuff is simple but taking the simple and fine tuning it for a specific environment takes discipline and skill. The traditional project manager blankets a methodology over a project and tucks it in here, loosens it there. They are nervous right now, except in large organizations where silos hold the place up (and those same places will be overtaken in my opinion by leaner, more Agile, more realistic and “human” organizations).
I am already displaying my prejudices. It may be important to note that I am going to write this with all prejudices towards Agility aside, to the largest extent that it is possible to not have an opinion. I will try to say what is my opinion and what is not only my opinion but what is established as a kind of “industry fact”. Agile, in my mind, is undergoing a bit of a transformation at least in part to Mark Kennaley’s book: SDLC 3.0 Beyond a Tacit Understanding of Agile. I cannot recommend the book until it is released with legible figures and diagrams, but Mark is working on that and we have discussed that. It turns out that the “Waterfall” approach is an erroneous interpretation of a paper from the 70’s. People were developing in an Agile, or at least, iterative, fashion a long time ago. It is just that Agile was not commoditized or productized like it is today and that (and this is my opinion) the OO development tools that were of the “Visual” nature like Visual Basic and PowerBuilder make iterating that much easier. In fact, OO (object orientated development as a whole) made it 956 times easier (that figure is mine, not a standard) to write code in an encapsulated manner. Either way, methodology will not come into play for purposes of this article. At least, I will try to not assume one or the other before going down a mental path. As a side note, mostly for Scott Ambler, I do not think that calling someone a Certified Scrum Master implies super powers any more than calling them a Scout Master or Master of Ceremonies does. People are not silly enough to accept things without asking questions, although I have had HR approach me saying they want a CSM with 10 years of SharePoint 2007 experience. HR is a different animal.
Before I get too much into bug/issue tracking, it might make sense to address testing, since testing is supposed to prevent bugs or catch them before they manifest as expensive problems. The common way to estimate how expensive a bug fix is involves two approaches: with a live system, what is the cost of the functionality not working. On a live or new system: basing the cost of a bug on which of three phases it is found in: design, development, post-deployment. A $100 bug caught in Design will cost $1000 to fix during development and $10,000 to fix once the code is deployed. This is the most conservative of all estimates I have heard, with the more contemporary views showing that a bug caught post-deployment is 800 times more expensive to fix than if it is caught during a true unit test in a true iterative test-driven SDLC development methodology. These figures obviously depend on architecture and the nature of the bug as well as a bunch of other things like dependencies, available resources, how many cooks are in the kitchen, etc. Either way, this basic idea of determining the cost of a bug (or a feature) depends on abstracting data that is gathered from metrics gathered during discreet project phases, as well as “sizing” the effort involved and even developing against specs while delivering against specs that have a value assigned to them, but that has not been the case in any obvious way for the initiatives some organizations I have worked with have led, as far as I could see. There are “iteration zero” tasks like environment setup, but even these have been problematic. I would point to the lack of internal planning and vendor management (demanding value is demonstrated) as the culprit. I would suggest that a proper Discovery phase was not executed, and that Agile is not figuring things out as you go; Agile is about moving forward as soon as you can, with knowledge, and eliminating waste.
Projects have thousands of pages of Requirements Documents, but Design is not Requirements Documentation nor is Discovery. 999 times out of 1000, the final product does not reflect anything besides “broad and shallow” Requirements Documentation. Even then, what is that statistic? Only something like half of the features in a software product are ever used once the application is up and running. A big reason flexible methodologies are getting tra
ction now is because if you look at a project plan on day 1 and then look at the project plan as it actually happened once you are live… chances are there is almost no resemblance beyond the Phases and the fact that they started, worked, fixed, and eventually finished (not always in that order). This is my opinion, but again rests on the reasonable assumption that software designed to obviate bottlenecks, waste, and to track change makes those things easier to do. The larger the project gets, the more dissimilar the “Soothsaying GANTT” is to the “GANTT of Reality”. Kennaley refers to the idea of “Big Requirements Up Front” or “BRUF” as “perishable goods”. He is an IBM guy, so it is very meaningful to me that he would say it. Regardless, as pointed out in my most popular article, the major schools of thought share quite a bit in regard to structuring projects (and I happen to disagree with myself but leave it up anyhow): http://mittechnical.com/steps-towards-creating-a-custom-project-management-methodology/2007 – the common thought has been to track every requirement and have traceability matrices, but the rug gets pulled out once you realize that things change. Inevitably.
It would be easier to win the lotto twice than to do a high fidelity project plan for something more complex than a simple brochure site with complete OOTB functionality. Even in unit testing, we have to realize that it is not “eyeballing” the code and that there are tons of tools out there to help us. Ask your friends which they like. Ask them to show you how to use it. See if it does not make sense to you and see if they find it useful. Unit testing is often “well, we just write good code and we know because we comment it and look at each other’s stuff”. This is a horrible mistake and the foundation for sinkholes down the road. For Java applications, J-Unit is the most commonly used tool. J-Unit and Cucumber (Behavior-driven development tool, not the vegetable) makes some developers have epiphanies that are amazing to watch. For .NET projects, N-Unit is the most common. It is simply a matter of being aware of them. I have yet to introduce a developer to these tools, or Test-Driven Development / Behavior-Driven Development and not had them fall in love. Once they did not like it. One individual did not. He wound up losing his job because he was by far the least productive, stuck in his old methods. I do not blame myself for that. If you get a tool that helps you be better, you are kind of saying something when you refuse to use it. When your stubbornness turns to financial loss, you are going to be fired. All he had to do was ask for help. Lord knows he did enough chatting about the girls he could see from his window taking smoke breaks.
There are tons and tons of other automated testing tools out there where people would rather rely on a tool than discipline combined with an automated helper or where they would do the ideal thing and combine the two. Point is, overkill is overkill, but I will borrow once again from Scott Ambler and say that “Barely Good Enough” documentation is required and anyone in IT knows that there is analysis paralysis as well as cowboy coding. We want to be in the middle somewhere but above all else, I believe we (as Clients) want to be the place where the information is and the center of collaboration. The Client organization, if possible, wants to be the “Single Point of Truth” for both documentation and process control. That is knowledge that will be invaluable at some point and it will prevent obfuscation. If a vendor already has something like Bugzilla mastered, you want a login. Do not force change where it will not be helpful, please.
If we miss the aforementioned large chunks of substantive work, we will inevitably see (later rather than sooner) the rise of “issues.” The semantics of calling bugs “issues” have several points of genesis. Many Developers like to call bugs “issues” because after all, they don’t make bugs. Bugs are dirty and involve mistakes. The funny developers call bugs “features” and I guess after 20 years that is still funny. I prefer separating bugs from issues at least in category buckets myself because then the difference between them becomes more obvious and they are handled differently. If math is being executed and addition is inaccurate, we have a bug/issue but if we are not sure if interfacing with an overseas payment processor is possible without their being PCI audited, that is an issue. An issue I actually encountered not too long ago: a development shop wanted to build a RoR font end on top of Ektron. This was one of those delicate issues that was tracked offline.
As with most things, our objective should be to prevent the disease instead of band-aiding the symptoms. To do that, you also need visibility into the overall project health. You do not need granular detail, but if you know that we have 100 units of stuff to fix last week and this week we have 140, things are not getting better. This concept of assigning points to features can be applied to bugs as well. It is all is the estimate and difficulty level. You wind up getting a very visible “velocity” over time where you can see time on the X axis and total open points of the Y axis… you want that line to wind up at zero by a date you have in mind or, more realistically, let it tell you when you will hit zero and adjust the feature/bug (product backlog) accordingly. A tell-tale sign that your vendor is not as familiar with bug tracking or issue tracking is, in my opinion, if their unique identifier (ID) changes or if both entities are tracked in the same manner. The idea of velocity and the multiplicity of velocity is staggering. It does not simply apply to Scrum burndown charts.
Mike Cohn and others divide testing into 4 types: Automated and Manual, Automated, Manual, and Tool-Based. I suppose a footnote belongs here, but as this little tidbit is as much from experience and a variety of media as it is his book (I am not sure which one, sorry).
- Automated and Manual Testing involved prototyping, simulations, functional testing, and user story testing. I personally extend this to service testing (interface testing, touch point testing, etc). For all of these we have information driving acceptance criteria (did it pass?) and the creation of the test (what are we trying to do?) These lead to trackable bugs and potentially, issues. “Fail fast” is made easy by prototyping.
- Automated Testing includes having a Continuous Integration server, Unit tests (this is looking at and optimizing actual code in an XP peer programming model or not and specifically not doing manual testing where you enter orders or send an email to multiple recipients), and Component testing (also done in Tool-driven testing if you have systems within the System configured in a way to allow for it). Your CPU and compiler will tell you a lot. Automatic processes should be set up so that they take place before anything goes live (or even into a sandbox environment). I have seen development shops where if one of these tests failed, a flashing red light went off and in Toyota’s “stop the line” fashion (“Everyone stop developing and look at this!”) people fix what is broken before taking one more step. These lead to trackable bugs and potentially, issues, however these are more likely to be pounced upon immediately because you do not want something icky in the soup, so to speak.
- Manual testing is what we have been calling Unit testing: poking at the GUI, verifying data is validated, things work from a UX/UI perspective, UAT (User Acceptance Testing), and others. UAT is the most important in my opinion. The vendor shows the Client functionality and it either passes or fails. You can have formal acceptance criteria (unscrupulous vendors mandate defined and exact Scope as well as the leverage inherent in highly detailed acceptance criteria). When I am representing a Client who interfaces with vendors, I try to steer clear of hard acceptance criteria and change orders and instead I lean towards reality where things change even if they are correct according to original spec. Clients have had bugs marked as fixed before anyone at the Client even checked or were able to check. This creates haze, and bugs are lost among larger
issues, and the excuse is valid, a natural extension of chaos. Vendors have professional responsibility, and this is not my opinion. A company I worked for lost a court case because they did not perform what the court considered part of their responsibility as a solutions provider: proper management of issues. That is why I stopped worked for development shops. Being honest and forthright costs consulting teams money because delivering something measurable that cannot remain the same but must pretend to hold shape will lead to change orders and cries of “OOS! (Out of Scope)” while defining the system in terms of what it has to do at a high level leaves room for change in ways that do not impact behavior. Maintenance contracts bring in a lot of revenue. - Finally, there is Tool-driven testing. This is load testing, security testing, and hardware testing. This is stuff that your point a tool at and let it crank away and come back with analysis. All projects have a front end that is connected to a back end (not ALL I guess) which is why separating the sites from SAP has always been something I have not understood and so Tool-driven testing is extremely important). Also in this group would be analytics, for SEO, SEM, E-commerce stats, etc).
So I would first recommend enforcing a real, proper, full lifecycle SDLC (Agile if the project is new but if the project is not new, something as close to what exists) and in doing that, most of the testing problems will fall away because Test-Driven Development would catch the silly bugs and if stakeholders change their mind, they will get to weigh the cost of changing their mind before altering something that works.
I would recommend an iterative (every 2 weeks or maybe 3) process where the developers are testing before they say their code is ready and the tasks/bugs that make up the body of work are both large and small but always high in value. That means, in part, developers have to know what they are building against and to have UAT feedback come back to them through a Product Owner (the singular person who approves the list of features and bugs that are coming in the next release and has final Product signoff). I would also mandate a versioning tool like Subversion properly where you do not create a fork until there is a release or spawn a separate effort that will not be merged again down the line. Without proper version control, you can track bugs and issues all day and it will be rendered an exercise in futility if the software is not versioned correctly. There is a marked inability to define what code is shared and what code is not. Some like to track the version of the software where they found the bug. I find that annoying, because we are moving forward, always.
Bugs: There are a billion tracking tools out there. I like any of the automated tools besides Trac (meant for developers, when you look at who winds up using it). The main things we want to capture as a team where vendors have an appropriate level of visibility into the open issues and the ability to assign items (again, keeping the power within the Client instead of the black box of “we understand and you will see it next build”) are present in all of them, by and large. Again, ask a friend. For bugs I think small teams can use Excel but like the workflow and quick views as well as processes that can be enforced by many bug tracking systems.
Identity Field: May not be visible to the user of an automated system, but if possible, I like to show it for purposes of quick reference.
What caused the bug? Evidence, like narratives, screenshots, are important and maybe even critical. If it cannot be reproduced, it should not be presented as a bug unless it cannot be reproduced on another machine but can be reproduced on a singular, or set of, machines. This would be the place for Brower type, version, OS, etc (if needed).
What can we call the bug? A summary of what is wrong.
Who owns it? If it is to make the entry in the tracker more informative, to gather more information, to approve in an approval queue, or to fix, this “owner” may be almost any type of team member. Product Owner will or should give the final approval that it is not active. I say accepted instead of “fixed” because we are going to do regression testing and the owner will be one of the aforementioned depending on if we are doing disaggregation because a bug is really several, holding for re-test, or any number of statuses. Also, the owner should be a group (such as “XYZ Software Development Consultants”) for XYZ to assign to “Jane Jones” who is a developer particularly adept good at fixing that sort of issue.
From here down, the Product Owner / PM / CSM / CSPO assigns the values
What is its Business Impact? From low to high and “showstopper” with numeric values attached to the single decimal point. I maintain that with every bug there is associated Business Value, Difficulty, and User Value. These things are not obvious to the tester but they are to the Product Owner. They combine, via simple math, to give a final value. The “showstopper” assignation is there to really propel the bug’s value into the stratosphere. If low is 1 and high is 3, showstopper is 100. How severely will this impact the business if it is not resolved by go-live?
What is it’s User Impact? Like Business Impact in calculation, but answers the question: is this just annoying, or does it require a disturbingly complex UX?
How hard will it be to fix? This is often a tie-breaker between bugs when creating the next iteration of work and is also potentially an indicator of weaknesses in the team. I like it when these are estimates or are story points.
Status: Unassigned, assigned, UAT, accepted. You can add or remove as you choose but I like this foundation. When in UAT, it is waiting for Client approval. These should be no bugs that depend on other bugs if possible, and if a bug can be broken into separate bugs, they should be. Open is vague and not required. If it is assigned to someone or a group, it is not closed.
It makes sense to run through bug priorities as time allows to be sure that their values have not changed. If functionality has been cut out, those bugs will have less Impact. However, one bug does not lose value simply because 8 other high value bugs have been found. New functionality should *not* be added until a development team that is constrained by resources can catch up. It is better to have 50 percent of the features complete with 50 percent buggy that 100 percent of the features unusable. Maybe the System cannot be used with 50 percent functionality, but you have value in your pocket there and have paid for something that can be moved to another vendor, another release, or repurposed/sold. Always try to retain rights to and a copy of your source code in your contracts involving custom development. Otherwise, with a new vendor, you are better off starting over. Earned Value will be nil. This is only one of many reasons why “percent complete” is a false indication of anything. Others include that the task/bug changes, has facets revealed that were hidden before, and are as apt to change as anything else.
When a bug is introduced, anyone can introduce it. The default owner should be the Project Manger / Coordinator / Technical Liaison, or whatever the role is called. The Product Owner may also do this but are better off using their time in other ways. This is where the traditional Project Manager can breathe a little sigh of relief amongst all the talk that Agile and Scrum are making the PM role irrelevant. The Project Manager, in a technical setting, should have knowledge of technical systems. They can and should work with the Product Manager to tune value and assign bugs or issues after they are introduced into the System. However, the project manager does not assign to a person. People come and go. They get hit by the proverbial bus. Bugs and tasks are assigned to groups, such as “Development” and then divvied up within that group (hopefully using the same tool) so people are mapped to bugs and there is a face at the standup who will be speaking about it. Also, much as we get a sense of project velocity, we can get personal velocity. I say that and bite my lip, because there is always the “X factor” leaning on everything we do and using pure metrics to ascertain the value of a person is bad, bad, bad, and horrible. This, again, is where the classic project manager could be knowledgeable. You may notice that I am mixing and matching Scrum terms, Agile terms, and more common terms to describe roles. That is not without reason. There is a basic methodology-independent rationale here: keep track of things, limit dependencies, keep visibility and communication high, and stay flexible. Look at what work there is to do and compare it to your deadline instead of trying to shove 10 pounds of tasks into a 5 pound box.
For now, I would propose a bug tracking system with
- Reported By
- Date Reported
- Date Due (optional, more in regard to critical path issues or business commitments made outside planning sessions, etc)
- Owner (defaulting to the PM/Technical Liaison)
- Summary/Description
- Acceptance Criteria
- Supporting Documentation (documents with screenshots, etc)
- Site/s
- Business Value (1-5 or 100 with one decimal place)
- User Value (1-5 or 100 with one decimal place)
- Technical Difficulty (1-5 or 100 with one decimal place | high numbers mean more risk).
- Status
- Date of Status Change (so if something lies around for a week, for instance, it is obvious).
- Date Needed (usually driven by a business need or requirement)
I am sometimes a fan of multipliers, like taking the Technical Difficulty field and multiplying it by 1.2 if the team is distributed, contractors, etc. When appropriate. You will know; when you feel like “Yeah, but…” think of multipliers.
I think when the bug is initially entered, the individual reporting it enters the first 7 fields, (maybe not #3) with 8 and 9 and maybe 3 being a Product Owner field, and 10 and being a development field (supplied in an estimate but entered by an informed PM). The initial owner is the PM or Technical Manager (I really just do not like being called a Project Manager because of the connotations and the fact that it does not describe what I do, so know this is a personal preference, please). 11 and 12 are not restricted but can only be moved to “Ready for UAT” by the vendor and “accepted” once tested. This is of course, very much and almost mandated to be customized to your needs.
Best,
Josh
This is a watchdog. Cute, eh?
I don’t really think so either. Looks constipated.












