Archive: August 6, 2008

<<< August 5, 2008

Home

August 7, 2008 >>>

predicting bugs

Wednesday,  08/06/08  11:35 PM

Today I spent considerable time on a problem that confronts all software developers: how can you predict bugs? I don't actually need to predict specific bugs, I need to predict how many bugs I'm going to find.  This comes up when you're in the middle of testing something; you are some percentage of the way through testing, and you have found some percentage of the total number of bugs you're going to find.  If you could predict how many more bugs you're going to find, and you know about how long each bug takes to fix, then you can predict when you'll be done.  There is some of estimating involved, but it is better than holding your finger up in the air and taking a guess! So what is the relationship between "% test coverage" and "% bugs left to find"?  At first you might think this is linear, but it isn't; you find way more bugs in the beginning than you do at the end.  This is mostly because many bugs are "global"; you will encounter them regardless of what part of the software you're testing.  Installer bugs, for example, or bugs which keep you from accessing a database, or user signon bugs.  As you get to the end of testing, you are finding bugs which only afflict a small number of things, or just one thing. After playing with real data and bouncing "what if" scenarios off real developers and real QA engineers, here's what I came up with: This is an inverse logarithmic relationship; as the test coverage increases, the percentage of bugs left to be found decreases, asymptotically approaching zero.  After you've tested 10% of the software, 80% of the bugs remain, after test 25%, 50% remain, after testing 50%, 20% remain, and after testing 75%, 8% remain.  This "feels" right, at least in my experience (and in comparing to actual data), but of course it could vary significantly for your team :) Here's the actual equation: Here t is the "test coverage", and f is the "resulting bugs left to find".  The parameters a and b adjust the equation, for my purposes I determined a = 2.75 and b = 0.05, which yield the graph shown above.  Here's what that looks like in Excel: =POWER(2,-LOG(1/(1-(t/100)),2)*a)+b To use this, you have to substitute the t, a, and  b with the cell addresses which contain these values. Having done all this, it turned out to be rather useful; I could apply this to each area of software in a release that we're in the middle of testing, and predict how many more bugs we're going to find, and hence, when we'll be done!  It might not be right - that remains to be seen - but it feels better than just guessing :)

 

Wednesday,  08/06/08  11:41 PM

Today I nerded out from dawn to dusk, with no time for anything else; between various meetings and demos, predicting bugs, and working on code I didn't eat, ride, or do anything...  by 7:30 I was toast.  Driving home from Vista I stopped in Dana Point and had dinner at the Charthouse, which revived me wonderfully; nothing like a rare prime rib and a nice Pinot to recharge the batteries.  Still pretty wiped out though...  so what's happening?  Well... So are times really that bad?  My personal theory is that yes, there are problems, but nothing out of the ordinary.  However the media companies are experiencing huge challenges (newspapers and television networks), and therefore the people who work there tend to think everyone is having the same challenges.  For example, check this out: down economy spares no one, including the rich.  "It may have taken longer and it may not be as acute, but there are early hints that the economic slump is crimping the lifestyles of the wealthy."  Note the careful language, a telltale sign the reporter is manufacturing a story to fit a theory.  And sure enough this story has zero facts, just statements from individuals using words like "suggest".   My own view is that the mainstream media are doomed, especially since they've stopped running factual stories and have resorted to this sort of piffle. BTW as a counterpoint, here's an interesting factual article on CNN about married women who choose not to work.  Apparently the economy is sparing them :) And perhaps related, here's a cool review of the first class cabin in an Emirates A380 airplane.  This seems to be designed for rich people; apparently the economy is sparing some of them :)  Times really are that bad for Obama, however; he is sinking in the polls, which now have him tied with McCain.  I'm actually surprised, but maybe I should give the American people more credit.  Obama's supporters are starting to worry; check out why Obama can't win, posted on the Huffington Post, a typically liberal group blog, and Pundits beginning to worry about Obama. Blogging.LA helpfully reminds everyone to register to vote.  Wait, stop the presses!  Don't do it!  I am a contrarian on this, if people don't want to vote, let them.  Why have people who are not motivated to vote dilute the votes of those who are?  Gerard Vanderleun links the great Paris Hilton "response" to McCain's ad which featured her; looks like he's planning to vote!  You know, despite all the bad press and weird behavior, I like Paris, she seems smart and she has her head screwed on the right way (as opposed to, say, Britney Spears, also featured in the McCain ad, who doesn't).  View this video and listen to her describe her "energy platform", and tell me you don't agree...  BTW her thoughts on energy seem to make more sense than Obama's plan to inflate everyone's tires. This is pretty amazing, check out this chart, showing the T-shirt size distribution at Linux Symposium.  Wonder if the same thing is happening at JavaOne, or Microsoft's PDC?  Is there a correlation between the size of the programmers and the size of their code?  Inquiring minds want to know :)  Timing is everything - as noted the other day SpaceX flight 3 failed to reach orbit, and they've posted a detailed explanation of what happened.  I find it extraordinary how transparent they're being about their issues and what they're doing about them.  Good for them!  Finally, I love this; an article from the early 60s about the 505 class.  The boat is completely recognizable, despite all the high-tech innovation which has occurred in the intervening 50 years...  Check out the crew's hat!

Return to the archive.