[techtalk] Testing

[Malcolm Tredinnick]

On Wed, Nov 22, 2000 at 09:09:53AM +1100, Mary Gardiner wrote:
> On Tue, Nov 21, 2000 at 01:25:38PM -0500, Laurel Fan wrote:
> <snip open source scenario>
> > 
> > What factors would you take into consideration, and what
> > would your plan look like?
> 
> And where do you concentrate your energies?
> 
> Do you subscribe to the 10% of the code == 90% of the bugs thing and look
> in the areas where known bugs are encountered, or do you think, we've been
> over that with a finetooth comb, let's look here where we haven't seen
> many bugs?

My day job involves developing centralised systems that talk to banks
and stock exchanges and the like. They are meant to be up 24x7 (a
certain amount of redundancy is involved) and some certification by the
bank or stock exchange is always involved before rolling out code for
customers. Our testing policies tend to focus on two areas:

(1) Trying to ensure with some degree of confidence that each individual
code component behaves in an understood fashion. These are basically
functional tests that are written by the original developers of each
component and demonstrate how it is used and what it can do.

(2) End-to-end testing that simulate typical customer behaviour. If bugs
exist in the software (and they always do), we would prefer that they
lived in areas that are not exercised very often. Short of having
everything work, the next best thing is to have the 95% case working, so
we make sure that the common paths get exercised vigorously.

If bugs are found in the first case, they are fixed in the individual
area of code that contains them. Because of a large amount of common
code reuse, bug fixes in things like networking code and error
reporting, monitoring, etc, tends to fix the same bug in a few places
all at once. Bugs in non-common code are, almost by definition in our
case, going to be localised to the specific component. Sometimes a bug
in one piece of code is surrounded by many other bugs because a
developer is having a bad day or doesn't understand something, but this
doesn't seem to happen as rule -- possibly because we do a fair amount
of programming in pairs and there's a lot of talking about approaches to
coding problems going on between everybody.

In the second case, the bugs are serious (from a business point of view)
and often result in somebody thinking about whether the same problem
occurs in some other sub-system. This style of testing is really looking
at the interaction between lots of different components, so it's not too
hard to initially get it completely wrong and it also happens that
sometimes the same misconception is being implemented somewhere else at
the same time.

So how does this relate to Mary's original question? Well, with
reasonable test suites and lots of logging in production, we are never
really left in the case of scratching our heads wondering where the bug
occurred or what happened. This means we usually only have to think hard
about how to fix it. This is partly because the systems I work with are
under our control and we can go in an look at what's happening and roll
out new code to fix things on our own time schedule.

Unfortunately, this is much longer than I originally intended. If this
thread continues, I may post some thoughts on the opposite side of the
coin -- debugging Open Source stuff from user-filed bug reports ...
which is something I tend to do to while away the evening hours. My
experiences are quite different in that department.

Cheers,
Malcolm

-- 
Malcolm Tredinnick            email: malcolm at commsecure.com.au
CommSecure Pty Ltd