Re: AV/FW Adoption Sudies

[Eric Rescorla <ekr () rtfm com>]

Valdis.Kletnieks () vt edu writes:

> On Thu, 10 Jun 2004 12:23:42 PDT, Eric Rescorla said:
>
> > I'm not sure we disagree. All I was saying was that I don't
> > think we have a good reason to believe that the average bug
> > found independently by a white hat is already known to a
> > black hat. Do you disagree?
>
> Actually, yes.
>
> Non-obvious bugs (ones with a non 100% chance of being spotted on
> careful examination) will often be found by both groups.  Let's say
> we have a bug that has a 0.5% chance of being found at any given
> attempt to find it.  Now take 100 white hats and 100 black hats -
> compute the likelyhood that at least 1 attempt in either group finds
> it (I figure it as some 39% (1 - (0.995^100)).  For bonus points,
> extend a bit further, and make multiple series of attempts, and
> compute the probability that for any given pair of 100 attempts,
> exactly one finds it, or neither finds it, or both find it.  And it
> turns out that for that 39% chance, 16% of the time both groups will
> find it, 36% of the time exactly one will find it, and 48% of the
> time *neither* will find it.

The problem with this a priori analysis is that it predicts an
incredibly high probability that any given bug will be found by white
hats. However, in practice, we know that bugs persist for years
without being found, so we know that that probability as a function of
time must actually be quite low. Otherwise, we wouldn't see the data
we actually see, which is a more or less constant stream of bugs
at a steady rate. 

On the other hand, if the probability that a given bug will be
found is low [0], then the chance that when you find a bug it
will also be found by someone else is correspondingly low.
         
-Ekr


[0] Note that this doesn't require that the chance of finding
any particular bug upon inspection of the code be very low
high, but merely that there not be very deep coverage of
any particular code section.