Penetration Testing: Re: Vulnerability scanners don't work

[security curmudgeon <jericho () attrition org>]

On Wed, 7 Jan 2009, Adriel T. Desautels wrote:

: Greetings all. I've finished another entry on our blog. This time the 
: entry was about why vulnerability scanners do not work. It goes into a 
: little bit of detail and is intended for the average reader. My goal was 
: to help to educate people about what vulnerability scanning really is.
: 
: http://snosoft.blogspot.com/2009/01/vulnerability-scanning-doesnt-work.html
: 
: As always, comments are more than welcome.

Hi Adriel,

I would disagree with at least part of what you wrote. I'm not sure if you 
are making too broad of a generalization or not considering your wording 
carefully. For example:

   "The fact is that vulnerability scanners can not detect vulnerabilities 
    unless someone has first identified the vulnerability and created a 
    signature for its detection."

This is not fact, this is actually false. Consider two types of 
vulnerability scanners, both of which prove this wrong. First, take a more 
network-centric vulnerability scanner like Nessus and look at some of the 
plugins. While a bulk of them are 'signature' based like you mention, 
there are several plugins that are designed to look for general problems 
in services such as smtp_overflows.nasl or any of the www_too_long_*.nasl 
plugins. Second, consider a vulnerability scanner that is more 
application-centric such as AppScan. It will find custom vulnerabilities 
in applications such as XSS, SQLi and more, as well as provide you with 
the request/response and highlight the portions that indicate the presence 
of the vulnerability.

As many have said for years now, it isn't just a matter of "what's best", 
it's a matter of "what's best for your org, right now, for the money you 
will spend". In some cases, that is throwing a vulnerability scan against 
a class B network, something that a pen-test shop can't do in a short 
amount of time or inexpensively. Other times it is hiring a quality 
pen-test shop to do a three week application test against one web server 
running a custom banking application. I think the lesson that you should 
be impressing upon readers is that they fully understand the benefits and 
limitations of each method for conducting vulnerability scans, and pick 
the one that serves their immediate needs.

Overall your post is on par with the sentiment of many people in the 
industry, and something that many pen-testing shops try to explain to 
(potential) customers. Hopefully your next article goes into depth on why 
a really good pen-test shop can still be quite limited and why they still 
doesn't always find all of the vulnerabilities present =)


security curmudgeon


disclaimer: i've worked for the type of pen-test shop you describe for 
many years, and i currently work for a security product company that makes 
a vulnerability scanner among other things. my opinions are my own.