Security Basics: Re: Vulnerability scanners don't work

["Adriel T. Desautels" <ad_lists () netragard com>]

Good point, I don't think that I was as clear as I could have been.   
The truth is that vulnerability scanners do contain signatures or  
scripts that allow them to hunt for certain types of vulnerabilities  
as well as the specific known vulnerabilities.  But are you saying  
that they can actually identify new vulnerabilities? I'm still saying  
that they can't.
Lets take your www_too_long_auth.nasl script into consideration only  
because it is the first one that I noticed.  That script just connects  
to a web service and blindly dumps a 2048 bit payload into the  
authorization buffer. If the service stops responding then the script  
tells the scanner that the service is vulnerable, but is it? If the  
service keeps responding then the script tells the scanner that the  
service is not vulnerable, how accurate is that? Would you consider  
that to be positive vulnerability identification? Can we be certain  
that the scripts are finding real, exploitable conditions and not  
false positives?
Sure they might be able to identify a problem that might be a  
vulnerability via the ad-hoc perl -e style testing, but in my opinion  
thats not good enough. That is not a positive identification of a new  
vulnerability.  That is the identification of a theoretical  
vulnerability which isn't technically a real vulnerability until its  
been proved by a human, right?
So is this inaccurate, or just unclear:

"The fact is that vulnerability scanners can not detect  
vulnerabilities unless someone has first identified the vulnerability  
and created a signature for its detection."
Perhaps I should write:

"The fact is that vulnerability scanners can not positively identify  
vulnerabilities."
I think that "what's best" is a major part of the problem. Most people  
don't know the difference between a vulnerability scan and a manual  
vulnerability assessment.  Most people think that they are both the  
same thing, same quality, etc. Thats an advantage for the  
vulnerability scan vendor, but its a disadvantage to the people who  
don't know "what's best".
I'd like people to be able to make well informed decisions so that if  
they use a vulnerability scanner they know what they are really  
getting. The fact of the matter is that vulnerability scanners are an  
invaluable tool with respect to maintaining the security of a network  
and doing nightly checkups, but they are not nearly as accurate as the  
human teams. As a result, we recommend to our customers they perform  
vulnerability scans frequently and undergo intense manual penetration  
testing once or twice a year.
I might actually take consideration and write the article that you've  
suggested.
On Jan 8, 2009, at 6:03 AM, security curmudgeon wrote:

> On Wed, 7 Jan 2009, Adriel T. Desautels wrote:
>
> : Greetings all. I've finished another entry on our blog. This time  
> the
> : entry was about why vulnerability scanners do not work. It goes  
> into a
> : little bit of detail and is intended for the average reader. My  
> goal was
> : to help to educate people about what vulnerability scanning really  
> is.
> :
> : http://snosoft.blogspot.com/2009/01/vulnerability-scanning-doesnt-work.html
> :
> : As always, comments are more than welcome.
>
> Hi Adriel,
>
> I would disagree with at least part of what you wrote. I'm not sure  
> if you
> are making too broad of a generalization or not considering your  
> wording
> carefully. For example:
>
>   "The fact is that vulnerability scanners can not detect  
> vulnerabilities
>    unless someone has first identified the vulnerability and created a
>    signature for its detection."
>
> This is not fact, this is actually false. Consider two types of
> vulnerability scanners, both of which prove this wrong. First, take  
> a more
> network-centric vulnerability scanner like Nessus and look at some  
> of the
> plugins. While a bulk of them are 'signature' based like you mention,
> there are several plugins that are designed to look for general  
> problems
> in services such as smtp_overflows.nasl or any of the  
> www_too_long_*.nasl
> plugins. Second, consider a vulnerability scanner that is more
> application-centric such as AppScan. It will find custom  
> vulnerabilities
> in applications such as XSS, SQLi and more, as well as provide you  
> with
> the request/response and highlight the portions that indicate the  
> presence
> of the vulnerability.
>
> As many have said for years now, it isn't just a matter of "what's  
> best",
> it's a matter of "what's best for your org, right now, for the money  
> you
> will spend". In some cases, that is throwing a vulnerability scan  
> against
> a class B network, something that a pen-test shop can't do in a short
> amount of time or inexpensively. Other times it is hiring a quality
> pen-test shop to do a three week application test against one web  
> server
> running a custom banking application. I think the lesson that you  
> should
> be impressing upon readers is that they fully understand the  
> benefits and
> limitations of each method for conducting vulnerability scans, and  
> pick
> the one that serves their immediate needs.
>
> Overall your post is on par with the sentiment of many people in the
> industry, and something that many pen-testing shops try to explain to
> (potential) customers. Hopefully your next article goes into depth  
> on why
> a really good pen-test shop can still be quite limited and why they  
> still
> doesn't always find all of the vulnerabilities present =)
>
>
> security curmudgeon
>
>
> disclaimer: i've worked for the type of pen-test shop you describe for
> many years, and i currently work for a security product company that  
> makes
> a vulnerability scanner among other things. my opinions are my own.


        Adriel T. Desautels
        ad_lists () netragard com
        --------------------------------------

        Subscribe to our blog
        http://snosoft.blogspot.com