Penetration Testing: we are security critics was: Re: Using 0days as part of pen-test?

[Pete Herzog <lists () isecom org>]

Hi,

> 0-day tests, by definition, cannot test anything other than the 
> quality of the anomaly-based detection system. 
I'm unaware of this definition and I think it may be flawed. A 0-day, 
we can say is an undocumented vulnerability and as such not widely 
known. Without thorough testing, we err on the side that every piece 
of software may have undocumented vulnerabilities which may or may not 
currently be known to someone. That means we must configure our 
systems, define our networks, and control our environments so that we 
can be sure that even the vulnerabilities we did not know to 
specifically protect against are protected against. Which is what 
makes 0-days in pen-tests so important. 0-days are a device to prove 
that a client is unready to handle the unknown whether it be to spot 
it, stop it, or minimize its impact.  It is also the reason why 
controlling the environment is so vital in security, much more so than 
patching or blocking any specific vulnerability or vulnerability 
class. It's also the whole point to the Hacking Exposed Linux 3 book 
we released which too many in the security industry sadly missed. 
When we test security (pen-test, vuln scan, sec test, red team, etc.) 
we are acting as security critics who are looking to see if security 
works in this particular instance and if it will stand the test of 
time. Like a movie critic, they don't care if the plot is formulaic, 
the characters are flat, or the production value is cheap, as long as 
it works for this particular movie and then they communicate 
effectively what does and does not work now and in the future. To do 
this, we need to look at how well they separate the asset from the 
threat and control the environment to minimize the effectiveness of 
the threat. To do that, we need to test classes or types of 
vulnerabilities which includes those that are new to the environment. 
Since specific attacks change over time, classes of attack are rarely 
new. Considering that there are 10 controls which can mitigate all 
possible attacks, we need to test against the controls and not the 
specific vulnerability. This way we can test preparedness as well as 
control effectiveness. There are even better tools coming out all the 
time to do this but if the tester can't recognize that the controls 
work, how they work, or even what is the attack surface then no tool 
will help them communicate the problem effectively and the right 
control won't be applied.  We see this a lot when a report is made 
that says, you have vulnerability X, see CVE X, apply patch X. It says 
nothing about controls in place that can mitigate the attack and make 
it unnecessary to patch. If they knew this, they could make sure all 
their systems have those controls to protect them.
I've already gone on too far but this comes back to people still 
choosing the witness protection style of security over the prison 
style. Witness Protection works fine as long as the user follows the 
rules. That can't be guaranteed even if the user is a faithful 
employee not a random web user. But it will work until there's a 
mistake or an accident.  The Prison style assumes the rules won't be 
followed, by prisoners, visitors, or by guards. So they define 
controls for the environment so that things run smoothly and when a 
mistake or accident occurs, it can be quickly and efficiently dealt 
with.  As testers, we need to make clients aware what they're leaving 
to chance and let it go on record. This way they can accept risks that 
they know about.
Sincerely,
-pete.

OPST, OPSA, OWSE, OPSE
Hacking Exposed Linux 3: http://www.isecom.org/hel