Home page logo

pen-test logo Penetration Testing mailing list archives

Re: Verify Your Security Provider -- The truth behind manual testing.
From: Aarón Mizrachi <unmanarc () gmail com>
Date: Sat, 18 Jul 2009 00:57:29 -0430

On Jueves 16 Julio 2009 11:06:05 Adriel T. Desautels escribió:
A recent blog entry that we thought "some" of you pen-testers might
find interesting.  Feel free to leave comments on the blog:

Direct URL:

Verify Your Security Provider -- The truth behind manual testing.
Something that I’ve been preaching for a while is that automated
vulnerability scanners do not produce quality results and as such
shouldn’t be relied on for penetration tests or vulnerability
assessments. I’ve been telling people that they should look for a
security company that offers manual testing, not just automated scans.
The price points for quality work will be significantly higher, but in
the end the value is much greater. After all the cost in damages of a
single successful compromise is far greater than the cost of the best
possible security services.

I’ve noticed that there are a bunch of vendors who claim to be
performing manual testing. But when I dig into their methodologies
their manual testing isn’t real manual testing at all, its just
vetting of automated scanner results or testing based on the results.
In other words they test on what the automated scanner reports and
don’t do any real manual discovery. I’m not saying that tools like
nessus (an automated scanner) don’t have their place, I’m just saying
that they aren’t going to protect you from the bad guys. If you want
to be protected from the threat, you need to be tested at a level that
is a few notches higher than the threat that you are likely to face in
the real world.

Agree.  Nessus is a complement on pentesting. I have to say that is a very 
good complement :P. But have a lot of false negatives and false positives that 
need to be confirmed by hand.

This is akin to how the Department of Defense tests the armor on its
tanks, and I’ve probably mentioned this before somewhere on the blog.
But, we don’t test our tanks against fire from bb guns and .22 caliber
pistols. If we did that they wouldn’t be very effective in war.We test
the tanks against a threat that is a few levels higher in intensity
than what they are likely to face in the real world. As a result, the
tank can withstand most threats and is a very effective weapon. Doing
anything less isn’t going to protect you when the threat tries to
align with your risks; you’ll end up being an expensive casualty of war.

So why do some security companies test at this lesser level? Its
simple really, they are in the business of making money and care more
about that then they do about actually protecting their customer’s
infrastructure. Additionally, there is a market for that sort of low
quality testing. There are some businesses that don’t actually care
about their security posture; they just care about passing the test so
that they can put a check in their compliancy box. Then there are
other businesses that unknowingly get taken advantage by of vendors
because they don’t know the difference between high quality and low
quality services.

So what is the difference between high quality and low quality? From a
high level perspective it’s the difference between real manual
research based security testing or not. Once hackers have access, they
can do anything to your data from steal it, to install back door
technology in your product's source code. Its happened before, and its
going to happen again.

Auditory and even pentesting is not like testing tank armors. Here on infosec, 
we don't have a linear cannon, that .22 caliber are the small one and .50 with 
depleted uranium are the big one.

infosec audit are like sets:

you have a lot of vulnerabilities present on your system:

- Some vulns are well known (finite number)
- Some vulns are well known by pirates in the black market. (finite number)
- Some vulns are undiscovered yet (unknown number, assume a lot, possibly 

And from the set of vulns you have:

- Exploitable vulns 
  - Could be non-exploitable by a patch or time in the near future
  - Could be exploitable forever
- Non Exploitable vulns 
  - Could be non-exploitable forever
  - Could be exploitable in a near future when you change a condition

From vulns well known by pirates in black market you have two subsets:

- You are a target
- You aren't.

And also, you can place every vuln in a rank of severity. (ex.)

0. Very low compromise (Acceptable)
1. Low compromise
2. mid compromise
3. high compromise
4. severe compromise


----------------------- (and make your mix's)

And all of this, is needed to be mixed with social engineering and sequence of 
exploitable bugs to do effective escalation. 


Could be complex, but the main goal of this audit process is:

- Vulns that are well known: patch it.
- Others (undiscovered or well known by pirates): the goal is try to move this 
set to non-exploitable branch permanently using different strategies. 


I'm a pentester, but i have to say that pentest is only the first stage when 
you show the impact and risk of an attack to justify a more extensive and 
white box based security plan. 

Good pentesters could define a security strategy based on his observations, not 
only a patching guide, it could work _if you don't have enough time and money 
to spend on security_. 

Best pentesters could also define security policies that protect you against 
future bugs.

Follow the best practices, then, consider every software as a potential 
vulnerability and trace the impact... Then, make your perimeter based on your 
study, Then, make and follow a maintenance plan. And finally, you could 
initiate a research on every software that you have, starting with homemade 

When a company tells you that they perform manual testing hold their
feet to the fire. You can do the following things to verify it:

      • Dig into their methodology and ask them specific questions about
how they perform their testing. (See our white papers on how to do
      • Don’t swallow jargon and terms that sound cool and don’t mean
anything, use Wikipedia to look up the terms and make sure that they
make sense.
good idea!

      • Ask them for the names of their security experts and then use tools
like Google, LinkedIn, Facebook and PIPL to do research on those
experts. If nothing comes up then chances are their experts aren’t
experts at all.
I disagree.

First of all, we need to differentiate the expert concept and the ethical 
concept. A hacker could be ethical or not, but is a hacker (expert). Public 
content could be released with a pseudonym for several reasons:

- He don't want to be identified (Several reasons nested...)... ex. some 
truecrypt programmers (correct me if i'm wrong). Do you believe that truecrypt 
are not an ethical work?
- He don't want to be identified (Part 2): Some states or countries consider 
illegal many information released by researchers (from bugs, even if you 
follow the ethical disclosure process, to some software like tor). Then, 
anonymity is required.
- He is not ethical or is a reformed hacker.

Secondly, an example, i can work for "123 anonymous auditors", an intermediate 
company,  BUT... due to NDA, a serious auditor haven't public information 
about his final clients.

I know. It is very difficult task to trace a security expert. There are two main 
well known strategies:

- A private meeting with the expert. 
- Certifications
- public information (that is not determinant)

      • Search vulnerability databases like milw0rm, securityfocus,sirtfr,
secunia, packetstormsecurity, etc. for the vendor’s name to see if
they have research capabilities. If you don’t get anything in return
then chances are that they don’t have research capabilities. If that’s
the case then how do you expect them to perform quality manual
testing? Chances are that they won’t be able to.

Testing/Vuln research and pentesting are not the same, and not the same skills 
are required.

I know a many experts on ASM debugging with no idea in privilege escalation in 
a network environment.

Also, you need to be focused in the security strategy (big picture) when you 
are doing infosec audit, not only in unknown bug discovery. Infosec is an 
statistical process, not a deterministic one. 

Well, i cannot describe how important is protect the network against common 
bugs rather than unknown bugs. And also, i can not describe how important is 
to have a security strategy. .

Remember that unknown bugs are not in many hands and most of bugs work only in 
certain conditions (usually when you don't follow the best practices on 

Remember you’re putting the integrity of your business and its
respective name into their hands.

      Adriel T. Desautels
      ad_lists () netragard com

      Subscribe to our blog

This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can
actually do a proper penetration test. IACRB CPT and CEPT certs require a
full practical examination in order to become certified.


Ing. Aaron G. Mizrachi P.    

Mobil 1: + 58 416-6143543
Mobil 2: + 58 424-2412503
BBPIN: 0x 247066C1

Attachment: signature.asc
Description: This is a digitally signed message part.

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]