mailing list archives
Re: [PEN-TEST] Evaluating Auditors Abilities
From: Deri Jones <Deri.Jones () NTA-MONITOR COM>
Date: Fri, 8 Sep 2000 09:40:28 +0100
At 10:56 07/09/00 -0700, Mark wrote:
1. Management got ripped off by what you describe
Yup! We've heard stories like this so often - poor testing at inflated
from 'big name' suppliers... It's the norm, not the exception, I would say
2. Many auditing firms unless they present their credentials will just run
the typical commercially available toolsuite plus a couple of hobbled
together tools, produce a nice report and never validate the results
9. No there are no certifications or Industry Groups that monitor or
endorse the auditors.
Actually this is not quite true - in Europe, the UK Government seems to be
unique in having a certification scheme for testers - maned by the govt
body 'Communications Electronics Security Group' (www.cesg.gov.uk) - the
scheme is called the CHECK scheme.
We were one of the founding members back in January 1999.
(Here at NTA Monitor we do more pen testing than any other company in
Europe (eg over 520 test assignments in 1999, over 200 clients on current
quarterly or monthly test contracts - from all 5 continents).)
Now whether the CHECK scheme is a high enough quality standard- humm?... It
was quite a debate in the set-up phase - just how high to set the bar. We
think it's too low...but we're just one voice...
But, using a CHECK member *does* provide some level of assurance.
But the *most* important thing you can do
- is ask the vendor for a *list* of customers - if they can't show they've
done work for say 30 or 50 companies - then have they really got the
experience? Once you've got the list, you can select the references *you*
want to speak to - not just call the 2 or 3 names the vendor offers - it's
easy to have 2 happt customers!
One thing that differentiaites us from the 'Big 5's of this, is we have
customer lists on our web site and in our literature - and we invite
new prospects to choice their choice of who they want to call.
Maybe that's why we are picking up a rush of new clients who are dropping
the Big 5..., and the like :<)
ANyone with enough money and political saavy can
open up shop (whether you are name or not), invest some money in a fancy
web site, claim to have all the vulnerabilities and exploits, and provide
cruddy service, but are backed by large VC..
At 12:46 AM 9/7/00 -0400, Derrick wrote:
Recently I underwent something that had me thinking about
companies and others (Big accounting firms that offer a side service of
auditing). Management decided that we needed to be audited by an outside
firm, which I am in full favor of. The problem came about in what an
un-named auditor did. Firewalls tend to cause false positives in some tests
and other anomalies that many auditors may not be aware of. So they
performed this audit which we did pick up and were aware of. What happened
next is what baffles me. The auditors did not understand the results that
nmap and other tools gave them. Near the end of the business day they
contact management proclaiming they have found numerous security issues and
even some backdoors in our network. After a long couple of days of testing
we found none of these issues were correct, and we then spent many hours and
several meetings explaining that the firm hired didn't seem to know what
they were doing. Management made the default comment of "We are paying them
a lot so they must be right, fix these problems". After several days of
explaining why they results were wrong and verifying the network we came out
to show that the auditors did in fact improperly interpret the results.
The end result is management walks away wondering if they got
ripped off or
if we were just trying to cover problems. It also caused a lot of overtime
and extra work for us to explain and prove the network to management. So the
end questions are these.
How can companies decide which auditors really do a decent job and are worth
their value ?
Are there any certifications or Industry groups out there or on the horizon
that will evaluate and endorse auditors ?
What is the best approach from a Network Admin position to counter end
results delivered by auditors if they seem to be in error ?
Has anyone else been through this, and is destined to get worse before
getting better ?
Thanks for any thoughts or comments,