Re: [PEN-TEST] Evaluating Auditors Abilities

["Teicher, Mark" <mark.teicher () NETWORKICE COM>]

1. Management got ripped off by what you describe
2. Many auditing firms unless they present their credentials will just run
the typical commercially available toolsuite plus a couple of hobbled
together tools, produce a nice report and never validate the results
3. A majority of the auditing companies use the security audit as a trick
to get the follow on work.  If we find something, we can also fix it.
4. Hiring an outside firm is more than hiring some Big Auditing firm it is
a much different approach, you want research the company that you are
hiring and interview the candidates that will be doing the assessment, not
just the sales/SE that come onsite and do the demo type thing..
5. Check the liability and legal wording of the contract, there is usually
a lot of verbage in there.
6. Remember a security audit is a snapshot of an organization's security
stance over a couple of days, not an overall view of the network
architecture, policies, or the business model analysis

7. In my mind, none of them do an outstanding job of performing thorough
security audits, many of them claim they do, but how many of them are
really up to snuff.

8. Prior to the report presentation, ask for a copy of the data, so that
you can go through the data yourself, and validate the findings, ask for a
list of tools they used (should be provided on the last page of the
report). and replicate some of the tests, if your results come out
different, than counter the final report

9. No there are no certifications or Industry Groups that monitor or
endorse the auditors.  ANyone with enough money and political saavy can
open up shop (whether you are name or not), invest some money in a fancy
web site, claim to have all the vulnerabilities and exploits, and provide
cruddy service, but are backed by large VC..

/mark

At 12:46 AM 9/7/00 -0400, Derrick wrote:
> Dear Pen-Testers,
>
>         Recently I underwent something that had me thinking about
> Security Auditing
> companies and others (Big accounting firms that offer a side service of
> auditing). Management decided that we needed to be audited by an outside
> firm, which I am in full favor of. The problem came about in what an
> un-named auditor did. Firewalls tend to cause false positives in some tests
> and other anomalies that many auditors may not be aware of. So they
> performed this audit which we did pick up and were aware of. What happened
> next is what baffles me. The auditors did not understand the results that
> nmap and other tools gave them. Near the end of the business day they
> contact management proclaiming they have found numerous security issues and
> even some backdoors in our network. After a long couple of days of testing
> we found none of these issues were correct, and we then spent many hours and
> several meetings explaining that the firm hired didn't seem to know what
> they were doing. Management made the default comment of "We are paying them
> a lot so they must be right, fix these problems". After several days of
> explaining why they results were wrong and verifying the network we came out
> to show that the auditors did in fact improperly interpret the results.
>         The end result is management walks away wondering if they got
> ripped off or
> if we were just trying to cover problems. It also caused a lot of overtime
> and extra work for us to explain and prove the network to management. So the
> end questions are these.
>
> How can companies decide which auditors really do a decent job and are worth
> their value ?
> Are there any certifications or Industry groups out there or on the horizon
> that will evaluate and endorse auditors ?
> What is the best approach from a Network Admin position to counter end
> results delivered by auditors if they seem to be in error ?
> Has anyone else been through this, and is destined to get worse before
> getting better ?
>
> Thanks for any thoughts or comments,
> Derrick