RE: Re: rules of engagement scope

["Omar A. Herrera" <omar.herrera () oissg org>]

Hi Nasty,

Just a few comments below...

> -----Original Message-----
> From: mr.nasty () ix netcom com [mailto:mr.nasty () ix netcom com]
> Sent: Tuesday, May 16, 2006 3:30 PM
> To: pen-test () securityfocus com
> Subject: Re: Re: rules of engagement scope
>
> Some of the pro ROE responses appear to have a serious disconnect between
> 'reality' and the seriousness of the subject.
>
> As far as a pen-test contract is concerned, I'd want to make sure that I
> get my money's worth.  Speaking from the standpoint of a taxpayer,
> shareholder or CEO.  Hence from this perspective I wouldn't want to see
> what I would consider WASTE.

Nice thoughts and I agree with the good intent in this play of words. Still,
the fact that other's people reality does not match yours is not proof that
they don't live in a reality themselves. Have you consider the slim
possibility that you might be an exception or that you might be
misinterpreting your own reality?

I'm not trying to discredit or offend you in any way, but from what I see
from your post I doubt you got all your facts straight. Bringing this
example of financial audits, instead of clarifying your opinion on the
uselessness of a ROE scope in a pentest leads me to believe that you don't
have a clear idea of the intent of security assessments within the context
of financial audits. Hence, it became an argument against your opinion
instead of supporting it.


> What on earth does this have to do with PEN-TESTING?  I'm an AUDITOR, just
> like a MARINE, you are never and ex-MARINE, you are never an ex-AUDITOR!
> I currently work as an ISO for a large organization who oversees PEN-TESTS
> in my organization.  When these folk visit a site and perform their tests,
> I want them to find the low hanging fruit.  Then I don't just want them to
> take screen shots I want them to leave behind a gift, a worm in the apple.
> (Not a Morris worm - it's a euphemism)

Well, I've been an auditor myself in the past for one of the remaining big 4
(doing security assessments in support of financial audits, started as
consultant, then Sr. consultant and finally as manager) and I'm not
convinced that you perception is at all correct.

If you are referring to information security people that do assessments
during a financial audit (brought in by the auditors) then their job is
definitely not what you say. They are there to support the financial
auditors, not to find the low hanging fruit. If you want this then simply
hire a pentest team for this specific purpose. 

Sometimes (many times actually) these firms include additional security
tests (or a wider assessment scope) that looks like a pentest as an "added
value" to their customers, but that's it (and it may be one of the reasons
of this misconception). They will rarely meet your requirements/expectations
for a proper pentest, so you have to pay for one if you want it.

> Now how is all this related you ask?  Just like any organization there is
> a method and certain requirements that logically fall into place.  Before
> a financial auditor can perform any type of confidence testing on your
> internal controls or transactions they must be assured that the mechanism
> (the network - IT) in place is secure within a specific confidence level.
>  
> If however the organization dictates the methods of pen-tests to provide a
> favorable result without disclosure the financial auditors sample
> calculation will be wrong.  (We're not addressing the ROE of the financial
> auditors at this point.)
>  
> What do we mean by ROE of the pen-test?  That's probably the first step in
> addressing this question before it wanders off into 360 different
> directions.  In my experience I've seen organizations dictate how they
> want the pen-test done to the point of restricting the testers to a
> specific IP and to alert IDS prior to any testing.
>
> As a pen-tester myself I was given an edict, restricting me to not connect
> to the network, and not to touch a keyboard at the facility I was testing.
> Yet I was to perform a pen-test.  So how did I break in?  I thought like a
> hacker and social engineered my way right in front of the director, chief
> of security and my escort and took their sam file through locked doors and
> a "secure" network all within the confines of the letter.  But then that's
> because I'm good; another story for a later date.

Well, as a support for a financial auditor your mission, as far as I'm
concerned, is not to break in. And for that is why (at least where I used to
work) we hardly ever mentioned the term pentest (which no matter the
differences in the definitions, it does seem to be related to "breaking
in"). We used to call security tests in this context simply "security
assessments to support financial audits", and in this context the ROE scope
is not only important, it is essential.

If the client defines the scope of the assessment you have a problem. If the
assessor (or pentester in your case) defines the scope you also have a
problem. And if there is no scope at all, then you have a huge problem. It
is the financial auditor who, based on her/his audit plan, defines the scope
(and yes, this involves information provided by the client if there is no
previous knowledge of the systems supporting the financial transactions). 

Auditors don't simply guess which systems need to be verified, they should
know exactly. The purpose of a financial audit is not to assess IT security;
they call security assessors (or pentesters in your case) to give them an
evaluation of the security in place protecting the systems that support the
relevant financial functions and data. It is quite different.

> The point I'm trying to make here is that these tests (risk analysis,
> vulnerability tests, pen-test) are for a purpose and not in themselves a
> goal.  They are there to support the reliability of the information
> security of the organization through its financial statements.

Exactly. From the point of view of financial auditors, if you find that
security is lax, they will consider performing additional tests to ensure
the integrity of financial statements.

That's why you don't even need to break in for them to assess this. If you
find no controls at all or if you find things like a lack of audit trials or
shared passwords, they will consider doing additional testing because they
can't rely on the systems. Reliability is the key as you mention. Also you
might claim to have broken into a machine but that might be completely
irrelevant from the financial auditor's perspective if there is no risk to
financial functions and data. Maybe it is just an issue with the use of the
word pentest in this context, but you don't typically verify things like the
existence of backups during a pentest (in my opinion), and these things are
very important for a financial audit. 

Decades ago financial auditors were able to test all statements soundly.
With the amount of information and transactions (possible due to the
existence of information systems) manually testing everything is impossible
(they have time limits: fiscal year). So they do sampling and rotation, and
ask for an expert's opinion to decide if they can rely on the information
provided by some of the financial systems to do their tests.

This is precisely the reason why a scope is essential. Information security
support personnel must ensure that they cover at least the systems selected
by the financial auditors. Otherwise they will be misled in judging whether
additional testing of financial data is required or not. 

> Believe me no one (taxpayer or shareholder) is going to review the pen-
> test.  They rely on the financial statements.  Without full disclosure of
> this ROE within their financial statements this, in my opinion, is
> considered FRAUD, WASTE & ABUSE.  It is misleading to the financial audit
> and to the taxpayer and shareholders alike.

You are right about shareholders and taxpayers never going to review a
pentest in this context, and they don't need to. All the responsibility
rests on the auditor's shoulders once they sign their observations. 

Shareholders and the public just expect that the financial audits were done
in a professional manner and that they are correct. Implicitly, we know (or
hope) that financial auditors will rely on the expert advice from
information security professionals to guide their audits.
 
What good would do to include the scope of the security assessments done in
support of the financial audit? Will taxpayers and shareholders be able to
make anything out of it (it is rather technical in nature)? 

If they have doubts they will ask for the CISO's opinion which in turn might
ask for your opinion as an ISO. And we all know that someone within the
company (usually at some technical position, probably you and the CISO) does
get informed of what is being tested. 

Now, if you keep thinking of this assessment as being like any other pentest
and base your opinion on that, "that" opinion is probably going to be
misleading for your CEO and everyone else at the top who wants to get a
clue.

> Sorry to take so much bandwidth but I'm very sensitive to this.

That can be seen ;-), and I don't think it is bad that we get sometimes
passionate at what we do. But passion should not reign over reason, and
there are plenty of good reasons why clearly defined scopes are essential,
especially in the case of the financial audits that you mention.

Kind regards,

Omar Herrera


------------------------------------------------------------------------------
This List Sponsored by: Cenzic

Concerned about Web Application Security? 
Why not go with the #1 solution - Cenzic, the only one to win the Analyst's 
Choice Award from eWeek. As attacks through web applications continue to rise, 
you need to proactively protect your applications from hackers. Cenzic has the 
most comprehensive solutions to meet your application security penetration 
testing and vulnerability management needs. You have an option to go with a 
managed service (Cenzic ClickToSecure) or an enterprise software 
(Cenzic Hailstorm). Download FREE whitepaper on how a managed service can 
help you: http://www.cenzic.com/news_events/wpappsec.php 
And, now for a limited time we can do a FREE audit for you to confirm your 
results from other product. Contact us at request () cenzic com for details.
------------------------------------------------------------------------------