Re: Evaluating Pen Testers

[Daniel Kennedy <danielkennedy74 () gmail com>]

I don't normally like to engage in back and forth over a mailing list,
but there is a tone of unwarranted dissmissiveness combined with
points that I think are incorrect or incomplete, and in reading your
usual comments I know you're better than that so I'm going to address
it (see below).

Forgive that I'm going to be pretty direct, as the argument over "what
is a good penetration test" needs some new points raised in it,
otherwise its the same tired discussion that I've been listening to
for years.

On Tue, Apr 13, 2010 at 12:49 AM, Andre Gironda <andreg () gmail com> wrote:
> On Thu, Apr 8, 2010 at 9:18 PM, Daniel Kennedy
> <danielkennedy74 () gmail com> wrote:
> > There ought to be a "who's who of penetration testers, especially with
> > some of what I read about and hear at conferences when it comes to
> > penetration testing, for many years now, and its not getting any
> > better. That said, it wouldn't be easy to put together. A firm in the
> > UK was testing pen testers for a while, but their approach left some
> > questions to be answered.
>
> Are you referring to CHECK? They are still verifying
> penetration-testing capability at the company-level.

Don't recall, it was a presentation years ago at a conference. Doesn't
matter, the point remains, as you point out in your next paragraph, it
would be arduously difficult to accomplish a list. Most firms that
would maintain it would do so with self interest (in other words they
would probably be security companies or consultants themselves)
calling their results into question.

> I strongly discourage anyone from building a list of individuals;
> Microsoft and others have tried this before and the ethical
> consequences of these actions is somewhat revolting (to at least
> myself). It would be impossible to keep a list current because people
> come and go all of the time (at least by the hour).

Good points, and MSFT is certainly not the entity to spear head such
an initiative given a strong self interest. Maybe the concept has to
reverse itself, like a better business bureau style implementation of
reporting charlatan penetration testers. There are too many out there,
I sympathize with the security consumer when trying to find someone
competent to perform a test.

> > #Confusion
> > Many customers, and many security testers, confuse what is a
> > vulnerability scan with a penetration test. A scan for vulnerabilities
>
> I do not like the words "manual" or "automated". Not all
> penetration-testing activity can be automated, but a lot can. Not all
> application scanning activity should be automated. Humans HAVE to be
> involved. Vulnerability scan activity should not be automated without
> humans, but Qualys QG has convinced people (read: managers) that it
> can.

Whether you like the terms or not is not really material. A good
penetration test likely has some automated tasks for time savings
(these are time boxed tests) and some hand, or manual, or custom
testing, whatever you'd like to call it. That said there are some
penetration testers out there who use no well known tools and are at
the top of the game.

We are in complete agreement that fully automated vulnerability
scanning is not effective and that human involvement (using
knowledgeable humans) is a key component in a successful vulnerability
management program.

> Penetration-testers MUST be allowed to choose their own toolchain and
> decide which parts to automate and which parts to leave with some
> manual intervention.

If it makes sense, then yes. With an in house team, there are all
kinds of company policies affecting the type of software that can be
used. But if your point is that a knowledgeable person must be
equipped with adequate tools that the person requests, then sure.

> > can be a recon activity in a pen test, its valuable information, but
> > its not a pen test. Pen tests involve exploitation (usually a
>
> Please read
> [PDF] http://www.securityacts.com/securityacts02.pdf
> [PDF] http://www.net-security.org/dl/insecure/INSECURE-Mag-25.pdf
> before continuing...

Not taking reading assignments that aren't linked as a reference to a
point ;) (although INSECURE is worth a read if anyone hasn't looked at
it).

> > non-damaging one like opening a shell or dropping a text file) reached
> > under some rules of engagement. This doesn't suggest one is better
> > than the other, frankly its completely dependent on what the client is
> > hoping to accomplish.
>
> I think I disagree with you. Penetration-testing can certainly stop
> before exploitation, assuming that something is found that is believed
> or known to be capable of exploitation.

What's happened with penetration testing is the same as what's
happened with many other terms and services in information security,
namely that people have redefined them to mean what they would like
the term to mean. You want this, I offer this service, let me find a
way to smash these two things together. I'm watching it now play out
with the term: "advanced persistent threat".

So, to me, I expect to see results in a penetration test that show, at
a high level, what was attempted, what is believed to be exploitable,
and what was exploited, with exploitation within the penetration
test's time frame being the end goal of the testing team.

What you're proposing could be interpreted as being handed a report of
possible vulnerabilities (an incomplete one at that since you're
stopping testing at something 'believed to be capable of
exploitation'. That's probably useful information, but for me not
useful enough to warrant spending money on a penetration test over
having someone do a vulnerability scan which will show me all possible
or believed routes of exploitation.

Actual exploitation, which involves finding a vulnerability or
chaining vulnerabilities together, in a custom environment, to achieve
a proof that a system can be exploited, is difficult and what I'm
looking to have be attempted in a penetration test.

> > #Standards
> > The OSSTMM is an interesting project but its miles from being a
> > standard where you can eliminate people that don't follow its
> > methodology. It would be akin to saying you only accept software from
> > CMM level 5 companies - the model is thorough but smart people raise
> > legitimate objections to it.
>
> The art of drafting a proper RFQ to potential penetration-testing
> consultants is a WIP by OWASP as seen here --
> http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers&discussionID=17288173&gid=36874&commentID=14543002
> Almost all RFQ analysis is followed by Case Study analysis and
> extremely high-quality References before hiring an application
> security consulting company.

To assume that all security companies/consultants are hired after a
thorough request for quotation process is an incorrect assumption. To
even assume that this is the best way to hire security consultants is
an arguable point.

> I recommend retaining in-house penetration-testing talent. I suggest
> letting them follow any penetration-testing methodology/standards
> which they choose, as long as it gets results. Most will choose all
> standards and no standards at the same time. Penetration-testers are
> very talented at being in two places at once given their paradoxical
> natures. Allow them to do this.

In house is valuable because you retain available talent and generally
can spend more time testing more things. That said it does not have
the economy of scale of hiring outside consulting help, and many
companies (especially in this economy) are not of the shape and scale
to justify maintaining a full time penetration testing team. Having a
single resource runs the risk of having that resource leave at any
time (no coverage overlap), and robs the penetration testing team of
the benefit of collaborating during testing (most testers are not
experts in every system or type of system they encounter). Having a
resource that does other things, but sometimes tries to do penetration
tests, leaves you with a party not fully committed or immersed in the
infosec industry doing your testing.

> > #Certification
> > Alongside things people present as standards are certification. They
> > tell you something about the person, namely that they are willing to
> > take the time/cost to prove some level of proficiency in an area, but
> > there are a great many security luminaries without any. The CEH is
> > gaining some traction, not sure if that's a good or bad thing yet.
>
> I think it's a bad thing. What does China use to certify their
> penetration-testing talent?

Not sure why what China is doing or not doing is important. But I
think what you're saying is along the lines of what I said: "there are
a great many security luminaries without any [certification]".
Certification has its place, but to get into it would be an entire
additional post, and it does have many shortcomings.

> If somebody has written some new code that steals some stuff that
> nobody has stolen before, then that should be certification enough. In
> other words, the first question in the interview should be "Which
> BackTrack tool did you write or contribute to?" and the second
> question should be "When was the last time that you spoke at an OWASP
> local chapter meeting?"

What is they're not a member of OWASP, or just don't want to speak at
OWASP meetings. OWASP is a great outfit, but not everyone is a member.
What if its not an application penetration test? What if they don't
use BackTrack (which is a great tool as an aside)?

Further you're making my point below for me, that a company or person
with verifiable talent is a better hire then one without.

> > #Nessus
> > As you say, one who runs a scan and hands you a Nessus report is not
> > doing much. However Nessus is a sophisticated tool for vulnerability
> > scanning, has a professional license model, and compares favorably to
> > more expensive options. So you can't eliminate someone for using
> > Nessus, only for only using Nessus.
> > #Open Source Tools
> > The suggestion that using open source tools reveals some lack of
> > sophistication or worthiness is silly. I would rather have someone
> > capable of making contributions to the Metasploit project, someone who
> > understands what they're running and can do hand testing, then some
> > bozo who just points Core Impact at my environment and hits 'go'.
>
> I have no idea what you're talking about, but as I said before -- let
> the penetration-testers choose their own toolchain.
>
> There are plenty of badasses at Core that were instantly disrespected
> by your remark.

I don't think I said anything about anyone at Core Impact. Core Impact
is a tool that has reached a point in maturity where even a fairly
non-technical person can know an IP address or range, run a scan, run
a set of vulnerabilities based on that scan, and install the Core
Impact backdoor on the target which would meet the definition of most
penetration tests, but which is probably not worth paying someone to
do. Its moronic to think such a statement was an insult.

In the hands of an experienced person, Core Impact is a powerful tool
and one that can be a help during a penetration test. So is
Metasploit. The point is that if I'm paying the money to bring someone
in, I want that experienced person. The point I made above is that I'd
rather have an experienced person with Metasploit then someone with no
experience using Core Impact. You can write it the other way too, I'd
rather have an experienced person with Core Impact then an
inexperienced person with Metasploit.

This all leads to not using "the person lists Metasploit as a tool" as
a way to eliminate candidate companies or persons for doing your
penetration testing.

> Also see "automation" vs. "manual" above.
>
> Finally, if you want to know the differences between tools, read this
> -- http://stackoverflow.com/questions/72166/penetration-testing-tools/74513#74513
>
> > #Legal Considerations
> > You should consider that if something goes wrong with a company you
> > are essentially sharing confidential information with, whether you
> > will have protection under the law. That usually means dealing with a
> > firm or person who is legitimately 'filed' (has a background you can
> > check) and using someone in your firm's country or a country where
> > your familiar and comfortable with the legal environment in place.
> > Further you might be more comfortable with folks from certain
> > backgrounds (educationally, professionally, whatever), so check out
> > linkedin or something similar.
>
> Every application security consulting company has insurance to cover
> themselves. Here's an RFQ/RFP hint: Make sure the ones that you hire
> have insurance.

Insurance, especially with limitations in coverage, may protect the
security company in cases of legal liability but provides a small
amount of protection to the hiring company. In most cases, if a
penetration tester went rogue with information from a penetration
test, the resulting reputation damage and bad publicity would be of
greater value then the insurance settlement.

So I stand by checking people out, both from a legal protection
standpoint, but also because you want the engagement to be successful
in your environment and therefore should check out the backgrounds of
the people involved with the test. I don't view possessing insurance
as an end all indicator of anything.

> > #Reputation
> > Most companies that can provide value in pen testing have at least
> > some names that will show up when you Google. They've been quoted in
> > some article, done some presentation or talk, and so forth.
>
> Sometimes people use pseudonyms or like to keep a low profile for
> personal reasons, so be careful with this one.

With respect to their personal wishes, one would immediately ask why
they want to keep a low profile. Assuming there is nothing untoward
there, those folks should understand that there abilities have to be
known to someone in order for a demand to be there for them. Even
folks with pseudonyms usually leave a trail to find them, they just
don't want to be identified trivially and sent nonsense correspondence
by people who don't understand the information security industry.

> Please do not hire people based on Google, their blog, or some claim
> of "specialty" or "generality". You need a formal RFQ/RFP process

An RFP can be copied, are a lot of times very generic, are filled out
in the best of times by sales engineers/testers and the worst of times
by straight sales people, and are at best an incomplete way to find
the best company to work with.

Further many companies are hired outside of anyone ever performing an
RFP process. Further there are those RFP processes that are prima
facie, but not actually used for anything. Some companies do this
because its a matter of policy. Finally there are those companies that
use the process effectively as part of their vendor management. But if
you're staring at three documents to make a decision, that is a
limited amount of information to use.

In reality, the decision to hire one company or individual over
another is based on a range of factors (that could include an RFQ or
RFP) some more legitimate factors than others.  But if I wanted to
hire someone, and one candidate had something like this online:

Automated Thrash Testing By Andre Gironda:
http://www.owasp.org/images/3/32/Auto-thrash-testing.pdf

And the next guy had no information I could verify, then I would
probably look more favorably on the skills of the first candidate.

> If you really need a starting point, check this out --
> http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2

Good example of how RFP processes can be rife with document templates
filled with boilerplate language.

> Cheers,
> Andre

-DK

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified. 

http://www.iacertification.org
------------------------------------------------------------------------