mailing list archives
RE: Vulnerability Assessment vs. PenTest
From: "Craig Wright" <cwright () bdosyd com au>
Date: Wed, 9 Aug 2006 14:35:28 +1000
To add a little to this debate.
First, there are two types of Audit, internal and external. An audit,
consisting of an evaluation of an organisation's systems processes and
controls, is performed against the set standard or documented process.
Audits are designed to provide an independent assessment through a
qualified independent assessment of representations about the system or
process. An audit may also provide a gap analysis of the operating
effectiveness of the internal controls.
An audit differs from an inspection in that an audit makes
representations about likely future results. An inspection evaluates
past results. Or an audit to be valid it must be conducted according to
accepted principles. In this, the audit team and individual auditors
must be certified and qualified for the engagement. Numerous "audits"
are provided without certification, these however are qualified reviews.
A penetration test is an attempt to bypass controls and gain access to a
one system. The goal of the penetration test is to prove the that the
system may be compromised. A penetration test does not assess the
relative control strength nor the system or processes deployed, rather,
it is a "red teaming" styled exercise designed to prove illicit access.
The real strength of a penetration test is marketing the need to improve
controls to internal management. A penetration test is of limited value
in the greater scheme of a systems information security due to the
restricted nature of the test and the lack of inclusion of many key
A vulnerability assessment is an assessment and gap analysis of a site's
or a system's control strengths. A vulnerability assessment is a risk
based process. The process involves the identification and
classification of the primary vulnerabilities which may impact the
system. Often, methodologies such as fault tree analysis end cause
consequence analysis are employed in this review.
Both vulnerability assessments and penetration tests may be conducted as
a white box or black box analysis. A black box analysis is instigated
with little or no knowledge of the system being tested. A white box
analysis is conducted for knowledge of the system.
A vulnerability assessment is a critical component of any threat risk
assessment. Following the vulnerability assessment and impact analysis
is conducted and used in conjunction with a threat report to provide for
an estimation of the organisation's risk to selected attack vectors.
External audits are conducted (or at least should be) by independent
parties no rights or ability to alter or update the system. Internal
audits involve a feedback process where the auditor may not only audit
the system but also potentially provide advice in a limited fashion. And
external auditor is precluded from advising their client. They are
limited to reporting any control gaps and leading the client to a source
of accepted principles.
The common perception that running an automated scanner such as Nessus
or one of its commercial cohorts is in itself a vulnerability or
penetration test is false.
Most of the so-called penetration tests that are provided are no more
than a system scan using tools. A penetration test it correctly provided
will attempt the use of various methodologies to bypass controls. In
some instances this may involve the creation of new or novel
The issue is not that many people commonly use the words interchangeably
but that so-called professionals fail to differentiate the terms. Of
particular concern is the use of audit and the designation auditor. This
is as these terms are often restricted in code. This is that most
jurisdictions have statutory requirements surrounding their use and
Information security systems provide many of the functions that
construct a control system. Of particular concern are controls that
limit access to accounting and financial records. This includes records
held by systems that provide an e-commerce transaction path. In many
jurisdictions it is an offence to sign off an audit report when you are
not a certified auditor. Traditionally the path around this has been not
to call the process of testing the system and audit, but rather to call
it an agreed procedures review.
An agreed procedures review or simply a review is an analysis of
controls performed against an agreed process.
Some example's of an audit include SAS 70 (part 1 or 2) audits, ISO
9001,17799:2/27001 certification audits, HIPPA audits. There are many
different types of audits and many standards that an audit may be
There are various processes and procedures used to provide vulnerability
assessments and threat risk analysis. Standards such as AS/NZS 4360:2006
I commonly mandated by government organisations.
Penetration testing, if done correctly, may provide some value in its
free-form approach. When correctly implemented, a penetration test adds
a level of uncertainty to the testing. The benefit of this uncertainty
is that it might uncover potential flaws in the system or controls that
had not been taken into account when designing the control system. To be
of value, a penetration test must needs to do more than scan a system.
It needs to do something novel and unexpected.
There is little similarity between a penetration test, vulnerability
assessment, risk assessment or audit. The lack of understanding of these
differences impedes the implementation of effective security controls.
Bus to finish, 'Stylewar' is correct in stating that "an audit must
follow a rigorous program...". Christine's appraisal of a vulnerability
assessment would more correctly be termed as a controls assessment. A
controls assessment may also be known as a security controls review.
As for the need to develop a structured taxonomy (naming system), there
is already one in existence. None of these terms or services is new. All
these services have been provided for as long as computers have been
used by business and government. They were definitely employed as far
back as the 70s.
From: StyleWar [mailto:stylewar () cox net]
Sent: Wednesday, 9 August 2006 3:19 AM
To: 'Christine Kronberg'; 'Arkem Paul'
Cc: pen-test () securityfocus com
Subject: RE: Vulnerability Assessment vs. PenTest
Point of fact that an audit must follow a rigorous program, and has a
documentation and traceability requirements with it that an 'assessment'
does not. They are 'approximate' in the hands of a well disciplined
assessment team - but I would stop a hair short of calling them equal..
From: Christine Kronberg [mailto:seeker () shalla de]
Sent: Sunday, August 06, 2006 11:54 AM
To: Arkem Paul
Cc: pen-test () securityfocus com
Subject: Re: Vulnerability Assessment vs. PenTest
On Sun, 6 Aug 2006, Arkem Paul wrote:
A Vulnerability Assessment should be a comprehensive look
and procedures to implementation of security in the network
include such things as patch management, virus protection, user
education, SOE hardening, infrastructure configuration, etc.
So basicly an assessment is equal to an audit? The
is what I usually expect from someone doing an audit.
A vulnerability assessment I tend to understand in terms
a specific application (in far more detail than a
There are a couple of term mixed every now and again (like someone
else just stated: funny that we professionals don't come
up with _one_
Did I miss one?
Liability limited by a scheme approved under Professional Standards Legislation in respect of matters arising within
those States and Territories of Australia where such legislation exists.
The information contained in this email and any attachments is confidential. If you are not the intended recipient, you
must not use or disclose the information. If you have received this email in error, please inform us promptly by reply
email or by telephoning +61 2 9286 5555. Please delete the email and destroy any printed copy.
Any views expressed in this message are those of the individual sender. You may not rely on this message as advice
unless it has been electronically signed by a Partner of BDO or it is subsequently confirmed by letter or fax signed by
a Partner of BDO.
BDO accepts no liability for any damage caused by this email or its attachments due to viruses, interference,
interception, corruption or unauthorised access.
This List Sponsored by: Cenzic
Concerned about Web Application Security?
Why not go with the #1 solution - Cenzic, the only one to win the Analyst's
Choice Award from eWeek. As attacks through web applications continue to rise,
you need to proactively protect your applications from hackers. Cenzic has the
most comprehensive solutions to meet your application security penetration
testing and vulnerability management needs. You have an option to go with a
managed service (Cenzic ClickToSecure) or an enterprise software
(Cenzic Hailstorm). Download FREE whitepaper on how a managed service can
help you: http://www.cenzic.com/news_events/wpappsec.php
And, now for a limited time we can do a FREE audit for you to confirm your
results from other product. Contact us at request () cenzic com for details.
RE: Vulnerability Assessment vs. PenTest Bob Radvanovsky (Aug 07)
Re: Vulnerability Assessment vs. PenTest Gray Ghost (Aug 07)
RE: Vulnerability Assessment vs. PenTest Craig Wright (Aug 09)
RE: Vulnerability Assessment vs. PenTest David M. Zendzian (Aug 10)
RE: Vulnerability Assessment vs. PenTest Craig Wright (Aug 10)
Re: Vulnerability Assessment vs. PenTest harshal . mehta (Aug 10)
Re: Vulnerability Assessment vs. PenTest lakshminarayanan79 (Aug 21)
Re: Vulnerability Assessment vs. PenTest Marco Ivaldi (Aug 29)