From: Omar A. Herrera (omar.herrera@oissg.org)
Date: Mon Aug 07 2006 - 10:28:05 EDT
Hi Bob,
I believe that even if we can't agree on a unique definition for both we
should be able to at least distinguish between them in general terms.
I've taken into account many of the comments from other people in this
discussion and have come with these general definitions that try to include
the most relevant aspects of both concepts; comments are appreciated.
So, in general terms:
1) Vulnerability assessment
A series of tasks consisting of the identification and classification of
vulnerabilities, as well as their assessment in terms of technical and/or
business impact.
2) Penetration Testing
An activity used to assess the feasibility and the technical and/or business
impact of intrusions within an organization. Here, intrusion refers to the
capability of an attacker to illegally extract, modify, destroy or create
information and information processing functions.
Notes:
* Scope is left out. You can have Vas and PTs at network, O.S. or
application levels for instance, comprising any number of systems. The scope
is particular to the engagement. Also, we have internal and external
assessments, and there are a number of additional scope requirements such as
0-knowledge restrictions (and these restrictions are more or less useful
depending on the context).
* Depth (i.e. level of detail) is left out. I don't think that a general
definition should describe the level of detail of the tests nor the exact
methods to perform them. This also means that I don't manual testing for any
of them. In fact, the only way to perform an effective VA on a proprietary
application might be through manual testing. This also means that manual
testing (although I personally encourage and support it) is not strictly
required for a task to qualify as a penetration test. From the definition
above it is clear that using automated tools to prove that intrusion is
possible and then analysing the results would be enough. If we would like to
refer to a particular level of detail we can always name the particular type
of test (e.g. "manual penetration testing" or "tool based, automated,
vulnerability assessment").
* I'm also using a couple 'and/or's in the definitions. Both assessments can
be executed in two dimensions: the technical and the business dimensions. I
personally recommend considering both dimensions if possible, but it would
be naïve to think that this is always required. For example, a company might
only require a tool based, automated and technical, vulnerability assessment
while considering performing the business assessment on their own.
* The quality of the report is not included in the definitions. This is a
matter of the quality of the assessment (good or bad) and not part of the
description itself. We deduce from experience (many past posts in this forum
seem to back this) that the analysis of results should involve some manual
work to produce quality assessments (i.e. putting the results in context to
produce more accurate evaluations), but stating that any assessment will be
good just because some manual analysis was performed is not true. Stating
that the automated analysis included in the reports of tools is always bad
for any given situation is also false.
* There is a big issue of whether one assessment is part of the other. From
the definition above I would say that VAs are, in general, independent from
PTs, although that wouldn't prevent an assessor from stating the theoretical
impact of an intrusion based on VA results. In the case of PTs it is implied
that at least some tasks from the VA need to be performed (before or during
the PT), for instance, the vulnerability identification (the only way to get
in "illegally" is through a vulnerability, which does not always means a
technical vulnerability; procedural vulnerabilities also count).
* General definitions like these should only serve to distinguish between
both concepts and shouldn't be used alone to refer to a specific/required
type of assessment. Defining additional scope and quality requirements is
essential to give the assessor a clear idea of what is being requested (or
for the client to understand what is being proposed in a proposal). Good
examples of 'not sufficiently well defined' assessments (in my opinion) are
included in the PCI DSS standard (I hope they will take care of this in
future revisions), for instance:
- 11.2 specifies when and what (a VA) but it never states clearly the level
of detail required. However, it gives a few examples of what to include and
states that qualified vendors must perform these scans (which implies that
those qualify vendors would know exactly the level of detail and the scope
that is required). The Security Scanning Procedures document gives some more
information.
- 11.3 Again, it states what (a pentest) and gives a more or less clear
idea of the scope (network infrastructure and applications) but fails to
describe the level of detail (must this include manual testing? Are there
any time restrictions?).
- 11.1 This is not a PT and I wouldn't exactly call it a VA because you
don’t search specifically for vulnerabilities. This requires a series of
tests to assess the effectiveness of security controls and measures to
prevent unauthorized accesses (of course, ineffective controls/measures make
you vulnerable, but the goal of the test is different). It gives a few
examples of what to include in the scope but is again too general.
In the case of standards I would expect to find detailed requirements for
the scope and measurable results for the QA function (after all, that is
what a standard is all about: "a required level of quality or proficiency").
This is what should have happened in PCI DSS with respect to VAs and
Pentests, but it looks more like a set of guidelines than a standard.
I think that the opposite is causing the confusion that reigns here. We are
all trying to define our own standards for what we consider
good/acceptable/best VAs and PTs, but we are forgetting to extract the
relevant elements of each concept and set aside the aspects that define a
specific type of assessment. Maybe it is time to create the "Securipedia"
;-).
As StyleWar previously stated, being in part an art is not necessarily a bad
thing, and I would say that we require some degree of it in both types of
assessment (probably more with pentests, where creativity is quite useful).
But there is definitely a lack of agreement on key things in our industry. I
also agree with the sentiment of Daniel that it is surprising that this lack
of agreement exists in relatively basic concepts. To put it in other words I
would say that we are not sustaining and developing our profession with
adequate levels of formality.
Regards,
Omar Herrera
> -----Original Message-----
> From: Bob Radvanovsky [mailto:rsradvan@unixworks.net]
>
> ----- Original Message -----
> From: Daniel Accioly Rosa [mailto:listas.accioly@terra.com.br]
> To: pen-test@securityfocus.com
> Subject: RE: Vulnerability Assessment vs. PenTest
>
>
> > What I find most interesting in these discussions is that even tough we
> are
> > all seasoned professionals, we can't agree 100% on a definition neither
> to
> > Vulnerability Assessment or Pen Testing.
> >
> > What lesson should we take from this? I'm not saying that we don't know
> what
> > we are doing (most of use here are very good professionals), but maybe
> there
> > is too much "art" in this job... Each day that goes by I believe more
> and
> > more that we need to agree on common grounds on how we perform our
> duties...
>
> I think much of it comes from the "human factor" in that we view things
> differently; thus, the method of approach we take will always be
> different. Even the so-called "security standards" are variable in of
> themselves, too. So, don't just be pointing fingers at the individuals.
> Organizations that are representative of this industry, too, have issues
> with agreeing upon a set of standards or guidelines. ISO is one way, SANS
> is another, DoD yet another, NSA even still another, etc., etc.
>
> To me, the overall schema of things is how we (as an industry) perceive
> threats and their vulnerabilities, and what are associated to them. Once
> that is agreed on, the rest of it might fall better into place. ;))
>
> >
> > You are right StyleWar, coffee now would be nice.. :)
> >
------------------------------------------------------------------------------
This List Sponsored by: Cenzic
Concerned about Web Application Security?
Why not go with the #1 solution - Cenzic, the only one to win the Analyst's
Choice Award from eWeek. As attacks through web applications continue to rise,
you need to proactively protect your applications from hackers. Cenzic has the
most comprehensive solutions to meet your application security penetration
testing and vulnerability management needs. You have an option to go with a
managed service (Cenzic ClickToSecure) or an enterprise software
(Cenzic Hailstorm). Download FREE whitepaper on how a managed service can
help you: http://www.cenzic.com/news_events/wpappsec.php
And, now for a limited time we can do a FREE audit for you to confirm your
results from other product. Contact us at request@cenzic.com for details.
------------------------------------------------------------------------------
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:56:35 EDT