From: Ivan Arce (ivan.arce@coresecurity.com)
Date: Thu Feb 28 2008 - 18:26:09 EST
Hello Andre & pen-testers
Since Core was been mentioned as having lied on this list I felt compelled
to reply and ask you to better qualify that statement so I can verify it
and address it properly if need be.
I'd also like add my comments regarding your assessment of CORE IMPACT
capabilities with the *obvious caveat* that I am Core's CTO. But before
that, I'd like to ask you to clarify how did you come to your conclusions
and if you were or are a licensed user of a current and up-to-date version
of CORE IMPACT because I suspect you may be providing opinions that are
based on a partial or limited view of our product. Please feel free to
contact me directly or through any of Core's Customer Support channels so
we can follow up on any particular feedback (or complain) you may want to
provide.
We really appreciate constructive criticism and value the help of the
security community to improve our product, then again, many may choose not
to help us and that is totally fair too.
Going back to the original comments about CORE IMPACT and the 'count of
exploits' I'd like point out just that throwing numbers without qualifying
the measurement criteria and the relevance of the methodology is not a
very serious assessment of a product's capabilities, its suitability for a
given use or the value it may provide to a security professional.
Nonetheless, the last time I checked (a few minutes ago as of today) a
fully-functional legitimately activated and up-to-date CORE IMPACT 7.5
distribution has 665 exploits targeting 4414 unique target combinations
(OS, SP, service, application version) distributed across 466 modules and
that's not counting the components that detect and exploit SQL injection
and Remote File Inclusion vulnerabilities in custom/proprietary web
applications.
Note that, all of our exploits are fully-tested to meet quality standards
before they ship to all our active customers and thereafter continue to be
supported through the entire life cycle of the product (all of which is
included in the product's purchase and licensing terms). We do not ship
0day exploits (that is, exploits that target vulnerabilities for which
information is not publicly available without access or confidentiality
restrictions), we do not ship different sets of exploits for subsets of
users in our customer base and we do not ship or support exploits (or
other modules) that are not developed by our own team, so it is not so
easy to a provide fair comparison just with "count numbers".
Exploit quality assurance seems to be an often overlooked topic, perhaps
because we are the only organization (that I know of) that does full
system tests running all the supported exploits against all the supported
platform combinations with all the supported connection methods *daily* in
order to have constant visibility of the quality of the "exploits"
component of our product (there are many "non-exploit" modules that
provide valuable penetration testing capabilities as well). This requires
a level of investment in processes and infrastructure that is not
insignificant and perhaps only available to a commercial organization that
is fully committed to ensure a certain degree of product quality.
Lastly, as I said above, we maintain all the modules shipped with the
product and provide regular product updates on a weekly basis. The number
of module updates (which include new exploit modules as well as
improvement and bug fixes to existing ones and other types of modules) has
been at an average of around 22.5 per month in the past 6 months.
But of course... all of this information is provided directly to all our
users in the product's dashboard (the first screen that you see when you
run CORE IMPACT) so I am merely transcribing it here without much analysis
so far.
In any case, I don't know what is the "right" criteria to use for counting
of exploits or what's the criteria to determine their relevance, although
I've talked about this and presented several ideas at many public venues
in the past years and I'd welcome further discussion on that topic.
However, I do think that throwing numbers without a consistent and sound
rationale for a comparison is usually miss-leading. In fact I believe this
because that had been the case during the early 1990's when "number of
vulnerability checks" was the common currency for comparisons of
vulnerability scanning products. At the time I was part of the development
team of one of the three emerging leaders in the vulnerability scanning
software market and I clearly remember that every vendor and all otherwise
biased parties had their own (self-serving) methodology and product
marketing strategies to count "vuln. checks". In the end those comparisons
did not prove very useful to anybody and today there aren't many
organizations that use number of vulnerability checks as substantially
relevant information to determine if a given scanner will help them
fulfill their requirements and provide them the value expected from it.
So that moves the evaluation criteria discussion to a discussion about
whether a given product may or may not provide its customer with enough
value to justify its purchase and subsequent use and whether it
appropriately fits what its expected from it.
I can't speak for other tools or even for CORE IMPACT because I am not a
paying customer :) but I can certainly say that at Core we view our
product as much more than a bunch of features and a bag of exploits and we
make a very serious effort to understand our customer's needs and to
combine all our capabilities to build software that gives back to our
users the money they paid (and hopefully more!) in the form of valuable
and usable tools to improve their security or the security of their customers.
We may or may not succeed in that endeavor but we firmly believe -across
our entire organization- that to be the promise that commercial
penetration testing software vendors must deliver on.
-ivan
Andre Gironda wrote:
> On Wed, Feb 27, 2008 at 1:38 PM, Trygve Aasheim <trygve@pogostick.net> wrote:
>> This doesn't mean I don't like or use Metasploit, Canvas or any
>> other...I just want to point out that the quality of a product is not
>> based a number, and Core Impact has proven its quality many times, and
>> in many ways.
>
> The numbers show that Core Impact is superior to Canvas and Metasploit.
>
> Unfortunately, it also shows that Impact is missing quite a lot. The
> point I was trying to make is that you can't use only one exploitation
> engine.
>
> However, I also fail to see the point of using an exploitation engine
> except in the case of testing IPS/IDS or similar. In this case,
> anyone would clearly be better off using BreakingPoint Systems
> BPS-1000.
>
> Using exploits on production or IT networks is unethical. This isn't
> the wild west. You're overpaying by about $19K-$26K for what you need
> when you go with Core Impact. I don't know about ya'll, but the idea
> of propagating a pseudo-worm through a corporate network seems about
> as good of an idea as asking the power company to shut off electricity
> to a hospital for "just a minute, to see what will happen".
>
> Instead of RPT, I suggest asset management combined with regular,
> good-old fashioned vulnerability scanning. Most of the "experts" I
> know don't even understand the difference between a vulnerability and
> an exploit. More of those people don't even understand how unreliable
> exploits usually are (let alone scanning errors in vulnerability-only
> scanners).
>
> Core already lied once on this list about how many modules vs.
> exploits vs. CVE's they support. They could make anything up. The
> money numbers do not lie. Compare to Rapid7, Tenable, Lumension, or
> McAfee for yourself.
>
> If you have to raise awareness by running live exploits, try
> Metasploit. It's free. Management still not convinced? Already
> covered all the Metasploit exploits? Try Canvas, it's cheap.
> Management still not convinced? Already covered the Canvas exploits,
> too? Add an exploitation pack or two. Start writing your own
> exploits. Management still not convinced? Already covered all of the
> Canvas exploitation packs and started writing your own in-house
> exploits specific to your architecture? Maybe Core Impact will help;
> call them for a demo.
>
> I have no idea why people are so quick to jump to Core Impact first.
> You can't just throw money at these types of problems. Security is a
> very careful and gradual process.
>
> Cheers,
> Andre
>
> ------------------------------------------------------------------------
> This list is sponsored by: Cenzic
>
> Need to secure your web apps NOW?
> Cenzic finds more, "real" vulnerabilities fast.
> Click to try it, buy it or download a solution FREE today!
>
> http://www.cenzic.com/downloads
> ------------------------------------------------------------------------
>
-- "Buy the ticket, take the ride" -HST Ivan Arce CTO CORE SECURITY TECHNOLOGIES http://www.coresecurity.com PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836 B25D 207B E78E 2AD1 F65A ------------------------------------------------------------------------ This list is sponsored by: Cenzic Need to secure your web apps NOW? Cenzic finds more, "real" vulnerabilities fast. Click to try it, buy it or download a solution FREE today! http://www.cenzic.com/downloads ------------------------------------------------------------------------
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:58:26 EDT