From: Omar Herrera (oherrera@prodigy.net.mx)
Date: Wed Dec 15 2004 - 09:44:48 EST
Warning: long responses below :-)
----- Mensaje original -----
De: rzaluski <rzaluski@ivolution.ca>
> A Penetration Test entirely depends on the scope and depth of the
> operation and what has been agreed upon. Sometimes finding a
> vulnerable system is
> enough. It all depends on the level of intrusion of the
> Penetration Test.
I agree. Flexibility is a key to sell PenTests. Clients do have problems (or not idea at all) defining the scope and vendors should help them defining the scope by making an adequate balance between cost and benefits. The benefit should also be very clear to the client, and this is the key to a successful Pentest.
> My point is if you are Pen Testing a client's mission critical
> productionsystems do you really want to bring it / them down? It
> is entirely possible
> to do so costing a company potentially large amounts of money in the
> process. This is especially if the testing is taking place with
> anything that processes monetary transactions.
I also concur. My comment regarding manual tests wasn’t meant to imply that the consultant gives added value by going beyond scope. You can bring a system down with automated tools as well in any case. The important thing here is not whether DoSing a system is in the scope or not, but rather whether the consultant knows what he is doing and is complying with the defined scope. It is within the scope that the consultant employs its abilities to make better tests.
There won’t ever be enough time to throughfully test everything within an organization, therefore automated tools are used to save time with some tasks (which should be also saving money for your client), but even configuring the tools correctly is not a simple task. You just can’t run a tool with all tests available turned on, it will be a waste of time and get you huge amounts of false positives that you will need to filter out. In the end, this costs money to the client and to the PenTest provider.
Making a good job is simply not good enough, results have to be of value and understandable for the client. Even if people who make automated tools make good efforts to make comprehensible reports, there is always a need to explain things. Filling that gap is also a responsibility of the consultant because he is able to identify certain situations and needs of their clients.
Summarizing some key points of what a good PenTest should have from my point of view (also including comments from other people in the forum):
* PenTest consultant should be flexible and understand the needs of their clients ( a methodology is a guideline, but in this business vendors do not necessarily need to go step by step by the book)
* Have a well defined scope and comply with it (no more and no less).
* Have capable consultants able to perform manual checks if/when necessary (basically having people that know what they do).
* Choose and adequate set of tests (manual and automated) that will yield the best balance of cost/benefit (example: if the client provides detailed information of their infrastructure to improve results and it is documented in the scope, then there is simply no need to include tests for plataforms/applications that are not part of that infrastructure. I emphasize this point because I’ve seen it done many times and it makes the consultant look bad to the eyes of some clients).
* Have a methodology (a PenTest consultant can’t just be cluelessly poking around; there is money paying for the job and even a black box test can be planed and documented).
* Make results understandable for the client (whether on the executive summary or on the technical report, it is the responsibility of the consultant to explain every finding and recommendation to the client until it is clear).
* Findings and recommendations should consider the clients infrastructure and business process; consultant should make use of all information available to him to make proper annotations. There is a big difference between technical impact of a vulnerability and the actual impact for business, since resources have diverse degrees of importance for an organization (automated tools can measure the first reasonably well, but measuring the impact for a business process is a job for a capable human being).
Now, if I interpreted correctly Adriel’s concern: clients are still distrust PenTest consultants. He discussed a specific points that cause this problem: the grade of use of automated tools and manual testing, and confusion derived from PenTest definition (what is included and how should it performed). There are many more causes that make selling and performing a PenTest a difficult task, and also from the clients point of view there are a lot of problems while buying a PenTest (Is the price of this proposal right? Is this vendor really good at it?).
Methodologies such as OSSTMM make things easier for consultants, but there is really few information and tips available for clients (not all clients have an Infosec department with experienced personnel that is able to discern between bad and good proposals/consultants). So, maybe providing the clients with such information will make things easier for both (probably extending OSSTMM with some questions that an organization might ask themselves when evaluating a vendor). Here are a few I can think of:
While evaluating a PenTest proposal, a client should ask himself:
* What certifications do the consultants posses(I know, I know… it doesn’t guarantee capability, but at least it attest that the guys making PenTest had a minimum knowledge of these arts.)
* What methodology does the vendor employs? If proprietary is he disclosing it for evaluation?
* What kind of tools will the vendor employ?
* What kind of manual test will the vendor perform?
* Is the scope reasonable from the business point of view? Are the proposed targets part of the critical business processes? Can it be adjusted to business needs?
* Can someone from the company be with the consultants during PenTest to see how the job is done?
And some more difficult questions:
* Is the price right? (you can’t check it out on the Internet like the price of a certain brand of firewall or a Kg. of Tomato).
* Where have these guys worked before? (sometimes, with permission, vendors can provide some references) Has this vendor/consultant screwed up badly elsewhere? (vendor won’t be telling, of course…).
Enough of it. I’ll stop here or moderator will ban me for DoSing the list with lengthy posts, written in bad English.
Best regards,
Omar Herrera
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:54:11 EDT