Free Article 1/2011

FREE ARTICLE

Penetration Testing
What Should You Look For?
So youre in the market for penetration testing? Do you know what youre buying? Do you know how to buy it? I will attempt to define the types of penetration testers out there along with a sound Request for Proposal process for penetration testing. Be careful what you buy, you only get what you ask for.

o right off the top, a disclaimer: I earned a living for many years as a penetration tester and now run the penetration testing team at Hurricane Labs. Now that I have that out of the way, here we go. Why would you ever need a penetration test? The answer, per usual, depends on your perspective and needs. Most companies that have sensitive data in their enterprises should look at getting some form of penetration testing. Why do I say, some form? Simply put, there are as many definitions of penetration testing as there are fish in the ocean. Seriously, sometimes I think we make this stuff up as we go along. If youre looking at hiring out a third party to do your penetration test (and you should, reasons later), I think it would be helpful to go through some of the definitions as I see them in the industry. This list should then help you form some intelligent questions to ask the companies you are considering. After the definitions well review some questions you should include in your RFP to help weed out some of the pretenders.

Sadly I still see it out there almost every week or so. These are the guys who point Qualys, Nessus, or worse, turned all the way up, at your network, press submit and watch the fireworks. Typically they do no tuning of the scan and just let it go. This is dangerous on so many levels, and this sort of thing can quickly bring down your whole network. Youll recognize them because their salespeople will say all the right things but their slicked back hair will give it away. Ask them about their manual testing methods and soon youll realize they have none. By the way, these are the sorts that will give you a clean bill of health in a day or so and offer no real opinion on your applications or network. Avoid this sort of tester at all costs.

Definition Number 2 (N2)

The Old Smash and Grab
I wish I could take credit for this but it was coined by a colleague of mine (Rick Deacon @rickdeaconx). Basically these are the guys who come in and simply destroy your network and applications with no regard for your business or money lost. They are also scanner heavy and generally are in love with BackTrack and Metasploit but only know enough to break things and usually offer no recommendation on fixing them. These are also the types that will drop a 120-page report (single-spaced) on your desk
http://pentestmag.com

Definition Number 1 (N1)

Lets scan them and hope they never actually get attacked.
This is my least favorite of the varieties out there, mainly because I thought this breed of service was long dead.
02/2011 (2) June

Page 1

and disappear. They are generally only slightly better than number one in that they employ some tools that take slightly more skill than just pressing submit. They can be harder to detect because they can talk a good game and make it sound like what you expect a penetration test to be. These are the ones you hear about that can tell you a whole lot about an application being broken but very little on how to fix it or why you should fix it. These are the sorts of testers you would want to use on pre-production testing. If your application or setup has an obvious flaw they will probably find it. You wouldnt look to these guys for any precision testing or advice about how your stuff should look or work.

a good team. Usually their upsides far outweigh the downsides but Im very biased here. We also tend to be a very anti-social lot but thats okay, no one is perfect.

Definition Number 5 (N5)

The Auditor
The auditor is a funny one: they arent actually auditors, but they walk and talk like one. Words like risk analysis and GAAS are tossed around without concern for anyones safety. This type tends to not be very technical and would prefer to just interpret the results of the more technical folks. This is the type that LOVES to assign scores to things and put everything into its neat little box. You find this type of tester at accounting firms and other odd places to find penetration testers. Again, theyre usually not doing the actual penetrating but just tend to work on the reporting the most. They also like to have a lot of letters after their names, I dont know why, thats just how it is.

Definition Number 3 (N3)

The Pacifier
This type of tester is basically afraid to break anything. He will run light scans and quote best practices in the reports but do little else. He wont venture to break anything for fear of the customers never hiring him again. He will go to great lengths not to offend any application developer or network architect, so some things tend to go unreported. You will most often see this type in smaller places that offer a ton of services and products. They typically have some other relationship with the customer on some other level, maybe they sell them products or repair their printers (dont laugh Ive seen this) and oh, by the way, they offer penetration tests too. Ideally you would want to engage a place that has nothing to lose or has sufficient division of penetration testing responsibilities. This will become obvious after a good round of questions.

Definition Number 6 (N6)

The Rich Kid
Ive run across this one quite a bit in my career and I can almost always predict their argument. Theyre the ones who invested $100,000 in the tool of the year and rely only on that tool to do the actual testing. They tend to be less technical that good testers but if all youre after is a check mark on your audit theyll suffice. They use terms like Big Three firms and Fortune 500 customers as if theyre going out of style. They are very buzzword-heavy and like to tell you how the tools they use rate very highly on various industry tests. Its not that Im opposed to a lot of the tools out there for professional testing; I just dont understand the point. My philosophy is and has always been that an attacker is probably not going to spend that kind of money for a tool to attack your network or application. The attacker will, more often than not, rely on some open source tool they find or write the exploit themselves. My belief is that a good tester should mirror the attacker and then add the value of professional experience on top of that skill. A good place for the rich kid is an internal team that is just looking for the so-called low-hanging fruit. Why would we want to outsource our penetration tests in the first place? The reason is simple: typically you want an unbiased look at your network and/or application. You want someone to look at it externally and tell you what they can see and can prove as
http://pentestmag.com

Definition Number 4 (N4)

The RFCist
This is a lesson I learned the hard way; you see, I used to be an RFCist. Im about 5 years clean now. The RFCist is the person that can quote to you every section of every RFC on the planet but not be able to tell you why your application or network should conform. The RFCist tends to be very poor at risk analysis and tends to not understand the need for executive summaries. Again I have been an RFCist and have managed and still manage quite a few of them. They can be challenging to manage but they are, typically, invaluable members of a good penetration testing team. The reason being is that they tend to be very technically proficient and detailoriented, both of which are absolute requirements on
02/2011 (2) June

Page 2

FREE ARTICLE
security issues. This can help you build a more robust application or network and alert you to any number of problems with your set-up. If you have internal folks looking at it from a security angle and you should do that too they will sometimes overlook something because the set-up is so familiar to them. Also, internal folks tend not to want to embarrass their buddy in the next cube; Ive see internal folks clue in their app dev buddy and ruin the integrity of the test by letting them fix it first. It is a great idea to have an internal team, but they should not be your sole source of testing. You need an unbiased look. Every company has different needs, so you want to weigh what you really need against what youre going to buy. If you have 10 desktops, no website, and outsource your email, you probably dont need a penetration test at all. If, however, you have 10,000 desktops and 15 websites running across 1,000 servers and take credit card transactions you should not be asking if you need a penetration test, you should be asking how many you should have a year. The process should and can be painless for you if you do your due diligence up front and ask the right questions. Your journey should begin with a simple RFP process, and I will tell you, as a guy who has done this for a number of years, long form RFPs will not get you the best testers out there. I have thrown away more long form RFPs than I care to remember. In some cases the RFPs were longer than the report would have been: I have seen 20-page RFPs for testing out a single /27 subnet.

Has any current member of your team written publicly about penetration testing or security in general? Please list links to articles and/or blog posts with dates

I find this question to be the most revealing and in running the business to which I am dedicated. If you have the best tester in the world but they cannot communicate their thoughts coherently, what is their end report going to look like? Probably pretty poor. If you can see writing samples and get an idea of their philosophy ahead of time you will build a greater comfort level with them.

Has any current member of your team written and/or released any penetration testing or security tools?

This can separate the good testers with a mastery of their toolsets from the truly great ones who can write their own tools if necessary. As with the article writing above it can give you a good idea ahead of time of their philosophy as well as their technical skill.

Please dene your testing philosophy

You want to start off with a clear mission statement for your test: what do you want to accomplish with this test?

Every good team should have a sound philosophy around their testing. It should include what they will do and, just as importantly, what they wont. For instance, I dont like to include denial of service testing in our standard testing offerings; thats spelled out up front. Others Ive seen include what sorts of tests will be run against a web application. This gives you an idea of how much thought has been put behind a companys service offering. It will also almost always rule out definition N1 from above.

This helps the respondent understand what you want to get out of the test and can help focus team members in on exactly what you want. Do not skip this step; it might be the most important. Now, what sorts of questions should you be asking? Im glad you asked; in no particular order:

Can a tester be reached during the test?

Ask for an emergency phone number where you can contact the tester during the testing phase just in case anything goes wrong.

How will the status of the test be provided?

How long have your individual team members been performing penetration testing?

Please only include the team member(s) who will be assigned to our account. This gives you an idea of the experience level of the team members as well as boxes the provider in to providing you experience levels of only the team members that will be testing your stuff. An old industry trick is to bait and switch you by providing your bios and experience of senior staff then assigning very junior staff to do the actual testing. You want to try to avoid this so try to work that into your RFP.
02/2011 (2) June

This is basically a project management question, but it can be very important. Youll want to know the frequency of updates as well as if youll be contacted immediately if a horrible/highly critical vulnerability was found.

Are remediation services available?

Some companies are simply not set up to help you with fixing up the holes they find. You have to decide if this is important to you. My opinion is that it helps tremendously if a company can offer these services because it helps the overall testing process and the reporting if real advice can be offered on how to fix the problems found. Again, its a personal choice.
http://pentestmag.com

Page 3

Will a re-test be included in the testing price?

Basically, will the company come back and re-test the things found to make sure they are properly fixed? I think this is becoming more the norm these days but its worth checking on during the RFP phase.

Ask for a sample report

This will obviously give you a good view of the final deliverable. The sample report should include a response section so that you can comment on vulnerabilities and provide feedback on the report. The report phase should be collaborative and should not exclude you as the customer. Often reports are done in the dark and in a vacuum; there is no way you can build a good report without taking into consideration the business impact of the vulnerabilities and there is no way to understand the business impact without involving you.

a different definition and a different idea of what a test should be. There is some work under way to standardize the way things are reported and the way tests are performed but not all providers are going to follow those standards. You need to be careful about who you hire because the right tester needs to have a combination of technical, business, and communication skills. You have to be able to trust that they will take care with your production systems and not have them come crashing down during the test. Can this always be avoided? Of course not, but a good tester knows how to take the necessary precautions. Asking the right questions will get you that good tester and your life will be better for it.

BILL MATHEWS
Bill Mathews is co-founder and lead geek of Hurricane Labs, an information security rm founded in 2004. He has been in IT almost 20 years, in security specically for 13 of them, and has been interested in security since C3P0 told R2 to never trust a strange computer. You can reach Bill @billford and @hurricanelabs on Twitter and read other musings on http:// blog.hurricanelabs.com

References

Ive always been torn on references. No one is going to give you a bad reference. Of course you still want to check them but I wouldnt weight the references too heavily. Ive always wanted to ask a vendor for some unfavorable references as well as good ones to give me a balanced view. So far no one will give me any bad references but I will keep you posted in case it happens. Penetration testing services can vary wildly from provider to provider as with anything. Everyone has

Subscribe
02/2011 (2) June Page 4

http://pentestmag.com/wp-login.php?action=register

http://pentestmag.com