A recent pen test changed my viewpoint on the industry standard practices. The old ways are no longer working. Not only are industry standard pen tests not working they are providing a false sense of security.
Given the latest generation of firewalls that are adaptive and resistant to scanning we would expect that port scanning and repeated door knocking would be less effective. So why are we still being charged for a type of scan that is ineffective?
Recently a fortune 500 company performed a costly pen test on a network segment running critical systems and returned no results at all. Thinking that was impossible I ran a few tests of my own with the same tools that they used. I got two ports on the first IP and then nothing after that. I tried to mix it up using NMAP with the most stealthy settings. I decided to pwn them using my Nessus skills and even tried a zombie and spoofing different addresses and nothing worked. I was shut out. Nessus and other tools failed to get anything from our ninja-like firewall.
Now that I had established that the standard tool-set was producing almost nothing except false warm fuzzies I broke out "my" set of tools. And I did what I do very well... break the rules. We know that companies on the net are spying on us in unprecedented ways and without restriction. Why not take advantage of their legally sanctioned espionage and leverage their data sets to reflect our own systems?
A day and a half later with 138 vulnerabilities and counting and a full network diagram including IPs, servers, web sites, email addresses, technologies, phone numbers, all laid out in perfect searchable and browseable order. A couple hours of hand testing and searching some specific vendor sites and I found even more issues not determined by any of the new "black hat" tool-sets that I use.
I realized five very valuable lessons:
1) The cost of a pen test does not predict its effectiveness
2) Standard pen tests against active firewalls are useless and give a false sense of security
3) Standard pen tests use tools that are basically out-dated in four ways
a) they rely heavily on port scanning
b) they rely heavily on CVE lists and CVE lists don't have a monopoly on vulnerability info
c) they don't leverage the Google factor enough
d) they require firewall exceptions which distort the important "view of the hacker"
4) We need to use the tools hackers actually use, not the ones we're sold by security intelligencia
5) There never will be a substitute for hand testing