On penetration testing – harmful?

Over at Sensepost Security, there’s a new blog entry wondering about Haroon Meer‘s talk “Penetration Testing Considered Harmful“. Those who know me know that I’ve had this view for a very long time. I’m sure you could find a few posts in this blog.

Security has to be a intrinsic element of every system, or else it will be insecure. Penetration testing as a sole activity and piece of assurance evidence makes security appear on the fringes of the development, something that you pass or fail, something to be commodotized, a box to be ticked, and ultimately ignored. Penetration testing as is done by most in our industry is incredibly harmful. It’s a waste of investment to most organizations, and they know it so they try to minimize wastage by minimizing the scope, the time, and poo-pooing the outcomes.

Penetration testing should be a part of a wider set of security activities, a verification of all that came before. All too often, we come across clients who want to do a one or two day test the day before go-live. They’ve done nothing else, and when you completely pwn them, they’re terribly surprised and upset.

We need to move on to make penetration testing the same as unit testing – a core part of the overall software engineering of every application.

Penetration testing should never be ill informed (zero knowledge tests are harmful and a WAFTAM for all concerned), and it should have access to source, the project, and all documentation. Otherwise, you’re wasting the client’s money up against the wall and acting unethically in my view.

Tests should come from the risk register maintained by the project (you do have one of those, right?), as well as the use cases (the little cards on the wall) as well as from the OWASP ASVS / Testing Guides. More focus must be made on access control testing and business logic testing.

Penetration testing has become vulnerability assessment – run a tool, drool, re-write the tool’s results into a report, deliver. No! Write selenium tasks and automate it. If you’re not automating your pentests, how can your customers repeat your work? Test for it? They should be taught how to do it.

Folks at consultancies will shriek away in horror at my suggestion, but getting embedded is actually a good thing. Instead of hearing from a client once in a blue moon, you’re integrated into the birth and growth of software. This is a huge win for our clients and the overall security of software.

3 thoughts on “On penetration testing – harmful?”

  1. Agreed totally. Showing clients how to replicate tests is a key part of what we do at SensePost. The problem lies with the fact that many dev teams still aren’t sure as how they should embrace these tests.

    I cannot tell you how many times we’ve say through feedback sessions, given Selenium examples and still been told it’s not enough.

    There is a long road ahead, but at least a few consultancies are on it already.

  2. Your thinking on “unit testing” reminds me of another aspect of Threat Modeling that I like: If the model is built well then it raises a very broad and deep set of potential security ‘concerns’ – that is is theoretical risks that may need to be mitigated. Security testing can then be aligned directly with those theoretical risks. For example, for a web application, the Threat Model may raise a concern about SQLi due to poor input validation. The function of technical testing then becomes to find any instance of poor input validation in the code, whereby that risk would be proven real.

  3. Folk will shriek, because consultancy’s remit is so often dealt out as black ops or dark art, and perception from both sides of the value ($) leveraged accordingly. This is a bad thing for various reasons. I like your assertion of how to involve and educate (wrt to checking/retesting) and thus build ongoing relationship with client. It’s honest, and I think a lot of management and board would acknowledge that in the long run.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>