Jeremiah sent me a survey to fill in. Normally, I don’t like participating in surveys, but this time I made an exception. Jeremiah noted that my responses, although not quite in the boxes he had set out, were still actually pretty useful.
So here are my responses:
1. How many code reviews did you do in 2006?
I do a few but very large code reviews, each involving more than 100,000 lines of code. So although not high in number, the programs process literally billions of dollars in transactions every day. Therefore, extreme care needs to be taken. I am not a automated scanner boy and would be negligent if I only used a tool like PMD or LAPSE to find my findings.
2. What reporting standard do you use?
Jeremiah’s choices here did not include many of the normals, including CWE / CVE from Mitre, OWASP anything (that said, Jeremiah has his own biases to WASC), etc. We also have regulatory regimes on top of webappsec specific lists, which are also not mentioned.
I’m not sure of the validity of this question except to say that it should be the subject of more research.
3. Do you use commercial application scanners during security assessments?
Actually, no.
I use PMD, Find Bugs, and LAPSE, all open source or freebie tools. They are for extreme low lying fruit, and in many cases, like not using “final” or “const” I never report on some of these findings as they have zero security impact.
4. Average number of man-hours required to perform a thorough web application vulnerability assessment on the average commerce website?
This should have been phrased to be PC “Average number of hours per review” as I know some hot chicks and some excellent queens working in our field. 🙂
I do > 100 kLoc code bases, I was happy to see that folks are spending more than a week doing code reviews. I dare anyone to do a code review on a system which has > 40 systems it talks to directly, with over 200 seperate value functions and over 100 types of data assets in a week.
Typically, for J2EE, I use the initial kLoc (as reported by sloccount) divide by 1000 to be the number of days and fatten the result by 25%. This works out most of the time. However, a revent Aspect Oriented Programming review using Spring Web Flow blew that estimate out of the water. 5000 lines took 2 weeks. ARGH. It pays to know your technology before you quote on a estimate, particularly if you’re doing fixed price code reviews.
5. Do you recommend Web Application Firewalls?
No.
Unless the organization is a CMM level 5 organization that has nothing else to do and needs a new challenge. Seriously, unless the organization is able to tailor the WAF to the application and keep it up to date, WAFs, particularly appliance (=usually dumb) send the wrong message: that’s there’s a silver $25k bullet to your security problem. This is not the truth and I will not perpetuate it. In addition, such devices nearly always add complexity and add fire to the response | request splitting harm which is real and unavoidable when you add unnecessary devices.
But an organization who sees it as defense in depth control, and is prepared to look after it, and investigate and escalate real problems rather than treat it as a “set and forget” will get a recommendation from me for a serious WAF tool, such as mod_security or similar.
I’ve used mod_security to prevent DDoS against a customer a few years ago, and used properly, WAFs are an invaluable asset. But plonked in and forgotten, they are worse than useless – they give a false sense of security and cost a bucket of money that could have been used for a code review. Most (>90%) organizations in my view are simply not mature enough at IT security to look after them and thus should not use them.
6. What do you think about the updated PCI Data Security Standard v1.1
It’s a good start. However, in the latest edition automated scanner vendors are rubbing their hands with glee. We’re going to have SMEs pay a scanning firm for a clean bill of health (“We do the OWASP Top 10 as the PCI requires” — no you don’t, some of these issues are NP complete problems), and thus will get attacked by a business logic error, or a process error which scanners CANNOT find.
I’m happy to work with PCI to fix up the next edition, but honestly, the most recent release is just better than before.
7. Checking for XSS on public websites without permission?
This is extreme grey area and I lean towards “illegal”.
My personal take is that now that methods are well known to craft really bad JS malware, that poking a public website without authority is just dumb. Don’t do it. If the sites are based upon a public piece of software like UltimaBB or phpBB, sure, go ahead download the software and test offline. That’s what security research is all about. But don’t prod or take out public websites.
In Australia, the computer crimes act and complementary state laws are deliberately vague to allow the book to be thrown at you. If you’re a nuisance, the terms of “unauthorized access” are so vague as to mean you are up a certain creek without a paddle if the owner takes offense. And it’s criminal, not civil trouble you’re in. Police are strapped for cash, and if they think they can obtain publicity and an easy conviction, they will come after you. That gets them more funds and resources if they are successful.
Here’s the actual text. You decide:
Computer trespass. "9A. A person must not gain access to, or enter, a computer system or part of a computer system without lawful authority to do so. Penalty: 25 penalty units or imprisonment for 6 months."
Daniel Cuthbert, an excellent OWASP contributor, was prosecuted and convicted under the much more nebulous UK Computer Misuse Act for having a go at a charity’s website. He now can’t emigrate to Australia, and had difficulty finding work in his chosen industry. Do not try this at home.