mostly useless crap from me

It’s been a slow haul working on converting all the database access to PDO. There’s just so MUCH crap in there.

So to keep things moving, I’m going to move www.gaiabb.com to run on the donated OLPCs at my home.

More details here.

I’ve been playing around with JSON recently, and I’ve discovered that most JSON implementations allow parameter pollution. This might be obvious to JavaScript experts, it’s not immediately obvious to most folks as JSON is just so much line noise.

{“varName”:value,”varName”:value2,”varName”:value3}

In the systems I’ve tried injecting, value3 is the one taken. Now if you have a hand crafted JSON decoder and coupled with a simple validator that only checks the first value, say a simple regex, you’re going to get past validation fairly easily. All the other caveats regarding parameter pollution apply.

Give it a try the next time you’re doing a gig and see if you can bypass validation and other rules. YMMV.

As we saw in the previous article, conversion to MySQLi is an awful lot of work. So let’s move to PDO.

Step 0. Get PDO working with your database server

Somewhere along the line, the PHP and MySQL folks decided to not be friends, so even though 99.99% of all PHP scripts require MySQL, in most cases, PDO doesn’t have MySQL support enabled.

Check that PHP doesn’t already have PDO for MySQL ready to go:

% php -i | grep -i pdo
Configure Command =>  '[...snip...]'
PDO
PDO support => enabled
PDO drivers => mysql, sqlite, sqlite2
pdo_mysql
PDO Driver for MySQL => enabled
pdo_mysql.cache_size => 2000 => 2000
pdo_mysql.default_socket => /tmp/mysql.sock => /tmp/mysql.sock
pdo_sqlite
PDO Driver for SQLite 3.x => enabled

You’re probably wondering about “dirty output sanitization”. Sanitization is for the weak minded! No seriously, my database is 7 years old. There’s crud in there – I can’t trust it to be safe or good. There’s some nice tools out there like Arshan’s Scrubbr to scan and clean a database, but Scrubbr is not going to be able to decode BBCode and check to see if there’s a nasty [color] based XSS or CSRF attack in there. Additionally, some versions of my software and some buggy versions of MySQL == binary blob crappiness. Blobs coming out sometimes (but not always) need to be stripslashed(). Additionally, some versions of XMB used client side checks to prevent files of the wrong type to be uploaded. I’ve found a JavaScript malicious file in my production database. So it’s worth adding checks so that I don’t serve bad things to folks. You can’t do this if you have 100+ attachment related SELECT statements alone, not without lots of bugs and lots of code. Going the DAO way, means I can do it once, for every single use of the attachment sub-system.

There’s gotta be a downside.

It’s not all sunshine and roses. You should not open both a PDO and old style connection to the database for busy servers like mine – it more than doubles the time to serve the average script, the poor little database server gets whacked, and performance will drop.

In GaiaBB, there’s a significant 7240 lines of shared code read in every single time a script does anything – header.php, functions.php, db.php, validation.php, constants.php, config.php, cache.php, english.lang.php, mail.class.php. So to only open ONE database connection requires these files to be ported to PDO first. But if I convert these files, every remaining single file totalling about 80,000 lines of PHP will also need to be converted.

So my solution for GaiaBB is to create a side-by-side approach, using kernel.php for converted files instead of header.php. kernel.php includes PDO ready files rather than the old files. This temporarily makes GaiaBB approach the 90,000 line mark, but in the long run, it will allow me to make many of the scripts smaller and more object orientated. Once converted to DAO, I can simply eliminate many security checks within the code of each page, as the DAO will simply throw an exception if you’re not privileged to do something with the data you’re trying to work with.

So my main piece of advice for you contemplating converting to PDO is to consider your data architecture. The old MySQL way is awful, buggy and insecure. Let’s exterminate it, but instead of standing still for months at a time, add in freebies like access control, audit, input validation and output sanity checking.

Okay, you’ve got like a zillion SQL queries in your PHP app, and probably 95% of them have a WHERE clause, and you need to make them safe so people will still download and use your app. Because if you don’t fix your injection issues, I will rain fire on your ass.

These are the steps you need to take to convert to prepared statements.

Step 0. PHP 4 is dead. Upgrade to PHP 5

All of the code in these samples is PHP 5 only. I will be using SPL and MySQLi, which are installed primarily on PHP 5.2.x installations. PHP 4 cannot be made safe, so it’s time to upgrade. This is non-negotiable in my view.

If you’re using register_globals, you have to stop. Do not use a function to register them for you in PHP 5, it’s time to do proper input validation. This will actually take you longer than converting all your queries to prepared statements.

To get a handle on this issue, what you need to do is:

1. Turn on error_reporting(E_ALL | E_STRICT);

2. Find all isset() and empty() and unless they are actually testing for a variable you’ve set, get rid of them. isset() and empty() are not validation mechanisms, they just hide tainted globals from view

3. Go to php.ini, and turn register_globals off. It should already be off

4. If your code has a construct like extract($GLOBALS) or some other mechanism to register globals, get rid of it

5. php -l file.php. This will give you a first pass which you will need to clean up

6. Use PHP Eclipse or PDT in Eclipse or the Zend IDE in Eclipse. This will give you warnings if you have uninitialized variables. Go to the properties, and make this into an error. Clean up all uninitialized variables

7. Start each script like this:

// Canonocalize

$dirty_variable = canonicalize($_POST['variable']);

// Validate

$variable = validate($dirty_variable);

// Use the variable

$stmt = $db->prepare("SELECT * FROM table WHERE id = ?");

$stmt->prepare("i", $variable);

$stmt->execute();

// and finally, if you need to output that sucker:

echo htmlescape($variable, $locale, $encoding); // $locale is probably 'en', and $encoding is probably UTF-8 or ISO 8859-1

Obviously, you need to canonicalize – that is make it the simplest possible form. If you have no idea about this extremely important topic, please consult the OWASP Developer Guide. Validation is essential. This replaces isset() and empty() and other mechanisms, with actual validation requirements. If you’re expecting an array of integers, make sure it’s an array of integers! If they have to be in a certain range, make it so. If the validation fails, put up an error message and do not proceed! This stuff is CS101, so please make sure you do this reliably for all variables without exception.

Step 1. Make sure your hoster has MySQLi

If your hoster is still running PHP 4, you need to see if they have the ability to run PHP 5. Most likely, your PHP 4 installation will not have ANY prepared statement compatible interface, like MySQLi or PDO. Of course, PDO is PHP 5 only… and it has a cool interesting feature – it emulates prepared statements for those databases that do not have support for it. But that’s for another post.

How do you check? Use phpinfo(). Create this small file somewhere with a random file name and upload it to your host:

<?php phpinfo(); ?>

Run the file, and note if you have the MySQLi interface. If you don’t, you can’t upgrade to prepared statements. It’s time to wage holy war on your hoster to make sure they install PHP 5.2.7 with MySQLi and PDO with MySQL 5 client libs for you … and all your other shared hosting friends.

Changing over to MySQLi

The simplest part of this process is to move to MySQLi from MySQL:

Instead of

$db = mysql_connect($db…

You have two choices: stay with functional MySQLi, or move to OO MySQLi. I think the latter is better, but that will be another post.

$db = mysqli_connect($db..

Now, this is where it’s important! You MUST check the value of $db for errors before continuing. You probably have this code today, but it’s important to realize that if $db == false, you didn’t get a connection.

if ( $db == false ) {

// Print up an error and stop

}

Simple Conversion

You may be tempted to just use the MySQLi extension, and move all your queries to place holder versions. That’s okay, but it can be a lot of work. Trust me, I’ve tried.

Although it seems like the easiest possible choice, converting MySQL queries to MySQLi’s prepared statements has a couple of issues.

Gotchya: There’s no easy way to bind lots of results when you use select *

With MySQL query, fetch_array will simply bind the current result set row to an associative array, and you can access it trivially. Most PHP apps use this data access pattern extensively, like this:

while ( $row = mysql_fetch_array($query) ) {

// do some work with $row

$blah = $row['column'];

}

Which brings us to the next gotchya:

Gotchya: There’s no fetch_array()

I don’t know why MySQLi does not have this most common of all the data access patterns, but it’s a right pain to fix. So let’s get a function to emulate fetch_array, including using anonymous field names as per above.

Okay, so we’ve decided that MySQLi sucks the proverbials… so in the next article, let’s talk about migrating non-trivial PHP apps to PDO.

Howard Schmidt has been appointed as the US’s cyber czar. The position has been open for months, which is … interesting … considering how vital IT is to the world’s economy and safety.

Mr Schmidt, if you read this blog entry, please consider the following:

• Web Application Security is the most pressing need for change. It’s the key to nearly all attacks today, and the least well funded. Help OWASP and others to improve developer education, get the message out to CIOs to apportion their training budgets and remediation efforts accordingly.
• Be positive not negative. The attackers really don’t care if you “keep your computer up to date”. Let’s work mostly on things that can stop the issue in the first place. The horses have already bolted. Let’s make a better stable and fences so they can’t get out again.
• Push for real security, not security theatre. Listen to Bruce Schneier, and not the profits of doom that want to sell you useless widgets for billions that do nothing but annoy folks.
• If you can do any change, the first change has to be removal of indemnity for negligence with software development from licenses and sales. If an ISV doesn’t have a security development lifecycle, doesn’t include secure business requirements, and doesn’t require its developers to be trained in security coding practices, it IS negligent, and must be open to lawsuits. What we have today is not working and must be changed.

Normally at this time of the year, I would talk about the industry’s achievements over the last year.

None. Zilch. Nada.

We’re seeing more SQL injection used in real world attacks than ever before. XSS is still with us, and one of the biggest offenders – PHP – has made zero moves to include proper encoding or encoding by default for echo and print, or including a safe by default generic SQL layer that is enabled by default and works with the three or four most common databases (e.g. MySQL). The adoption of PCI seems to have made little difference in the amount or severity of breaches.

Things like ESAPI, App Sensor and ESAPI WAF are the only true breakthroughs in 2009. But outside of OWASP DC, there’s no love for defences. Hats off to Jeff Williams and the entire ESAPI for * team, Michael Coates, and Arshan for the only true web app security efforts this year.

So let’s forget about 2009 and move on to 2010.

• Full disclosure / responsible disclosure / etc has failed (again) to improve security as it always has. We should stop doing it. Nearly every app has at least one or more of the OWASP Top 10 2007 / 2010 issues. It’s like shooting fish in a barrel or using dynamite to fish. Stop wasting time on it and come research how to put in safety by default in every language and framework, starting with woefully insecure frameworks and languages like PHP.
• Conference presentations about attacks are still getting all the sexy girls and media! Conferences and the media have to stop promoting attacks – it’s irresponsible and wasteful. Let’s start talking about defences instead.
• No more penetration tests! We have to stop doing penetration tests. They suck at predicting the safety of a system, particularly insider risks. Pen tests have value at mature clients who have done the hard work – an SDLC, secure requirements, secure development, peer reviews, code reviews, and extensive testing. They are a validation of the other security benefits, not as a “my X is bigger than yours” exercise and certainly not absolute proof of security.
• SDLC’s are still rare in the clients I visit. We need to encourage the adoption of SDLC, and require secure requirements.
• Agile still needs a lot of security as yet. User Stories still have no space for a security outcome in most environments. It’s hard to code review every milestone let alone every sprint. We as a security community need to do a lot more work here to fit in with the modern development methods in use.
• Developer training is still in the nascent stage and is the only workable method of producing secure apps by default. I donated my full two day deck to OWASP at the beginning of 2009, but as far as I can tell, it hasn’t been updated or given any love. I hope that can change over the year. Please go here and help make this deck the de facto developer training deck!
• We have to encourage or even mandate folks who outsource / out task / buy off the shelf software to only allow the acquisition of secure software, with the burden of insecurity firmly on the developer. Laws and licenses that prevent this must be changed as insecure software is not fit for purpose and thus defective. Obviously, there’s a huge difference between accidentally insecure and deliberately insecure software. If you don’t have an SDLC and a security program, an ISV is deliberately insecure and must face the costs of their negligence.
• Over-reliance on silver bullets (WAFs and so on) is harming the effort to fix the problem. Silver bullets don’t always work, and eventually, you’ll have to do the right thing. I don’t know what we can do here but yell at the sky as the marketing dollars for these things overwhelm that simple message.

Let’s not waste another year. Let’s get moving on secure defenses, SDLC, R&D in agile technologies, and developer education.

It’s Inbox Zero time again. Every year, I do the Inbox Zero thing and archive all my mail (read and unread) on January 1 from the year just gone. I also tell myself it’s time to start following the IZ rules, but … they somehow always fall to the wayside.

I get a lot more personally addressed e-mail than most folks do, and I don’t have a chance to action every item that needs it. I know I have missed replying to at least 20% of my mail – my bad. To make amends, I will work on replying to the last three months of outstanding mail by the end of the year. But I bet there will be mail that still needs actioning that will be archived.

Action Required: If you e-mailed me, and not had a reply from me by January 1, please re-send your mail to me after January 1, and mark it REPLY NEEDED in the subject. I have an e-mail rule that flags such messages and I will reply to you.

Today, the Labor Government, pandering to a tiny minority of voters who will NEVER vote for them, will proceed with censoring our Internet.

Many of these hard right wing “Christian” (who obviously missed the entire point of the New Testament) “voters” (Exclusive Bretheren, etc) do not have computers let alone TV’s or newspapers to be offended by the Internet. Worse still the Bretheren are some of the only people in Australia who are allowed not to vote. And for their vital electoral “support”, we all get censored. WTF!?!

FUCK NO!

Today, I start censoring the Internet for Australian Government departments. If your DNS name ends in “.gov.au”, there’s a pretty good chance you’ll not be able to see this site and the other sites I run. E-Mail from .gov.au sites will be delivered to /dev/null. In future works I create, I will make an explicit disallowance preventing Australian Government public servants and contractors from using my materials until the censorship mechanism comes down. I will encourage everyone I know to put up mandatory “.gov.au” filtering. See how you like it when the Internet is useless to you and you have to use personal Internet connections to get anything done.

I will fight this censorship scheme in every way I can. I will publish mechanisms on how to bypass it. I will encourage people to defeat it, even if they don’t have to. I will campaign against my local ALP member. You’ve made a political activist out of someone who used to just rant about politics around the water cooler. I am not the only one. Labor is doomed for a generation or more by this one heinous act.

Labor – shame shame shame. I’ve voted for you – stupidly it turns out – for my entire adult life. I’m sorry, but I’ll vote for Donald Duck before I grace your lice ridden corpse with the “1″ mark ever again.

Conroy – he who shall not be named from here on – you have are the Internet’s Public Enemy #1. You have cost Labor the next election, even with the Liberals in complete disarray. Labor cannot ever trusted to govern ever again.

Well, the Emissions Trading Scheme is dead – for now. Yay! I do a little dance on its grave. We’ll have to fight it when the double dissolution election comes up sooner than later.

However, I wasn’t expecting the mad monk, Tony Abbot, to gain the Liberal leadership. That was a surprise, as I bet it was to the majority of the Liberal party MPs.

With such a right wing, homophobic, anti-abortion, anti-pretty much anything we’ve achieved over the last forty years to several centuries, and top of that a truly hard core Catholic elected leader by the thinnest of margins (1 vote – a donkey vote *), the Libs will be in electoral wasteland for at least one and probably two more elections. Either the Libs will have to split into the electable bit and the unelectable’s, or they will have to try again in a few years after they get rid of Abbot.

Abbot is simply unelectable – even my wife who leans in the Libs direction doesn’t like him. Sure, Abbot will make the hard core religious and climate deniers happy, but they’re a tiny minority here – and they already vote Liberal. All the moderate swinging voters – they who elect our governments – will abandon ship once they realize just how backward Abbot is on so many things.

With Abbot being the mental giant that he is, he’s going to oppose pretty much all Government bills. I bet he opposes a really stupid little bill and that’ll be the trigger. KRudd could phone it in and win.

Bring it on – maybe enough of the disaffected voters will move to the Greens and we can get some real carbon reduction instead of the reward-the-polluters ETS.

* I bet the idiot ^H^H^H^H^H Member of Parliament who cast the deciding donkey vote (‘no’) is regretting their ineptitude tonight. The silly thing is that the vote was almost certainly cast by a moderate Liberal. That moron has ensured they stay unelected for at least another four and most likely seven years.

Unlike the deniers in the Liberal party, I understand climate science well enough to know that we should give our only planet the benefit of the (very little) doubt. It’s time to act. But not with an ETS. I hope that the Liberals (== conservatives, for my US readers) defeat the ETS a.k.a Carbon Pollution Reduction Scheme (CPRS).

The heart of the problem is that the Emissions Trading Scheme doesn’t help to reduce pollution. Why? ETS Traders have no skin in the game – you don’t have to be a polluter or seller to participate. Why would those traders be interested in carbon reduction. Over time, the value of the market will go up due to speculation and moves by the traders, making it more expensive for the Australian Government to buy back emissions credits to reduce the total emissions pool, or even worse, short changing the folks who need to acquire those credits. The folks who buy these credits on the open market will need to pay more, and we pay double through increased taxation and higher bills for pretty much everything even if you’re doing the right thing.

The Coalition have introduced a bunch of get of jail free cards to the heaviest polluters to provide their denying colleagues some carrots.

• Coal fired power plants are largely exempt, despite emitting about 50% of Australia’s total CO2 emissions
• Heavy users of power have tax credits to help pay for their credits, often up to 90% of the value of them or even free in the case of aluminium producers. Where’s my 90% reduction in my electricity bill? This is corporate welfare at the worst
• Agriculture has a wide range of exemptions, despite many inefficient processes that could benefit from better alternatives. They also get money for carbon offsetting, so in reality, they can be paid for sequestration activities, but have no economic harm from releasing that captured carbon. Way to go to buy the rural vote, Rudd.

So no matter what I do to reduce my carbon footprint, it will have little impact, as the largest polluters can simply keep on going on doing exactly what they’re doing today. I – and all Australians, even if you’re off the grid, grow your own food and don’t drive or fly will end up paying for this dumb scheme.

The Government should not distort an entirely new unproven market. Let it distort the current market:

• Announce the Government will only buy electricity from renewable sources as of 2015 or so
• Announce no more coal fired power stations will be built and approve nuclear power stations
• Set power consumption targets for the heaviest power users in the average business and house (computers, lights, fridges, ovens, aircons, etc)
• Require standby to be < 0.1 W (or it’s off), and prohibit clocks on things that don’t need them (like microwaves, fridges, ovens and toasters) so they can turn off when not used
• Ban crappy computer PSUs and require 80-Plus only PSUs. Make rackable servers like Google’s – no PSU in the device, and the power supply is > 90% efficient.
• Ban non-LED downlights (also have a positive impact on # of house fires from cheap iron core transformers setting fire to insulation)
• Fund or provide serious rebates for solar hot water for everyone with an electric water heater.
• Fund or provide serious rebates for passive solar cooling for every home, rented or owned.
• Continue the serious rebates for solar panels, and extended it to rented and owned properties.
• Required states to tax the hell out of cars that chew more than 7.5 l/100km
• Only buy cars with average fuel consumption of less than 7.5 l/100km from now on – there’s hundreds of thousands of cars in the government car fleet
• Mandate employers allow telecommuting where possible. This would eliminate hundreds of thousands of wasteful trips every day, and free up freeways for freight and necessary journeys. I enjoy my ten second commute and I don’t have to start the car most days.
• Provide incentives to get road freight back onto rail
• … anything other than an ETS

Trading schemes (like NEMMCO) have a proven history of epic failure. In California, traders caused widespread blackouts and damage not to mentioned sky high electricity bills. There is no incentive for an ETS to reduce carbon pollution. The market relies upon carbon being emitted. It will fail, not reduce CO2 emissions as the largest polluters don’t have to participate properly, and cost us billions.

ETS == Epic fail with our future. Bring on a double dissolution election.