Archive for the ‘olpc’ Category

How to migrate to PDO without it hurting… much

As we saw in the previous article, conversion to MySQLi is an awful lot of work. So let’s move to PDO.

Step 0. Get PDO working with your database server

Somewhere along the line, the PHP and MySQL folks decided to not be friends, so even though 99.99% of all PHP scripts require MySQL, in most cases, PDO doesn’t have MySQL support enabled.

Check that PHP doesn’t already have PDO for MySQL ready to go:

% php -i | grep -i pdo
Configure Command =>  '[...snip...]'
PDO
PDO support => enabled
PDO drivers => mysql, sqlite, sqlite2
pdo_mysql
PDO Driver for MySQL => enabled
pdo_mysql.cache_size => 2000 => 2000
pdo_mysql.default_socket => /tmp/mysql.sock => /tmp/mysql.sock
pdo_sqlite
PDO Driver for SQLite 3.x => enabled
The default socket location is fine for a development workstation, but not so good for a production host. Change php.ini and my.cnf to make it safer, such as /var/run/mysql/mysql.sock
If your PHP installation doesn’t show the above, whip out the compiler – there’s plenty of articles on the Interweb on how to get PDO and MySQL going. For now, let’s assume PDO and MySQL are good to go.
Step 1. Conversion
As in our previous article, it’s very tempting to just go through each file and make a 1:1 change from MySQL to PDO. That is actually a losing scenario.
  1. My non-trivial app has 1853 SQL queries littered through the code. At about 15 minutes to 30 minutes per query, with no test case, that’s about a year’s work with no change to the overall complexity of the app nor any improvements to the overall data architecture. Changing from one to the other is sure to introduce bugs.
  2. GaiaBB only has 13 tables. Using a DAO approach, with a list and search helper method for each (i.e forum list, search forum), that’s 6 * 13 = 78 properly written, tested DAO methods that need to be written from scratch. This is a saving of over 10 months work compared to just getting in there and getting my hands dirty.
  3. I can add security business requirements once to the DAO, such as fine grained access control, input validation, audit, and dirty output sanitization, thus fixing all my access control issues and dirty data issues in the same small piece of code. So each file that is converted to DAO gets more secure with no additional coding

You’re probably wondering about “dirty output sanitization”. Sanitization is for the weak minded! No seriously, my database is 7 years old. There’s crud in there – I can’t trust it to be safe or good. There’s some nice tools out there like Arshan’s Scrubbr to scan and clean a database, but Scrubbr is not going to be able to decode BBCode and check to see if there’s a nasty [color] based XSS or CSRF attack in there. Additionally, some versions of my software and some buggy versions of MySQL == binary blob crappiness. Blobs coming out sometimes (but not always) need to be stripslashed(). Additionally, some versions of XMB used client side checks to prevent files of the wrong type to be uploaded. I’ve found a JavaScript malicious file in my production database. So it’s worth adding checks so that I don’t serve bad things to folks. You can’t do this if you have 100+ attachment related SELECT statements alone, not without lots of bugs and lots of code. Going the DAO way, means I can do it once, for every single use of the attachment sub-system.

There’s gotta be a downside.

It’s not all sunshine and roses. You should not open both a PDO and old style connection to the database for busy servers like mine – it more than doubles the time to serve the average script, the poor little database server gets whacked, and performance will drop.

In GaiaBB, there’s a significant 7240 lines of shared code read in every single time a script does anything – header.php, functions.php, db.php, validation.php, constants.php, config.php, cache.php, english.lang.php, mail.class.php. So to only open ONE database connection requires these files to be ported to PDO first. But if I convert these files, every remaining single file totalling about 80,000 lines of PHP will also need to be converted.

So my solution for GaiaBB is to create a side-by-side approach, using kernel.php for converted files instead of header.php. kernel.php includes PDO ready files rather than the old files. This temporarily makes GaiaBB approach the 90,000 line mark, but in the long run, it will allow me to make many of the scripts smaller and more object orientated. Once converted to DAO, I can simply eliminate many security checks within the code of each page, as the DAO will simply throw an exception if you’re not privileged to do something with the data you’re trying to work with.

So my main piece of advice for you contemplating converting to PDO is to consider your data architecture. The old MySQL way is awful, buggy and insecure. Let’s exterminate it, but instead of standing still for months at a time, add in freebies like access control, audit, input validation and output sanity checking.

The OLPCs are here

Wow –  that was quick.

I have three OLPCs in my office. I need to go sort out some US power plugs for them so I can charge them, but they’re here! :-)

Proto GaiaBB runs fine in Browse

Phew.

Screen shot 2009-10-21 at 11.33.00 PM

Obviously, the default theme fails a bit when you see only two rows of topic activity. Will really need to make that work a whole lot better. However, the actual rendering is perfect. Yay! One less task to be done.

GaiaBB and OLPC

Peter Quodling. an old friend, e-mailed out of the blue last week. I have a lot of time for Peter as he’s one of the few Australian IT architects that really knows his stuff, plus he’s a really nice guy. He is involved in OLPC in the PNG region. Last Christmas, I nearly bought an XO under the Buy One, Give One program so Mackenzie could have a cool first laptop … and somewhat more honestly, so I could play with the OLPC until she’s old enough to type let alone talk. However, circumstances prevented toy purchases of that magnitude, and I forgot all about the XO until this week.

I thought through my various abandoned projects (for I have many), trying to work out which would help the OLPC project and kids all over the world. I’ve had an itch for a while to do something for the One Laptop Per Child project (OLPC), but never really had a substantial idea that would help transform kids’ lives rather than my own. But now I think I have just the project.

So I put in a proposal for two laptops to help develop GaiaBB (which is UltimaBB++ (sic gloria transit), which is XMB++, which is awful) into a OLPC specific product.

My current plans are:

  • Get GaiaBB.com back up and make it OLPC centric. Revitalize the Sourceforge project.
  • Finish OWASP ESAPI for PHP. I need it for this project.
  • Port GaiaBB to the OLPC, porting the database to sql lite, and probably using LightHTTPd. I could use Apache + MySQL as per now, but these are huge compared to SQL Lite and LightHTTPd, and on a device with limited NAND memory, every byte counts.
  • Ensure GaiaBB works properly with Browse, the XO browser. I might need to turn the nested tables into CSS templates a bit sooner than I intended
  • Beg, borrow or steal a graphic designer to come up with a GaiaBB theme that works well with the dual color display (it’s both black and white and color and more wide than tall), and possibly work out how to detect the XO’s current screen state from the web page so I can dynamically choose a grey scale or a color theme.
  • Simplify the product so that it’s more manageable by young ‘uns without dumbing down too much or removing some of the depth and surprise features of the product
  • Write an installer that makes it easy to install on the OLPC. I want kids to be able to create their own social communities and for them to easily share their forums with their friends.
  • And this is where it gets fancy… write a web service that allows authorized folks to replicate their version of the forum with other versions of the forum … without causing major security issues. That way when you’re at home and have no Internet service, you can still read the forum and write new posts and then sync when you’re at school. This is where I will have to significantly change the way GaiaBB works as right now, it’s a single database deal and assumes that there are trusted administrators.
  • Go through the code with a fine tooth comb, replacing all the crappy security bits with ESAPI for PHP. Some parts are truly ancient (circa 2000) and need refactoring. As part of this, I will ensure the code is easily modifiable. I learnt how to code by changing other folks’ code and then starting to write my own, and I want kids to modify the heck out of this forum so as to create a new generation of coders excited about programming and IT in general.
  • Lastly, possibly write a OLPC School Server centralized GaiaBB hub for schools to run “their” forum which students can sync with in a safe fashion.

The proposal has been approved, and the laptops are on their way. They are sending me three XO’s! Awesome. Better get cracking!

Return top

Say no to censorship - No Clean Feed!

This page is now black to protest the Australian Government's decision to censor the Internet. Censorship is possibly the most un-Australian act of all. Please write or call your local member and senators immediately to express your displeasure. Go to rallies. Twitter #nocleanfeed regularly. Blog. Facebook. Support the EFA. Vote for anyone but Labor. We must defeat this evil bill for our children's sake. Most of all - mass civil disobedience is vital.