Hacking a huge Drupal database import

This article describes how to use regexes and some bash tricks and bad voodoo to split a SQL file into pieces in order to get it into MySQL in a reasonable amount of time. Parsing SQL is notoriously painful and error prone, so if you see mistakes here (I'm sure there are), please comment!

I recently got called in to work on a project for a customer I had never worked with before.  It is a very well known media / retail company and the job was to build a new interface for a media player / video library they required.  Pretty standard stuff, some views a couple quicktabs, a bit of ajax coding and a lot of pixel-f**king.  Anyway, near the end of the week long project when it came time to stage it, I realized I had a pretty big problem.  This site had 2 million users and several tables of profile data and other information.  My SQL file was over 3GB and took 6-8 hours to import.  Eek!  

So knowing that I wasn't going to write my update functions 100% perfect the first time, and I would likely have to import several times until I was confident it was working okay, I needed to find a way to get the DB to import in a more reasonable amount of time.


Review: Sequel Pro

I've been using PhpMyAdmin since I switched to a Linux desktop about 7 years ago.  Before that I was a big fan of Mascon for windows.

I'm not sure why I tolerated the lousy web interface of PhpMyAdmin for so long.  Fully aware there were more usable solutions out there.  I guess it's just a case of "whatever, I don't use it that much".  But I think a good DB browser can really improve workflow in some situations.  Now that I'm a sellout and use a mac, I was expecting to spend $30 for some silly GUI app, but was pleasantly surprised to find SequelPro.


Sequel Pro


Nice right?  And it is really easy to use.  I can edit data, it's got a SQL editor that is smooth to work with and a table designer with all the fixins.  Supports local and remote DBs as well.

While I still support phpMyAdmin (as a stalwart x-platform FOSS citizen) this one is just too easy to use and pretty (and also Free software).

Check it out!


Subscribe to mysql