Hacking a huge Drupal database import

This article describes how to use regexes and some bash tricks and bad voodoo to split a SQL file into pieces in order to get it into MySQL in a reasonable amount of time. Parsing SQL is notoriously painful and error prone, so if you see mistakes here (I'm sure there are), please comment!

I recently got called in to work on a project for a customer I had never worked with before.  It is a very well known media / retail company and the job was to build a new interface for a media player / video library they required.  Pretty standard stuff, some views a couple quicktabs, a bit of ajax coding and a lot of pixel-f**king.  Anyway, near the end of the week long project when it came time to stage it, I realized I had a pretty big problem.  This site had 2 million users and several tables of profile data and other information.  My SQL file was over 3GB and took 6-8 hours to import.  Eek!  

So knowing that I wasn't going to write my update functions 100% perfect the first time, and I would likely have to import several times until I was confident it was working okay, I needed to find a way to get the DB to import in a more reasonable amount of time.


Alias a.ka. Alias a.k.a BashAliases

I found a cool little tool today for building bash aliases.  

Being the lazy SOB that I am, I'm just going to quote the project page since it describes well why it exists:

This is all well and good but ... the time and forethought required to create these definitions [aliases] is often a significant barrier that is difficult to overcome. Even though the time it would take to create an alias would save much time in the long run (and much stress on hands and wrists and arms) the task at hand always seems to take priority. And so one rarely does.


Here's a quite run down of a couple commands:

acd Alias Current Directory


$ cd ~/work/somewhere/deep/in/my/sourcecode/where/I/go/all/the/time
$ acd alias for /home/jacob/work/somewhere/deep/in/my/sourcecode/where/I/go/all/the/time?rsi
$ cd /
$ rsi
$ # Goes to the directory.

The alias is saved in ~/.bash/alias.

Also has alc

Alias Last Command

This one basically let's you choose from the last 9 entries in your history and make one an alias.


Free Ideas: Paper clip guy for your bash CLI

Warning for Non-nerds: you are not required to read this blog post.

I was chatting Barry Jaspan this morning about how he switched from tcsh to bash after 22 years!  It turns out the features he stayed with tcsh for had been available in bash for years.  He just didn't know how to use them, or didn't know they existed and couldn't be bothered to learn.  This is very typical.  I work at about 75% or less efficiency on the command line (something I often spend 1/4-/1/2 my day doing).  It's hard to try and figure out what tools you need to be a little more efficient and have a more pleasurable command line experience.  And then retaining that knowledge (especially for the more arcane ones) is a difficult task.

So what about:

A M$ Office paper clip dude for your terminal

You've retyped something 5 times he pops up with "Do you want me to show you how to create an alias?"

but make it a really geeky and pretentious fat linux dude w/ a beard

like "You obviously can't RTFman! You don't know about {,shell expansion} or do you just like to waste time?"



Whattya think?  Should we do it?


Subscribe to bash