Archive for August, 2012

My gmail box is full. Well Almost.

I’m at 24.5 gigs! I think it goes without saying that I get a lot of email. I’ve always been an email packrat, but when I switched to gmail, I latched onto the “gmail method”, which is to archive instead of delete. So as a result, my email storage space has skyrocketed. I’ve been close to running out of space for a couple years now, but I’ve always just gone through and deleted repetitive and chatty threads, large attachments I don’t need anymore, etc. But at some point, the low hanging fruit is gone. I’m at that point.

I would like to migrate all of my old email to a dedicated “archive” account so I can horde use it for reference later. I tried using the Google Apps Exchange Migration tool, but it kept timing out. So did the mail fetcher utility provided by google — which btw ONLY works using POP3, and doesn’t transfer labels. I also tried using Thunderbird to download all my mail and copy it over, but my 1 million+ emails caused the thunderbird UI to puke whenever I tried to do ANYTHING.  I spent hours on the phone with google support, and nobody there could help me. They tried, and they called me back a whole bunch of times, but they were powerless — which was a real eye opener for me. I assumed they would have a magic button they could click, and *poof* all my email would be migrated to another account.

Anyway, I found an application named GMVault. It will slowly and methodically download all your mail, and store it in a giant non-database (btw, it runs on linux, windows, and mac OS X — Score for interopability!). It took about a week to download all my mail, but its down. So now I go to upload it, and I get the error:

Invalid Arguments: Label name is not allowed: Migrated

Obviously at one point I had migrated mail INTO gmail (I can’t blame my email hording exclusively on the gmail archive feature — I migrated mail from the 90’s IN to gmail… so clearly I already had a problem — I also have all my Yahoo! IM, ICQ, and IRC logs going back to the early 90’s — yes you read that right – ICQ and IRC logs from the 90’s.. I have a problem). Anyway, so apparently Gmail has the exclusive worldwide rights on assigning things the “Migrated” label — a simpleton like you or I cannot do such a thing.. and so when I go to restore to my new archive account using gmvault, I get the above error message.

Anyway, on to how I fixed it. I certainly didn’t want to spend another week downloading all my mail again (btw, while I was downloading all my mail, I noticed the UI for my gmail account got much slower — which was already slower than all my friends without a archive addiction). The fix is really easy actually since gmvault stores all of the metadata for every single email in a file with a .meta extension! Score for simplicity!

find ~/gmvault-db/db -type f -name '*.meta' -exec grep -rl "labels\":.*\"Migrated\".*thread_ids" {} \; | xargs sed -i "" 's/\"Migrated\"/\"MigratedAgain\"/'

Thats it! You’ve now renamed your “Migrated” label to “MigratedAgain”.  I was using my mac mini to run gmvault, and running this command took about an hour to recursively search through my million mail archive. Much faster than re downloading. P.S. I’ve noticed in the past that wordpress tends to screw up quotes when pasted into the editor — If you get syntax errors please use your mad skills to try to figure out what I actually meant. I’m sorry. wordpress sucks.

A few notes about the above command:

* I assume you have installed the gmvault db in the default location — which is under your home dir

* I assume you are using linux or Mac OS X — I bet you could make this work with cygwin on windows — but if you are of the cygwin persuasion you probably didn’t need me to suggest that to you.

* It is possible there are some holes in the above regular expression and subsequent search and replace. I didn’t spend an incredibly amount of time reverse engineering the metadata format of gmvault — In fact it took me longer to write this than it did to actually figure out how to work around this issue. I tried it. it worked for me. YMMV.

On another (somewhat related) note. I now have a new method for not running out of space. I forward all of my email to a group called “”, then in that group, I can swap in any account that I want to send my mail to — for example “” Then when 2013 rolls around, you can just create a new account, start sending mail there, and delete everything from 2012 in your primary gmail account. I don’t want to have to do this again.