icculus.org: cvs.icculus.org has passed on from this world after a long battle with obsolescence. He was seven. cvs.i.o died in the comfort of his rackmount server, colocated in Chicago, in the early hours of March 31st. He was employed as a revision control manager, serving thousands of open source developers. He is survived by his children: Subversion, Mercurial, and Git. The family has requested that a donation be made to charity in lieu of flowers. Other stuff: This is a really really long, boring post about mailing lists. If your dog is whining, you should walk him before sitting down to read this, or he's gonna mess on the carpet by the time you're done.So, we have a new mailing list manager now. We were using ezmlm (specifically, ezmlm-idx), and it sucked for a lot of reasons. Now we've got GNU Mailman, which resolves all my complaints with ezmlm, and adds a bunch of really nice features I didn't even know I wanted at the time. I waited a long time to do this conversion, because I thought it would be really painful. It turns out, it wasn't, so for the sake of the next poor soul that googles for "migrate ezmlm to mailman," here's some information they might stumble upon. First, I thought I'd have to ditch qmail. I didn't. Dropping qmail would be way more painful than dropping ezmlm. The integration with qmail was dirt simple. Basically, I just installed Mailman, from source, following the directions in the manual. Ultimately, you end up setting up a few .qmail files per-list, and reserve some URL space on your webserver ("/mailman" and "/pipermail", to be exact). Sane people will just "apt-get install mailman" or something, but this broken-ass system didn't have that luxury (we're solving that with a nice, clean install of Ubuntu in the near future, thank you very much). Building lists is automated with a shell script. It creates the list from the Mailman command line interface, updates /etc/aliases, tweaks some default list settings to my liking, and sets up the .qmail files, so incoming mail for the list goes to Mailman for consideration. This works nicely for my setup, but of course you can build lists from the web interface, too. Please be careful, though: it will delete too many .qmail files in one case: we had quake3@icculus.org and quake3-bugzilla@icculus.org lists, and after converting the buzilla one ("rm .qmail-quake3-bugzilla*"), we lost some .qmail files on conversion of the main list ("rm .qmail-quake3*"). The real fun comes from migrating the existing mailing lists. First, we need to migrate existing subscriptions. To limit end-user confusion about the migration, I would send a note to each list before the move, explaining what was about to happen. Some of them I had to subscribe to before doing so, so the message wouldn't bounce. After about 10 iterations of this, I decided to automate it with a hacky script. Yes, it's a shell script that calls a perl script. It uses ezmlm command line tools to manage subscribing me for one post, if necessary, and qmail-inject to send the message. I'm not proud. It's messy. Get m.sh and m.pl, EDIT THEM FIRST so they don't email me, chmod +x, and then run "./m.sh $listname" ... consider everyone notified. Now, it's time to make the move. Build the Mailman list, as above. A side effect of the script is that it wipes out the existing .qmail files that tossed mail to ezmlm. Once that is done, the new list is accepting mail, and will bounce any incoming email from existing subscribers until you migrate them with this simple shell script. #!/bin/sh ezmlm-list /var/qmail/alias/$listname >/tmp/$1.subscribers /usr/local/mailman/bin/add_members -r /tmp/$1.subscribers -w y -a y $1 rm /tmp/$1.subscribers This sends every email address on the list a note with the new list details, as if they had just subscribed, and then a list of all email addresses to the list owner (presumably, you). The note to the subscribers tells them where to find the list's management page and their login details. It's all the pertinent info, which they'll probably delete without reading. This is why Mailman resends a reminder once a month by default. At this point, subscribers continue to use the list as before at the same email address, except they now have a nicer web interface for managing their subscription. You aren't done with the subscribers yet. Almost certainly you have some undesirables in there. Spammers and scammers tend to send email from "joe job" (that is, fake) addresses. If you ever saw your friend's email address on a spam addressed to you, this is what happened. Most of my mailing lists had addresses from PayPal, eBay, and Amazon (plus other spammier-looking things). What happened is a slimeball of some caliber sent a joe-job email to a mailing list from, say, support@paypal.com. Not someone that works for PayPal, mind you; that's a different kind of slimeball. Ezmlm manages subscriptions through virtual addresses: if you wanted to be on physfs@icculus.org's mailing list, you'd send an email (ANY email!) to physfs-subscribe@icculus.org and it'd get you set up...namely, it'd tell you "just reply to this email so we know it's really you and we'll start the subscription." Yeah, you can see where this is going already. Joe-job comes from support@paypal.com to physfs-subscribe@icculus.org. It's just a spam email, but ezmlm doesn't care what's in your initial email at all (we eventually started spam-filtering the *-subscribe addresses, but SpamAssassin still gets a percentage of false negatives anyhow). Ezmlm responds to joe-job by replying to support address, even though it didn't really originally come from there, with a helpful message like, "okay, just to make sure your friend isn't playing a prank on you by subscribing you to the list, just reply to this email without changing the subject line, okay?" It was a noble intention, and largely, it keeps out joe-jobbers and pranking friends. In normal cases, this is the end of the conversation: for prankers, the person would just delete the email. For joe-jobbers, likewise, if the address even existed. Otherwise, ezmlm handles the bounce fine. Some addresses, like dear support@paypal.com, however, would autoreply to the ezmlm response with a "Thank you for contacting PayPal support! Your business is important to us, so sit tight until we get to your email!!!!!!1" ...and they would leave the subject line intact. Ezmlm would see this definite non-bounce with the magic subject line and subscribe a bogus email address to the list. Not only can spammers now use this address to post spam to the list (in practice: rare, even before we spam-filtered all incoming list traffic, subscriber or otherwise), but now you might have every posting going to paypal, to which it replies KTHX4WRITING!!! ...and now you have a feedback loop. Or worse, you don't have one. Now there are bogus addresses on the list no one notices. Until they do. I didn't automate this, I just went through every list and deleted email addresses that looked suspicious (and a few that were questionable; if you didn't look like a human chose your email address, you probably deserve to realize one day that your subscription quietly vanished). But 9/10ths of the culprits could be found by grepping for "amazon", "ebay" or "paypal". At least the Mailman web interface made this easy enough to clean up by hand. Okay, now you're functional! You can quit here if you don't care about the mailing list archives. But I did. Converting them to Mailman archives was easy: #!/bin/sh ezmlm2mbox.pl --archive /var/qmail/alias/$1/archive --mbox /tmp/$1.mbox /usr/local/mailman/bin/arch --wipe $1 /tmp/$1.mbox rm /tmp/$1.mbox I don't remember where I got ezmlm2mbox.pl, but here is my copy of it. It just builds an mbox file from your ezmlm archives. Then we use a standard Mailman tool to import the mbox file. Done! Now, I have to confess something about myself: I am completely OCD about preventing broken URLs. If I move something, I try to find some way to keep the old URL redirecting to the new one if possible. ezmlm has a cgi-bin program for web access to mailing list archives. It's ugly, and did I mention it's some nasty C code that requires cgi-bin? It's completely awful in every way. But there are a lot of direct links to various list postings out there, and I didn't want them all to break. I briefly considered making any ezmlm URL just point to http://icculus.org/mailman/listinfo and let the user try to dig out what they wanted, but a trivial Google search showed there were too many people saying "this was an interesting comment over here: link!" without any context. Which list do you go to on the page full of lists? I also considered just leaving the cgi-bin program in place. It would keep all the URLs functional, but at the cost of gigabytes of disk space (since all the ezmlm archives would have to remain, despite another copy existing for Mailman) and having it looked like all conversation ceased the day we migrated...and having it look like ezmlm-cgi's output. This would not do. So I wrote some code. Now legacy URLs still work, but redirect to the correct posting in the new Mailman archives. There is a small PHP script that parses the URL, and does a look up in a 2 megabyte SQLite database, saving me gigabytes of disk space. Here's the gist. I wrote a script in Python (please don't laugh, it's my first!), since Mailman "pickles" their archive indexes, and I had to use Python to pull the data out...then I just figured, why switch programming languages, even if the rest is all regex stuff that Perl would excel at? Here it is: ezmlm-dump.py Put it in /usr/local/mailman/bin and run it like this: ./dump.py listname1 listname2 listnameN >/tmp/ezmlm-mappings.txt What it eventually spits out is a lot of SQL. It looks up the Message-Id in the Mailman archive indexes, which is fast and good about being unique per message, then it has to read your entire mailing list archives to find the same post in ezmlm's archives. Disk bandwidth is your enemy here, but I couldn't find a way to do this faster without losing reliability and writing a lot of nasty heuristic code. Once it knows where one archive maps to the other, it builds database INSERTs that store this information. We only had a few hundred emails that didn't have Message-Ids; some crappy email clients neglect to supply one. Mailman generates one for you in this case when importing the archives, but there won't be a match in the ezmlm pile. I felt that just dropping that email was acceptable, as it was a low percentage of the total content. Your mileage may vary. I wasn't interested in trying to compare message bodies to find the missing emails. Now, take ezmlm-mappings.txt and build an SQLite database: sqlite ezmlm-mappings.sqlite </tmp/ezmlm-mappings.txt (Protip: do things like this in one big transaction with SQLite. Tens of thousands of INSERTs took 5 minutes to run, but wrap it all in one BEGIN TRANSACTION and COMMIT TRANSACTION, and it takes 4 seconds.) sqlite3 produced a file that was 25% smaller, fwiw, saving half a megabyte, but I didn't want to bother reading the PHP manual for the cutesy object-oriented database interface, so I used version 2. This could be upgraded fairly easily, and I probably will do that work at some point. Now, put this PHP script somewhere your web server can get it, put the SQLite database in there with it, and tell Apache that the old ezmlm-cgi URL should run this script instead. The Apache configuration change looks like this, here: Alias /cgi-bin/ezmlm/ezmlm-cgi "/webspace/icculus.org/ezmlm/ezmlmremap.php" Now old broken promises like this... http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?64:mss:57:pcgggfcpkhbeledipkkh ...will automatically redirect to the updated URL... http://icculus.org/pipermail/ut3/2007-October/000057.html ...and you're good to go. Now you're done. Test it out, make a backup of your ezmlm archives, just in case, and delete them. Update anything you can with new subscription information: I intentionally didn't set up listname-subscribe addresses, since it's begging for abuse. It was best to have those break. Things out in the wild: READMEs in source tarballs, Tweets, forum posts...they'll just have to deal with the fallout. Update the canonical sources, like project webpages, and move on with your life. Otherwise, everything should be going smoothly now. In summary: this didn't suck as much as I expected. I thought there'd be a lot of pain, and some unfortunate tradeoffs I'd have to accept, but besides some time spent poking around with scripting languages, it all worked out. The next person, after reading this, will have an even better time than I did, I think. I hope. As email administration of any kind is sort of an unrewarding drain, next time I migrate mailing lists, I may wire everything up through Google Groups and call it a day. But since I'm managing my own server and lists, I feel like moving to Mailman made my life, and my users' lives about an order of magnitude better. Hopefully this will encourage someone else to make the switch, too. --ryan.