The workaround, found by siberoptik on the Ubuntu forums, is to reboot your system with the 2.6.22-14 kernel. Everything is peachy after that. Since I’m working remotely, the way I changed kernels was to edit /boot/grub/menu.lst and nuke the lines about the 2.6.22-15 kernel. That made 2.6.22-14 the top-most kernel. By the end of the upgrade none of those kernels will matter anymore anyway. If you are in front of the box, you can just press <Esc> at the grub menu.

If you find yourself in the middle of an upgrade, frozen at “Generating locales…”, all is not lost. Get another shell on the system, look for the process tree (`ps axfw`) with locale-gen running, and kill locale-gen off.¹ For me, the upgrade continued on until the only packages left were ubuntu-minimal and locales. I had to kill off the locale-gen process a couple of more times until the upgrade process was declared “failed”.

At this point I reboot the box and something went awry as it didn’t come back again. It’s unclear from the logs what exactly happened (I was working remotely so couldn’t watch the screen), but after a hard reset I was able to get back in again. Running `dpkg --configure -a` had the locales generate instantly and everything was fine.

The total joke about all of this is, as usual, it’s not documented on Ubuntu’s site in any useful place. Yes, I found it in the forum after I hit the bug and knew what to search for, but it’s a bit late at that point. You’d think there would be a “caveats” or “known problems” or “big nasty critical bugs you should avoid” section on the help page about Hardy upgrades.

In an effort to prevent others from going down this road, I’ve put information in the following places:

• Added a “Known problems” section to the HardyUpgrades help page.
• Added a comment to the update-manager fails 7.10 -> 8.04 LTS question.
• Opened a bug against the update-manager package.

Update (2008.08.19): The bug I filed against update-manager as been made a duplicate of a bug in linux-source-2.6.22. A lot of talk has been happening there and someone claims the developers are working on a fix.

1. IMHO, it’s a good idea to try killing the highest process in the tree and work your way down to the root until you get the result you want.

deb http://packages.devsuki.com/ubuntu/ gutsy main
deb-src http://packages.devsuki.com/ubuntu/ gutsy main

You can also download them manually.

Source packages were pulled from Debian sid and two modifications were made:

1. postgresql-8.2 was added as an alternative to postgresql-8.1.
2. The init.d scripts were modified to create /var/run/zabbix-* directories on start.²

Enjoy!

1. You may want to download the package signing key and add it with apt-key.
2. Ubuntu uses tmpfs for /var/run, so these directories will disappear on reboot. I followed the suggestion in this zabbix forum post.

My favorite place to buy/download music is bleep.com. There are two reasons:

1. very high quality, DRM-free tracks
2. an awesome music preview system

When I say “very high quality”, I mean it. All albums encoded after January 2006 come as 320K MP3s (the highest quality level) and are encoded with LAME (the highest quality encoder). And if that ain’t enough quality for you, they often have new albums available (for a little more $€£) in lossless FLAC! Furthermore, I’ve probably bought about 10 albums from them and none have ever had any ripping audio artifacts (skips, pops, clicks, static, drop outs, etc.) — they’re clean as can be.

As if all that wasn’t enough, their website lets you listen to every part of every song before you buy! This is just awesome. Typically, sites only give you the first 30s, which isn’t very useful for long and intricate tracks. I’ve spent many hours listening to tracks on bleep.com looking for a new album to buy. It’s an awesome feature and has gotten me to buy more music from them.

The downside to bleep.com is also one of their strengths. Originally, they were started as store for Warp Records, a highly influential electronic music label. Since that inception the store has expanded to carry music from a huge list of labels, however the predominant flavor is still electronic music (IDM, breaks, dub, glitch, ambient, minimal, experimental, etc.). Thus, the downside is you aren’t going to find that metal, hip-hop, or jazz album you’ve been wanting, but the upside is if you are into interesting electronic music, you’ll poop your pants when you discover hard to find albums are only a few clicks away from playing on your stereo.

1. I did buy one DRM-free track via iTunes when iTunes-plus launched (as a vote of support), but I have yet to find any other iTunes-plus tracks I’m interested in. Seriously, every time I look for an album it’s only available wrapped in iTunes’ DRM. With the advent of amazonmp3.com, I’m probably going to stop checking the iTunes store at all. More on that later.

Matt and I have been involved with the free software movement for the last decade. One of the consequences of using software developed by people in their spare time is that bugs and emails can be ignored for long periods of time (e.g., forever). After almost 10 years, and a bunch of our own software projects, I like to think we’ve developed reasonable expectations for how things work (and don’t work).

When I report a bug to Ubuntu I often include a debdiff (package patch). In other words, I’m not just whining about bugs, I’m fixxing them! This is because one of my day jobs is sysadmin for Mosuki. Every Ubuntu upgrade released has had problems of varying severity and it’s my job to patch the relevant packages and get them into our development apt repostory. I was annoyed by this process for a while, but I’ve come to accept it as the trade-off for getting a new stable set of software every 6 months (which we’ve come to rely on). This means, every Ubuntu release I find and fix a bunch of Ubuntu bugs.

There are many reasons I use free software, but in the end, I want to give back to the community. For me, this means creating debdiffs of my work and submitting it to Launchpad. There’s little practical benefit I receive from doing this.¹ But I believe in free software; I want to make my work available to others and I want to make Ubuntu a better distribution.

Simply, it comes down to this: If my work is ignored, then what is the purpose of doing the work? If the Launchpad bug tracker is not an adequate mechanism for getting my fixes into Ubuntu, then why should I use it? I’m jumping through their hoops for their own good, not mine.

To reap the benefit of my work without waiting for Ubuntu, add the Devsuki apt repository to your /etc/apt/sources.list and decide you trust my work (or inspect the changes I’ve made):

deb http://packages.devsuki.com/ubuntu/ gutsy main
deb-src http://packages.devsuki.com/ubuntu/ gutsy main

1. Of course, I’d rather the packages be officially fixed rather than in my own repository, but that’s about it.

You may have noticed that our site broke this morning. Apparently the power supply on our database server decided to commit suicide while everyone was asleep.

OMG. How can any social network site hoping for some semblance of success (or real funding) be running their DB on a single machine with no failover. I’m totally floored by this. Even when Mosuki was doing private alpha testing 3½ years ago we had a hot-failover cluster. I wonder if they do backups.

Even though I’ve read scores of articles, white papers and books on the psychology of terrorists in recent years, and even though I have (I think) a strong intellectual grasp of the roots of suicide terrorism, something about playing the game gave me an “aha” moment that I’d never had before: an ability to feel, in whatever tiny fashion, the strategic logic and emotional calculus behind the act.

device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table

Even worse, you may be unable to mount any partitions aside from /! That’s what happened to me.

There are lots of people having these device-mapper errors. Some explanation is given in the “Volume activation” section of the EVMS FAQ.

If you run into the problem where you reboot and no longer have a /var partition, you should try preventing evms from starting at boot and rebooting. For example, you can do this:

mv /etc/rcS.d/S27evms /root
reboot

If your system doesn’t want to reboot, check for processes hung up on shutting down (I had to kill off something related to Alsa). If everything comes up fine, clean and purge:

mv /root/S27evms /etc/rcS.d
aptitude purge evms

Everything should be happy at this point. You can also clean up the junk that ended up in /var when your /var partition couldn’t be mounted. Just mount / to another place, clean and unmount. Thanks to Ross Cohen for telling me about multimount¹.

1. ”Since Linux 2.4 a single filesystem can be visible at multiple mount points” –mount(2) man page.

echo letter | sudo tee /etc/papersize >/dev/null

Say what? Why on Earth use tee if you are going to dump it’s output to /dev/null? I assume whoever wrote it ran into a file permission problem due to the redirection.¹ How about the following:

sudo sh -c 'echo letter > papersize'

I’m not going to complain that their way spawns an extra process; I dislike complaints which ignore the capacity of modern computers and that this command is run once by hand. Probably, I just think their way is ugly.

1. The sudo manpage also suggests using sh -c for making redirection work.