I had the following happen for every new java process on one of my servers the other day:

server:~$ java
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f3e0c5aad9b, pid=17280, tid=139904457242368
#
# JRE version: 6.0_24-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x7ed9b]  memset+0xa5b
#
# An error report file with more information is saved as:
# /home/user/hs_err_pid17280.log
Segmentation fault

Turns out this is Java's way of telling you that the /tmp directory is full. It's trying to mmap some performance/hotspot-related file in /tmp which succeeds, but when it's trying to access this area, it will get the SIGBUS signal.

More info here

I'm using Hadoop Streaming a lot. It's exit codes has been something of a mystery, so today I decided to find out by looking at the source code.

The exit codes are listed in StreamJob.java, and are as follows:

0. Success
1. Job not successful, i.e. something went wrong with M/R code.
2. Bad input path
3. Invalid jobconf
4. Output path already exists
5. Error launching job. Could be any error, for example some HDFS communication error.

I'm working on a rather large reporting and analytics application that runs on top of Hadoop at work. It has tests. A whole bunch of them, actually. That's good.

So far, we've been running the tests manually when making new releases. But doing it more often is always better, since it gives you an indication on when things went wrong, and also forces you to keep your tests in a state where they pass. Some people call it Continous Integration.

Now, you can do all the work getting your builds to build and run tests yourself, via cron and scripts and other types of messiness. Or you can try an existing solution. Today I decided to try Hudson.

That turned out to be embarrasingly simple to get started with. Basically, it's a matter of:

1. Downloading hudson.war from their site.
2. Start it by running java -jar hudson.war
3. Go to http://localhost:8080 with a web browser of your choice. That would be Opera in my case. You have to eat your own dog-food.
4. Go to the Hudson management screen and enable the git plugin
5. Setup a new project. Tell it where the code is and on which branch.
6. Configure what commands to run to build and test. Make the test command output an xunit xml file.
7. Tell Hudson where that xml file is.

Result: Hudson will periodically poll git and run my build and test commands, then show a changelog and what tests failed. All this after 30 minutes of setup time. I'm impressed.

Yesterday some of the servers I care for at work were moved to a different network. After the move, all puppetd runs started to take a very long time. Where it would usually take 10-15 seconds, it now timed out with errors like:

Jan 12 19:39:16 host1 puppetd[15760]: Calling puppetmaster.getconfig
Jan 12 19:41:16 host1 puppetd[15760]: Configuration retrieval timed out

(Note the two minutes between the informational message about calling puppetmaster.getconfig, and the timeout)

Highly confusing, especially since puppetd was slow not only on hosts which had moved to the new network, but also on hosts which had not moved.

The reason turned out to be slow reverse DNS for the new network range. Puppetmaster it seems is doing lot's and lot's of DNS lookups for clients, and that seems to be a synchronous operation. I think what caused all hosts to slow down was that puppetmaster got busy looking up one of the hosts on the new network, and that would cause the request from a host that had not moved to be put on hold.

Fixing the DNS issue solved the problem.

This is on puppet 0.24.5. Later versions might have a better behaviour.

I was rebalancing one of the Hadoop clusters I run at work. It was not running very fast, so I modified the appropriate setting:

<property>
  <!-- 100Mbit/s -->
  <name>dfs.balance.bandwidthPerSec</name>
  <value>104857600</value>
</property>

I restarted the namenode and thought that would make the trick. But no, you also need to restart all your datanodes for the setting to take effect. Now I can see some action on my network graphs :-).

apt-get install pwgen

In the category Stuff I really should have learned several years ago, I now know that the keyboard combinations for copying and pasting in gnome-terminal is Shift-Control-C and Shift-Control-V

Now, if I could find out how to do select text without using the mouse...

They never work as intended the first four times you run them.

My girlfriend runs a blog on a cheap hosting firm that doesn't provide any way of doing proper SQL dumps of the MySQL database used by the blogging software.

There are plugins for Wordpress that can do full backups, but I prefer doing raw SQL dumps + a filesystem backup. That way, you know what you get, you don't have to trust the backup plugin author to do it right.

The hosting firm does provide access to a phpMyAdmin installation which you can use to download SQL dumps. The trick is of course to do this automatically, as good backups need to be unattended.

I wrote a python program that can do this, using what turned out to be an excellent library for programmatic web browsing: mechanize.

The backup script is available in my misctools project on GitHub.

This website runs on a virtual machine I buy from Slicehost. I've also choosen to use their DNS servers for my domain - the service is stable and included in the price.

The Slicehost DNS can be modified using the Slicehost API. I wrote two small scripts for easy modification of Slicehost DNS entries from the commandline or from scripts.

• update_entry, for adding or updating existing entries.
• dhclient_update_hook, which very easily can be used to update an entry from a dhclient script, to keep records that point to dynamic adressess updated automatically.

Both are available from by cloning my misctools project at GitHub.

When trying to get my local development copy of this website running after upgrading my Ubuntu, I got the following confusing error message from the psycopg2 python module:

psycopg2.OperationalError: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket  "/var/run/postgresql/.s.PGSQL.5432"?

My django settings file was correct:

DATABASE_ENGINE = 'postgresql_psycopg2'  # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = 'dbname'                 # Or path to database file if using sqlite3.
DATABASE_USER = 'dbuser'                 # Not used with sqlite3.
DATABASE_PASSWORD = 'dbpassword'         # Not used with sqlite3.
DATABASE_HOST = ''                       # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = ''                       # Set to empty string for default. Not used with sqlite3.

Confusing, since my Postgres server was running and I could connect using psql:

psql -U dbuser -W dbname

This turned out to be one of these problems when Google is of no help - others had the same problem, but I could only find posts where people asked the question, no posts where the actual solution was found.

The cause of the problem was that my PostgreSQL installation was configured to listen on port 5433 instead of the default 5432, and as seen in the error message, the port number is part of the path to the unix socket. The different port was probably setup when I upgraded my Ubuntu, since that installed PostgreSQL 8.4 without completely removing PostgreSQL 8.3. The latter is configured to listen on the default port.

The solution is to either configure the running PostgreSQL to listen on port 5432 by modifying /etc/postgresql/8.4/main/postgresql.conf, or by modifying the Django configuration by setting the port:

DATABASE_PORT = '5433'

For a while, I used Duplicity to make backups to an Amazon S3 bucket. That kind of worked, but I had to do a lot of scripting myself to get it working automatically, so after finding out about Jungledisk, I switched to that. Jungledisk has a nice little desktop applet that keeps track of doing my backups while my computer is on, etc. That's convenient.

Anyway, the Duplicity/S3 experiments left me with an Amazon S3 bucket with about 9000 objects. Getting rid of that proved to be something of a challenge - you have to delete all objects inside the bucket before you can delete the bucket itself, and there's no API call for doing that. I also tried the web application for managing buckets, S3FM but that didn't cope too well with that many objects - my web browser just hung.

I have to admit I could have put more effort into googling before solving it by writing my own script - but writing my own script was more fun :-).

My script managed to delete all 9000 objects without trouble, although it did take quite a while to complete - I let it run overnight.

If you need to do the same thing, it's available here:

http://github.com/forsberg/misctools/tree/0d6cbd189b80e52501d94742fa04bba36b7f34a6/S3

StackOverflow has several other solutions: http://stackoverflow.com/questions/27267/delete-amazon-s3-buckets

I got a new toy today - a GPS logger. Got it as part of a magazine subscription deal - two issues + GPS logger for about $12. That magazine was not my kind of magazine, but getting a GPS logger delivered home for $12 was worth having to throw away two magazine issues :-). For other people trying to get this to work under Linux, the magazine in question was Aktiv Träning.

After some fiddling, I came to the conclusion that the device is based on the SkyTraq Venus chipset:

skytraq: Venus device found: Kernel version = 1.3.3, ODM version = 1.4.5, revision (Y/M/D) = 07/12/11

After plugging the device into an USB port, it appears as a serial port, in my case /dev/ttyUSB0. The data can be retrieved using GPSBabel, but it has to be the development version - the stable version does not feature the skytraq driver required. The development version is available via CVS from sourceforge, see http://sourceforge.net/projects/gpsbabel/develop for details on how to access CVS.

Once you have the development version of gpsbabel, getting the data is as easy as:

./gpsbabel -D 9 -i skytraq,initbaud=38400,baud=38400,erase -f /dev/ttyUSB0 -o gpx -F out.gpx

This will write the GPS coordinate log to out.gpx in the GPS Exchange format - a lightweight XML format.

The above command will also erase the log from the device, so the next download contains just the latest log.

Since I like playing with Google Earth, I also learned that the GPX can be converted into KML - the format Google Earth can read, by running:

./gpsbabel -i gpx -f out.gpx -o kml -F out.kml

Just open out.kml with Google Earth, and you'll see where the GPS logger has been.

On popular request, here's a few images of the device.

The label inside the battery compartment has the following text:

GT-750F/L-Lite
GPS Receiver(Data Logger)
RoHS CE FCC ID:VHP-750F
Made in Taiwan CANMORE

Playing with my GPS logger, I wanted to know the distance travelled between all the points in a GPS log. For example, if I went for a bicycle trip and brought the logger, it would be nice to know how many kilometers I have travelled.

Trying to find a tool that did this without being cumbersome to use proved more difficult than I thought, so I did a quick hack with the help of the python bindings for gpsd. It's a very simple script:

$ ./distance.py promenad2.gpx
427.077378454

Given a GPX file, it will print out the number of meters travelled.

Fetch it in the fgpstools repository on github

If you run a modern computer, chances are high that you have a multi-core processor. That forces us into paralell programming to get more performance, which is much harder than doing serial programming.

The operating system and existing shell commands can do much for us, though. Let's say you have a bunch of files where each should be processed by a command. Of course, with multiple cores, this could be sped up by running multiple instances of the command simultaneously. And there's a very easy way to do that - xargs. An example for converting raw files into jpeg:

find . -type f -name '\*.CR2' \|xargs -n 1 -P 3 ~/bin/raw2jpeg

This will find all files named *.CR2 and feed that list to xargs, which will run up to three (-P 3) processes simultaneously, giving each command one filename (-n 1) on its commandline.