A Java on Linux Aside

So I promised a technical post but I ran into a few problems Friday afternoon that ate several hours of my evening. I was trying to deploy a Grails webapp onto a CentOS server of mine. CentOS wasn't by choice, it was dictated to me. I have much more experience with Debian and it's related derivatives like Ubuntu. Anyhow, I was trying to deploy a Grails app but it required Java 1.5/5 and all that was installed was Java 1.4.2. And to make things worse, it was the GCJ version of Java. I understand the philosophical arguments behind what the GCJ folks' reasoning. Sun's Java licensing terms wouldn't allow Linux distros to re-distribute the code. The GCJ have put together a completely free, completely re-distributable version of Java. The problem is, it just doesn't always work as well as Sun's JDK. The blame is as much the application developers as it is GCJ. But to make a long story short, my Grails webapp wouldn't work with "Java" as it is installed on CentOS.


This Java problem is/was (now that new versions of Java are open source) common to all distros. One thing that surprised me about CentOS, was that they didn't have a repository with Java pre-built and packaged. On Ubuntu, all I had to do was add a "non-free" repository and I could install the Sun JDK straightaway. And, once installed, it integrated directly with the system. On CentOS, I had to download the Java binary from Sun. Then I had to download a source RPM from another place. Then I had to manually edit the RPM spec file to match the version of the JDK I got. And then I had to setup an RPM build environment and rebuild the Java RPM with the binary bits from Sun. That ended up taking over an hour to get everything set up properly.

Once I finally got the RPM built, I thought my troubles were over. I manually installed the shiny new RPMs that I built, updated the alternatives, and checked to ensure that 'java -version' was calling the Sun JDK. So done and done I thought, but I was wrong. Imagine my surprise when I started up Tomcat 5 and it was still using the GCJ version of Java. What gives?! I made sure the default Java was the Sun JDK. So then I decided I'd never need the GCJ Java, so I decided to uninstall it completely. Imagine my surprise when Tomcat 5 decided it needed to uninstall itself with the GCJ Java.

I fiddled around with it for another hour trying to get things to work with no luck. So I ended up manually installing both the Sun JDK and Tomcat 5 in /opt and will manage it myself. This finally got things to the point where I could deploy my Grails app however, I lose all of the system integration. A user on the server can no longer just type java and expect it to work. They have to fiddle with their path and their JAVA_HOME variables to make things work. I guess the biggest disappointment to me was that I ended up settling on the solution that I had used several years ago while working on RedHat boxes. After all these years, and the advent of their 'yum' package management system, and even generating packages by hand, you still can't get the Sun JDK package to integrate properly with RedHat/CentOS.

I'm not saying it's RedHat's or CentOS's fault. I guess I've just had better experience with Debian/Ubuntu. I don't know if it's me and my rusty experience with RedHat-based distros, the (lack of) documentation, or the package management system. I'm sure it's a bit of all of them.

All in all, the Java situation is a mess on all of the major platforms. I think Windows got their growing pains out of the way early with the whole Microsoft Java fiasco. On Linux, it's been the inability to re-distribute the Sun JDK because of licensing issues. That should improve now that the newer versions of Sun's JDK are open source. Finally, the Java situation on Macs is wearing my patience pretty thin. It's about time Apple gets a version of Java 6 out that runs on my laptop. I've already shelled out the money to upgrade to Leopard with the expectation that I'd be able to get the next version of Java.

In the next post, I'll be talking about some work I've been doing with Simile Timeline.

Bacheloring it up

Elizabeth went to Rochester today through Sunday. She's having her final wedding dress fitting and a bridal shower. That leaves me and the cat to fend for ourselves. Tonight I had the boys over to watch some college basketball and to try out the cream stout Will and I just kegged. Everyone agreed that it turned out well.

Tomorrow I really need to finish up some Grails work I've been doing and to get a few screencasts shot. Since Elizabeth is gone, I'll probably end up working Saturday to catch up on some PSICAT work that I've fallen behind on. Apologies to anyone who's waiting on me. ANDRILL is keeping me busy.

Tomorrow I'll have a technical post. I haven't decide what yet, but I'll figure something out.

Automatic change tracking with cron and Mercurial

A while back I had to setup a system for sharing ANDRILL's expedition data among a geographically distributed group of scientists. For simplicities sake, I settled on serving the data via HTTP (for read only access) and WebDAV (for read/write access). It's not the most robust system, but it works, is relatively well supported across modern operating systems, and doesn't require users use some special software.

The only problem is that allowing read/write access to a large group of people means that something is going to get inadvertently changed or deleted. To combat this, I needed an automatic way to track changes that would allow me to revert or rollback any accidents.

The first step was to put the expedition data into a standard version control system. My VCS of choice is Mercurial, so I opted to use that. The initial import took a bit of time because there was about 30GB of files.

Once I had all of the data safely imported into a Mercurial repository, I needed a way to automatically detect and commit changes. I worked up a fairly simple shell script that gets run periodically by cron to check for and commit any changes:

#!/bin/sh

REPOS="/home/projects/sms/"
DATE=`date`

for r in $REPOS
do
 echo "Working on $r"

 # clean up temporary files
 find $r -type f -name "._*" \! -wholename "*.hg*" -exec rm {} \;

 # find changes
 hg addremove -R $r
 hg commit -m "Automated commit @ $DATE" -R $r
done

The script is relatively straightforward. It will walk through a list of repositories (in this case, only 1 is configured). Fore each repository, it first cleans any dangling WebDAV lock files outside of the Mercurial metadata directory. Then it runs hg addremove to detect any added or deleted files. Finally it commits the changes with an automated commit message containing the current date and time.

The script is configured to run periodically (hourly) by cron. This has the side effect that if the same file is modified two separate times within the same hour window, the intermediate changes won't be caught. For my needs, this wasn't a big deal. You could always run the script more often to have a better chance of catching more changes.

The final piece of the puzzle was to send out a daily email to the science team to notify them of the changes. For this, I developed another script that is run once a day:

#!/bin/sh

REPOS="/home/projects/sms/"
DATE=`date`

for r in $REPOS
do
 LOG=`hg log -d -1 --template '{rev}\n' -R $r`
 if [ -z "$LOG" ]; then
  echo "No changes"
 else
  echo -e "The following files have changed in the last 24 hours:\n" > /tmp/hg-tc-daily.log
  for c in $LOG
  do
   hg log --rev $c --template 'Changeset {rev} ({date|isodate}):\n' -R $r >> /tmp/hg-tc-daily.log
   hg status --rev `expr $c - 1`:$c -R $r >> /tmp/hg-tc-daily.log
   echo "" >> /tmp/hg-tc-daily.log
  done
  cat /home/projects/hg-tc/summary.legend >> /tmp/hg-tc-daily.log
  cat /tmp/hg-tc-daily.log | mail -s "Expedition Data Changes" foo@bar.com
 fi
done

If you can grok that in a single glance, you're a better shell scripter than me. It took me a couple passes to write the script and get it working the way I wanted it to. There's a bit of magic going on here, but I'll break it down.

This script walks through a set of repositories like in the previous script. The first magical incantation we run into is:
LOG=`hg log -d -1 --template '{rev}\n' -R $r`

This line asks Mercurial to tell us the revision numbers for all changes in the last day (-d -1) and stores them, one to a line, in the LOG variable. We then check to see whether the LOG variable contains anything. If it doesn't, then we're done and can exit.

If the LOG variable does contain something, then we need to send out an email to notify people of the changes. We start building up our email in /tmp/hg-tc-log. Next we walk through our list of revision numbers, and for each one we include some information using this incantation:

hg log --rev $c --template 'Changeset {rev} ({date|isodate}):\n' -R $r >> /tmp/hg-tc-daily.log
hg status --rev `expr $c - 1`:$c -R $r >> /tmp/hg-tc-daily.log

The first line simply prints out "Changeset 53 (2008-03-04 15:01 -0600):" or something similar for every changeset that occurred in the last 24 hours. The second line prints out the list of changed files and the change type (added, deleted, modified) from the previous previous revision (--rev `expr $c - 1`:$c).

Finally the script adds a helpful legend to the email and sends it out.

Overall, it's a bit hacky, but it works. The team can continue to collaborate in a central location with the relative safety of using a VCS to track changes. The best part is they don't have to do anything different; the change tracking is all automatic. I've even had the occasion to revert inadvertent changes and deletions, so it was a good investment of an afternoon.

rotateLeftAndFlipInSitu

Recently I was working with some large (200MB) vertically-oriented core images in Java. I needed a way to rotate these images CCW so they were horizontally-oriented and flip them vertically for display on an OpenGL canvas. It's a pretty easy task if you're using the Java2D API:

       // get our dimensions
       final int w = image.getWidth();
       final int h = image.getHeight();

       // create our new image and graphics
       final BufferedImage out = new BufferedImage(h, w, image.getType());
       final Graphics2D g2 = out.createGraphics();

       // setup up an affine transformation
       final AffineTransform at = AffineTransform.getRotateInstance(Math
               .toRadians(-90.0), h / 2, w / 2);
       at.translate((h - w) / 2, (w - h) / 2);
       g2.drawRenderedImage(image, at);
       g2.dispose();
       return out;

That snippet will rotate your image to the left. If you twiddle with the affine transform, you can also get it to flip the image for you. The only problem with this approach is that it requires you allocate a second image to hold the results of the transformation. So rotating a 200MB image actually requires 400MB of memory (give or take). This isn't a big deal if I only ever needed to rotate one large image at a time, but in the application, I didn't know how many large images the user was going to be working with at a time. So I decided to see if I could do the rotation in O(1) space complexity.

It turns out that this is actually a pretty difficult problem. It's been studied since the 70s under the guise of "in-situ transposition of a rectangular matrix". Once I found this out, it became a personal quest to come up with a solution. In the end, I was successful:

  // get our sample model and data buffer
  final SampleModel model = image.getSampleModel();
  final DataBuffer db = image.getRaster().getDataBuffer();

  final int w = model.getWidth();
  final int h = model.getHeight();
  final int[] t1 = new int[model.getNumBands()];
  final int[] t2 = new int[model.getNumBands()];

  // transpose
  int i, j, k, movesLeft;
  for (i = 0, movesLeft = w * h; movesLeft > 0; i++) {
   for (j = pred(i, w, h); j > i; j = pred(j, w, h)) {
    // spin
   }

   if (j < i) {
    continue;
   }
   for (k = i, j = pred(i, w, h); j != i; k = j, j = pred(j, w, h)) {
    // get our pixels
    model.getPixel(k % w, k / w, t1, db);
    model.getPixel(j % w, j / w, t2, db);

    // swap them
    model.setPixel(k % w, k / w, t2, db);
    model.setPixel(j % w, j / w, t1, db);
    --movesLeft;
   }
   --movesLeft;
  }

  // now return a new image with an updated sample model
  final WritableRaster raster = Raster.createWritableRaster(model
    .createCompatibleSampleModel(h, w), db, null);
  return new BufferedImage(image.getColorModel(), raster, image
    .isAlphaPremultiplied(), null);

with pred() defined as:

  return (i % h) * w + (i / h);

This algorithm works. It'll rotate the image in place. Thus, if you could load the image in the first place, you shouldn't run out of memory when you rotate it.

With that being said, this approach is pretty naive. There are numerous optimizations that could be implemented, from increasing the amount of temporary storage you use to optimizing the look up so you spend less time spinning.

In the end, I didn't end up using the code in production. It was a fun exercise to work through, though.

Wedding Update

Our wedding plans are progressing nicely. The actual wedding is on April 19th in the Bahamas (Euleuthera Island). It's going to be small; only a few family and friends. We're all flying down there on Thursday, the 17th. We'll have Thursday afternoon, all day Friday, and most of the day Saturday to relax and hang out. The actual wedding ceremony is going to be pretty low key. We're going to be married on the beach in late afternoon on Saturday. For the honeymoon, Elizabeth and I have rented a little house on a private beach for the following week. We return to the real world on the 27th, and a couple weeks after that, on May 16th, we're going to have a reception in St. Paul.

A blog reborn...

It's about time I reclaim this blog from dormancy. It's been a hectic couple of months since I returned from the ice. Things, however, are progressing well so it's about time I start blogging again. I'm changing the name of this blog to "Josh (Formerly) In Antarctica" as there are no Antarctic trips in the foreseeable future. I'm going to keep the blog format the same--pretty informal with a mix of personal and technical entries. I may even start tagging my posts so folks who are only interested in technical content don't have to wade through all of my posts.

Happy New Year

Happy New Year to everyone out there!

I made it home from Antarctica on Dec. 15th and I've been just relaxing and visiting family since then. With the new year, it's time to start whittling away at the pile of work that has built up. Even though the field portion of the expedition is over, my job doesn't change much--only the location where I'm working from. There are fewer day-to-day surprises that pop up, but there is still plenty to do on the data management, software development, and web fronts. And I guess that means I'd better get back to it!