ImageMagick DSL 2

This is a quick post that shows another way to work with the ImageMagick DSL (and other Java DSLs). It comes from a trick I saw in a DSL talk given by Neal Ford. Basically you can use initializer blocks to construct objects in a bit less verbose way:

new ImageMagick() {{
 option("-rotate", "90");
 option("-resize", width + "x");
}}.run(in, out);

This still requires external variables, such as width, need to be declared final. So to wrap up, here's three different ways to invoke the DSL:

Standard:

ImageMagick convert = new ImageMagick();
convert.option("-rotate", "90");
convert.option("-resize", width + "x");
convert.run(in, out);

Method Chaining:

new ImageMagick().option("-rotate", "90").option("-resize", width + "x").run(in, out);

Initializer Block:

new ImageMagick() {{
 option("-rotate", "90");
 option("-resize", width + "x");
}}.run(in, out);

ImageMagick DSL

I've been fighting with JAI/Java2D over the last day or two to manipulate (resize, crop, composite) some large images. I have working code that produces decent quality images, but I really have to crank up the heap space to avoid OutOfMemoryExceptions. If I try to process more than one or two images concurrently, OutOfMemoryExceptions are inevitable. Since this code is going to be called from a servlet, I'm expecting to handle multiple concurrent requests.

This is not a new problem and people have been tackling it in various ways. Since I was working in a server environment and have control over what applications are installed, I decided to use ImageMagick for the image manipulation. ImageMagick is great; I've used it quite often in various shell scripts.

There's basically two ways to work with ImageMagick from Java. You can use JMagick, a JNI layer over the ImageMagick interface, or you can use Runtime.exec() to call the ImageMagick command line application. I opted for the latter as it seemed simpler when I pushed the code from my Mac to my Linux server.

Since finding and invoking ImageMagick's convert command can be somewhat problematic, I decided to write a simple fluent API in Java to hide the details. The result allows you to invoke convert using method chaining:

ImageMagick convert = new ImageMagick(Arrays.asList("/opt/local/bin"));
convert.in(new File("in.jpeg"))
 .option("-fuzz", "15%")
 .option("-trim")
 .out(new File("out.jpeg")).run();
convert.option("-resize", "250x").run(new File("in.jpeg"), new File("out.jpeg"));

You create a new ImageMagick object. As a convenience, you can pass in a list of additional paths to check for the convert command in the event that it isn't on the default path. If the convert command can't be found, the constructor throws an IllegalArgumentException.

Once you have an ImageMagick object, you can execute convert by chaining various method calls, ending in a run(). run() returns true if the command succeeds, false otherwise.

In less than 200 lines of Java code, I had a much nicer way to interact with ImageMagick. A fun experiment would be to take the code and implement an even nicer DSL in Groovy. methodMissing() would allow fluent method chaining on steroids:

convert.fuzz("15%").trim().run(in, out)

As Guillaume Laforge tweeted, using metaprogramming, categories, and syntactic sugar like named parameters you could end up with a full blown DSL that looks like this:

convert fuzz:15.pct, trim: true, in: file, out:file

RESTful Query URLs

The last couple of days I've been working on writing a RESTful JSON document database. While a number of these already exist (CouchDB, FeatherDB, DovetailDB, Persevere, JSONStore, etc.), I decided to write my own because I wanted a bit more control over the URL scheme used by the REST interface, and I needed the ability to tweak the search functionality to achieve decent performance on some common but complicated queries. All in all it was an interesting diversion. The actual server clocked in at about 1000 SLOC, with much of that boilerplate because I wrote it in Java/JDBC rather than Groovy/GroovySQL.

The most interesting problem came in designing the query scheme for the REST interface. There seems to be a couple different ways to implement it with no real consensus as to which is the "right" way. As with most things, I suspect it depends on how you've implemented other pieces of the architecture and even personal preference. Below I describe three approaches I considered. The nice thing with REST is there's nothing stopping you from implementing all of these approaches in your interface.

NB: I'm no REST expert so the information below is my observations rather than any best practices. I'd love for anyone who knows better to chime into the discussion.

POST query parameters/document
In this approach, you provide a search endpoint, say something unoriginal like '/search', and queries are POSTed to that URI. The query is either a set of form encoded key-value pairs or a search document using a schema shared between the client and server.

This approach seems closer to RPC than REST to me, but may be the best approach if your search functionality requires a more complex exchange of information than simple key-value pairs allow. The obvious downside to this approach is that there is no way to bookmark a query or email/IM a query to someone else. This approach also can't take advantage of the caching built into the HTTP spec.

GET query string
Similar to above, you expose a URI endpoint, possibly something like /search, and queries are sent to that endpoint with the parameters encoded in the query string of the URL, e.g. http://www.google.com/search?q=REST+query+string

This approach improves on the bookmarkability of searches, since all of the parameters are in the URL. However, the use of the query string may interfere with caching as described in Section 13.9 of the HTTP spec. Overall, I think there is nothing inherently un-RESTful about this approach, especially if you provide more resource-oriented URIs than /search, e.g. /documents?author=Reed. In my head, I interpret the latter as "give me all of the document resources but filter on the author Reed. Removing the query string will still give you a resource (or collection of resources in this case).

Where this approach falls down is when you start trying to represent hierarchical or taxonomic queries with the query string, e.g. http://lifeforms.org?k=kingdom&p=phylum&c=class&o=order&f=family&g=genus&s=species as described on the RestWiki.

Encoding query parameters into the URI structure
In this approach the query parameters are encoded directly into the URI structure, e.g. /documents/authors/Reed, rather than using the query string. Another example of is described at Stack Overflow.

This approach solves both the bookmarkability and the caching issues of the previous approaches, but can introduce some ambiguity, especially if your resources aren't strictly hierarchical in nature. The biggest stumbling block for me was this: looking at the URI /documents/authors/Reed, it's not immediately clear what will be returned. For example, if I sent you the URI /documents you might infer that you would get a list or the contents of some documents. From the URI /documents?author=Reed, you might infer that the resource(s) returned would be documents authored by Reed. So what might you expect to get from the URI /documents/authors/Reed? Information about the author Reed or all documents authored by Reed?

How important is this? I guess it's really up to you. A machine likely infers about as much from
/documents/authors/Reed as it does from /documents?author=Reed.

Core Gallery

It seems like every couple of months I end up with a project that involves a fair amount of Javascript. Back in March it was working with Simile Timeline to visualize depth-based data. This time around, I wanted to create a lightweight way to visualize our drill core imagery. We already have full-featured visualization tools that scientists use, so I was looking to create something simple that would engage non-geologists.

The result is the Core Gallery. It shows an animated whole core image next to a split core image. Since the images are too large to display on the screen, there's a slider that lets you see different parts of the core. The page also displays some additional information about the core.

I'm really happy how it turned out. The page is 100% HTML, Javascript, and CSS. No Flash and no Java. For the Javascript, I'm using JQuery for no reason other than I wanted to see how it stacked up to other JS libraries I've used. It was perfect for this project and a treat to work with. Below I'm going to sketch out how various parts of the page are built.

Core Slider
The core slider is the most complicated part of the page. It uses the JQuery UI/Slider component. I used this screencast to help me acquaint myself with the slider. To achieve the highlighted core effect as the slider handle moves, I used two thumbnails of the core. One thumbnail is regular and one is washed out. I set the washed out thumbnail as a CSS background image on the slider element. I set the regular thumbnail as a CSS background image on the slider handle. The handle has a fixed size based on the height of the thumbnail vs. the height of the real core images so only part of the thumbnail is shown. From the slider's slide() callback, I simply update the CSS background-position property on the handle to ensure that handle's image is showing the same portion of the core as the underlying slider. I use this same technique to move the rotating whole core and split core images, taking the difference in image height between the thumbnail and the other core images into account.

Animated Whole Core Image
The slider was the most complicated but the animated whole core image was the most challenging. I wanted show the image animated in faux 3D. I initially started with a Java applet using JOGL. The applet worked on my Mac but not on Windows or Linux, so I abandoned it. I then got the idea to employ the CSS Sprites technique. So I used a tool to render the 3D whole core image 90 times each rotated by 4 degrees and montaged them together. Once I had this, it was simply a matter of setting up a Javascript Timer interval to fire every 50ms and move the image right by a fixed amount each time. This simulates animation fairly effectively. I keep track of the current rotation and vertical offset in global variables so the core keeps rotating when you move the slider.

Split Core Image
I use the same technique as on the slider handle to make the image track the slider's position.

Core Links
In the text description, it is possible to link to different parts of the core. This is a somewhat neat trick. To accomplish it, I wrap portions of the description text in span tags. Each span tag has an id attribute in the form of a ratio between 0.0 and 1.0. Using JQuery, I find these special span tags and add an onClick handler that updates the slider position based on the span's id attribute. So if the span had an id of 0.8, clicking on it would move the slider to the 80% position of the core. 0.0 takes you to the top and 1.0 takes you to the bottom.

Conclusion
Overall the Core Gallery turned out surprisingly well for being 100% browser-based. It took much less work than I originally envisioned thanks to JQuery. I'd definitely consider JQuery for future projects.

My First Griffon App

Sorry it's been so long since I posted here. Work keeps me busy.

Recently I had been given several GB of raw data from our two most recent scientific drilling expeditions in Antarctica. This data needs a fair amount of quality control processing to turn it into a usable datasets for the scientists. To do this, I needed to write a tool for the drillers to interactively plot and explore the data to determine regions of interest. Given the recent buzz about Griffon, I thought I'd give it a try.

I started by downloading and installing Griffon. Once I had everything setup, I created an app:

griffon create-app DrillingAnalytics

If you've done any Grails development, this will be a familiar idiom to you. The results of this command is a straightforward directory structure, focused around the MVC pattern. You'll recognize directories for models, views, and controllers.

My next step was to flesh out my model. When you create an app, Griffon automatically creates a model class called ${app.name}Model (DrillingAnalyticsModel for me) in the griffon-app/models directory. The main purpose of my app is to plot time series data so I defined two fields, startDate and endDate in my model:

import groovy.beans.Bindable

class DrillingAnalyticsModel {
 @Bindable String startDate = "2006-11-07 00:00"
 @Bindable String endDate = "2006-11-08 00:00"
}

You'll notice the @Bindable annotations on these fields. These fields will tie to these components in the UI, and the @Bindable annotation will automatically take care of keeping the UI in sync with the model via PropertyChangeEvents.

The model class is also where you can put other fields to maintain applications state:

 
 def plot = new CombinedDomainXYPlot(new DateAxis()) 
 def subplots = []
 def chart = new JFreeChart(null, JFreeChart.DEFAULT_TITLE_FONT, plot, false)

With the model sorted out, I moved on to developing the view. As with model, Griffon creates a ${app.name}View class for you in the griffon-app/views directory. Griffon puts the full power of SwingBuilder, SwingXBuilder, and GraphicsBuilder (with more on the way) at your fingertips for developing the UI.

I spent the majority of my time on the UI. It was a seemingly endless cycle of tweaking the code and testing with griffon run-app to get it to look the way I wanted. This is no knock on Griffon; writing Java UIs, especially by hand, just plain sucks.

After far too long trying to get the standard Java layout managers to do what I want, I did myself a favor and downloaded MigLayout. Despite not being built into SwingBuilder, MigLayout integrates nicely with SwingBuilder:

application(title:'Drilling Analytics', pack:true, locationByPlatform:true) {
   panel(layout: new MigLayout('fill')) {
    // chart panel
    widget(chartPanel, constraints:'span, grow')
    
    // our runs and time
    panel(layout: new MigLayout('fill'), border: titledBorder('Time'), constraints: 'grow 100 1') {
     scrollPane(constraints:'span 3 2, growx, h 75px') {
      runs = list(listData: model.mis.keySet().toArray())
     }
     label('Start:', constraints: 'right, gapbefore 50px')
     textField(id:"startDate", text: bind { model.startDate }, action: plotAction, constraints:'wrap, right, growx')
     label('End:', constraints: 'right, top')
     textField(id:"endDate", text: bind { model.endDate }, action: plotAction, constraints:'wrap, right, top, growx')
     label("+/-", constraints: 'right')
     textField(id:"padding", text: "30", constraints: 'growx')
     label("min")
     button(action: plotAction, constraints:'span 2, bottom, right')
    }
        
    // our plots panel
    panel(layout: new MigLayout(), border: titledBorder('Plots'), constraints: 'grow 100 1') {
     model.data.each { id, map ->
      checkBox(id: id, selected: false, action: plotAction, text: map.title, constraints:'wrap')
     }
    }
   }
}

SwingBuilder gets rid of all the boilerplate code and MigLayout makes it possible to code decent Java UIs by hand:


We've covered the Model and the View, now it's time to focus on the Controller. The controller mediates between the model and view. It contains all of the logic for handling events from the UI and manipulating the model.

One common pattern in the existing Griffon examples is the use of Swing Action objects to trigger actions from the UI. My UI was pretty simple so I could reuse a single action on all of the components to refresh the plot:

actions {
 action(id: 'plotAction', 
  name: 'Update', 
  closure: controller.plot)
}

I put this code in my DrillingAnalyticsView class, but it could just as easily be defined in its own file and imported into the view via the build() method. You'll notice that I give the action an id--plotAction--which I use to reference it from the components:

 button(action: plotAction, constraints:'span 2, bottom, right')

You can also see that the action just delegates to the controller.plot closure. This is convenient because it keeps all of the logic in one place and the controller has access to both the model and view. The actual code of the controller.plot is unremarkable. The big consideration is to properly manage your threading. Don't do long running actions in the EDT as it will freeze the UI, and don't update the UI from outside the EDT as Swing is not thread safe. Andres Almiray has a good description of how Griffon makes this easy.

Since my app is fairly niche (I doubt there's many of you visualizing drilling data), I'm not going to post the whole source code here. However, I want to point out that the source code consists of just 327 lines of code, and that's including blank lines and comments! The bulk of that code is the logic to query the database and update the JFreeChart plots. This truly demonstrates how simple and easy it is to build an app with Griffon.

If you're looking for more Griffon examples, check out the samples included in the samples directory of the Griffon distribution, and keep an eye on Griffon posts groovy.dzone.com

OSGi Command Line Applications

I'm a big fan of OSGi. One thing I always wanted to do but never got around to implementing until just recently was to be able to call services in an OSGi application from the command line. I've often wanted to be able to script PSICAT instead of having to fire it up and interact with the GUI. Turns out it's not all that difficult; you just need to sit down and do it. The only snag I ran into was that I couldn't find an implementation-agnostic way of accomplishing this, so the code I'm going to show is for the Equinox OSGi implementation. Though the same could easily be accomplished in Felix or likely other implementations with minor changes.

As with most things, there are multiple ways to skin a cat. The route I chose was to embed Equinox in a Java app and mediate command line access through this class. Fortunately, most of the work is already done for us via the EcliseStarter class (if you're on Felix, check out this). Assuming Equinox is on your classpath, simply calling EclipseStarter#startup() will fire up the Equinox runtime. More importantly, it will give you a BundleContext which you can use to interact with the OSGi framework. Once we have a BundleContext, we can do interesting things like install and start additional bundles:

 public static void main(final String[] args) throws Exception {
  // start the framework
  context = EclipseStarter.startup(new String[0], null);

  // install all bundles
  installAllPlugins();

  // start our platform bundles
  startPlugin("org.eclipse.core.runtime");

  // start plugins
  for (Bundle b : context.getBundles()) {
   startPlugin(b.getSymbolicName());
  }
...

The final piece is to do the command line interaction. For this, I created an interface that bundles can publish services under to make them available to the command line:

public interface ICommand {
 /**
  * Execute this command.
  * 
  * @param args
  *            the args.
  * @return the return value.
  */
 Object execute(String[] args) throws Exception;

 /**
  * Gets the help text that explains this command.
  * 
  * @return the help text.
  */
 String getHelp();
}

Unfortunately since there is a lot of classloader magic going on, we can't just get these ICommand classes from the service registry and invoke them directly (like we would do from inside OSGi). The OSGi classes are on a different classloader than the one we started things on. At first this may seem annoying but its actually a good thing--it means fools can't crash the OSGi implementation. So we either can specify some classloader chicanery (osgi.parentClassloader=app) or we can invoke the commands via reflection. I opted for this route because I was always taught not to mess with things you don't understand and the ClassLoader hierarchy under OSGi is definitely something I don't understand. Here's the two applicable methods:

 private static Object invokeCommand(final String name, final String[] args)
   throws Exception {
  String filter = "(&(" + Constants.OBJECTCLASS + "="
    + ICommand.class.getName() + ")(name=" + name + "))";
  ServiceReference[] services = context.getAllServiceReferences(
    ICommand.class.getName(), filter);
  if ((services != null) && (services.length != 0)) {
   Object c = context.getService(services[0]);
   if (c != null) {
    Method m = c.getClass().getMethod("execute", String[].class);
    return m.invoke(c, (Object) args);
   }
  }
  return "Command not found: " + name;

private static Map getAllCommands() {
  Map commands = new LinkedHashMap();
  try {
   ServiceReference[] services = context.getAllServiceReferences(
     ICommand.class.getName(), null);
   if (services != null) {
    for (ServiceReference r : services) {
     Object c = context.getService(r);
     if (c != null) {
      try {
       Method m = c.getClass().getMethod("getHelp");
       commands.put((String) r.getProperty("name"),
         (String) m.invoke(c));
      } catch (SecurityException e) {
       // ignore
      } catch (IllegalArgumentException e) {
       // ignore
      } catch (NoSuchMethodException e) {
       // ignore
      } catch (IllegalAccessException e) {
       // ignore
      } catch (InvocationTargetException e) {
       // ignore
      }
     }
    }
   }
  } catch (InvalidSyntaxException e) {
   // should never happen
  }
  return commands;
 }

Not my finest hour, throwing Exception, but it should get you on your way. It works like a charm in my app.

Cheers,
Josh

AT&T Update

Well, since I bitched about AT&T last time, I suppose I should post something with some technical merit. It'll be in the next post, so folks that want to read it don't have to read through this post. For those of you interested, things aren't fully resolved with AT&T but Elizabeth's mom got on the phone with AT&T and put them in their place. She took it to the AT&T National level and has direct lines to folks there that can actually get stuff done. Supposedly everything is almost sorted, I just need to bring my iPhone in and get it re-programmed to my new number. I say 'supposedly' because until the deal is actually done and it's been a month or two, I have absolutely no faith in AT&T. It was a bit comical, though, because Elizabeth's mom got things sorted in like 20 minutes. Both Elizabeth and I are dumbfounded after the numerous interactions with AT&T, both on the phone and in person, as to how she could be so persuasive.