RESTful Query URLs

The last couple of days I've been working on writing a RESTful JSON document database. While a number of these already exist (CouchDB, FeatherDB, DovetailDB, Persevere, JSONStore, etc.), I decided to write my own because I wanted a bit more control over the URL scheme used by the REST interface, and I needed the ability to tweak the search functionality to achieve decent performance on some common but complicated queries. All in all it was an interesting diversion. The actual server clocked in at about 1000 SLOC, with much of that boilerplate because I wrote it in Java/JDBC rather than Groovy/GroovySQL.

The most interesting problem came in designing the query scheme for the REST interface. There seems to be a couple different ways to implement it with no real consensus as to which is the "right" way. As with most things, I suspect it depends on how you've implemented other pieces of the architecture and even personal preference. Below I describe three approaches I considered. The nice thing with REST is there's nothing stopping you from implementing all of these approaches in your interface.

NB: I'm no REST expert so the information below is my observations rather than any best practices. I'd love for anyone who knows better to chime into the discussion.

POST query parameters/document
In this approach, you provide a search endpoint, say something unoriginal like '/search', and queries are POSTed to that URI. The query is either a set of form encoded key-value pairs or a search document using a schema shared between the client and server.

This approach seems closer to RPC than REST to me, but may be the best approach if your search functionality requires a more complex exchange of information than simple key-value pairs allow. The obvious downside to this approach is that there is no way to bookmark a query or email/IM a query to someone else. This approach also can't take advantage of the caching built into the HTTP spec.

GET query string
Similar to above, you expose a URI endpoint, possibly something like /search, and queries are sent to that endpoint with the parameters encoded in the query string of the URL, e.g. http://www.google.com/search?q=REST+query+string

This approach improves on the bookmarkability of searches, since all of the parameters are in the URL. However, the use of the query string may interfere with caching as described in Section 13.9 of the HTTP spec. Overall, I think there is nothing inherently un-RESTful about this approach, especially if you provide more resource-oriented URIs than /search, e.g. /documents?author=Reed. In my head, I interpret the latter as "give me all of the document resources but filter on the author Reed. Removing the query string will still give you a resource (or collection of resources in this case).

Where this approach falls down is when you start trying to represent hierarchical or taxonomic queries with the query string, e.g. http://lifeforms.org?k=kingdom&p=phylum&c=class&o=order&f=family&g=genus&s=species as described on the RestWiki.

Encoding query parameters into the URI structure
In this approach the query parameters are encoded directly into the URI structure, e.g. /documents/authors/Reed, rather than using the query string. Another example of is described at Stack Overflow.

This approach solves both the bookmarkability and the caching issues of the previous approaches, but can introduce some ambiguity, especially if your resources aren't strictly hierarchical in nature. The biggest stumbling block for me was this: looking at the URI /documents/authors/Reed, it's not immediately clear what will be returned. For example, if I sent you the URI /documents you might infer that you would get a list or the contents of some documents. From the URI /documents?author=Reed, you might infer that the resource(s) returned would be documents authored by Reed. So what might you expect to get from the URI /documents/authors/Reed? Information about the author Reed or all documents authored by Reed?

How important is this? I guess it's really up to you. A machine likely infers about as much from
/documents/authors/Reed as it does from /documents?author=Reed.

Core Gallery

It seems like every couple of months I end up with a project that involves a fair amount of Javascript. Back in March it was working with Simile Timeline to visualize depth-based data. This time around, I wanted to create a lightweight way to visualize our drill core imagery. We already have full-featured visualization tools that scientists use, so I was looking to create something simple that would engage non-geologists.

The result is the Core Gallery. It shows an animated whole core image next to a split core image. Since the images are too large to display on the screen, there's a slider that lets you see different parts of the core. The page also displays some additional information about the core.

I'm really happy how it turned out. The page is 100% HTML, Javascript, and CSS. No Flash and no Java. For the Javascript, I'm using JQuery for no reason other than I wanted to see how it stacked up to other JS libraries I've used. It was perfect for this project and a treat to work with. Below I'm going to sketch out how various parts of the page are built.

Core Slider
The core slider is the most complicated part of the page. It uses the JQuery UI/Slider component. I used this screencast to help me acquaint myself with the slider. To achieve the highlighted core effect as the slider handle moves, I used two thumbnails of the core. One thumbnail is regular and one is washed out. I set the washed out thumbnail as a CSS background image on the slider element. I set the regular thumbnail as a CSS background image on the slider handle. The handle has a fixed size based on the height of the thumbnail vs. the height of the real core images so only part of the thumbnail is shown. From the slider's slide() callback, I simply update the CSS background-position property on the handle to ensure that handle's image is showing the same portion of the core as the underlying slider. I use this same technique to move the rotating whole core and split core images, taking the difference in image height between the thumbnail and the other core images into account.

Animated Whole Core Image
The slider was the most complicated but the animated whole core image was the most challenging. I wanted show the image animated in faux 3D. I initially started with a Java applet using JOGL. The applet worked on my Mac but not on Windows or Linux, so I abandoned it. I then got the idea to employ the CSS Sprites technique. So I used a tool to render the 3D whole core image 90 times each rotated by 4 degrees and montaged them together. Once I had this, it was simply a matter of setting up a Javascript Timer interval to fire every 50ms and move the image right by a fixed amount each time. This simulates animation fairly effectively. I keep track of the current rotation and vertical offset in global variables so the core keeps rotating when you move the slider.

Split Core Image
I use the same technique as on the slider handle to make the image track the slider's position.

Core Links
In the text description, it is possible to link to different parts of the core. This is a somewhat neat trick. To accomplish it, I wrap portions of the description text in span tags. Each span tag has an id attribute in the form of a ratio between 0.0 and 1.0. Using JQuery, I find these special span tags and add an onClick handler that updates the slider position based on the span's id attribute. So if the span had an id of 0.8, clicking on it would move the slider to the 80% position of the core. 0.0 takes you to the top and 1.0 takes you to the bottom.

Conclusion
Overall the Core Gallery turned out surprisingly well for being 100% browser-based. It took much less work than I originally envisioned thanks to JQuery. I'd definitely consider JQuery for future projects.

My First Griffon App

Sorry it's been so long since I posted here. Work keeps me busy.

Recently I had been given several GB of raw data from our two most recent scientific drilling expeditions in Antarctica. This data needs a fair amount of quality control processing to turn it into a usable datasets for the scientists. To do this, I needed to write a tool for the drillers to interactively plot and explore the data to determine regions of interest. Given the recent buzz about Griffon, I thought I'd give it a try.

I started by downloading and installing Griffon. Once I had everything setup, I created an app:

griffon create-app DrillingAnalytics

If you've done any Grails development, this will be a familiar idiom to you. The results of this command is a straightforward directory structure, focused around the MVC pattern. You'll recognize directories for models, views, and controllers.

My next step was to flesh out my model. When you create an app, Griffon automatically creates a model class called ${app.name}Model (DrillingAnalyticsModel for me) in the griffon-app/models directory. The main purpose of my app is to plot time series data so I defined two fields, startDate and endDate in my model:

import groovy.beans.Bindable

class DrillingAnalyticsModel {
 @Bindable String startDate = "2006-11-07 00:00"
 @Bindable String endDate = "2006-11-08 00:00"
}

You'll notice the @Bindable annotations on these fields. These fields will tie to these components in the UI, and the @Bindable annotation will automatically take care of keeping the UI in sync with the model via PropertyChangeEvents.

The model class is also where you can put other fields to maintain applications state:

 
 def plot = new CombinedDomainXYPlot(new DateAxis()) 
 def subplots = []
 def chart = new JFreeChart(null, JFreeChart.DEFAULT_TITLE_FONT, plot, false)

With the model sorted out, I moved on to developing the view. As with model, Griffon creates a ${app.name}View class for you in the griffon-app/views directory. Griffon puts the full power of SwingBuilder, SwingXBuilder, and GraphicsBuilder (with more on the way) at your fingertips for developing the UI.

I spent the majority of my time on the UI. It was a seemingly endless cycle of tweaking the code and testing with griffon run-app to get it to look the way I wanted. This is no knock on Griffon; writing Java UIs, especially by hand, just plain sucks.

After far too long trying to get the standard Java layout managers to do what I want, I did myself a favor and downloaded MigLayout. Despite not being built into SwingBuilder, MigLayout integrates nicely with SwingBuilder:

application(title:'Drilling Analytics', pack:true, locationByPlatform:true) {
   panel(layout: new MigLayout('fill')) {
    // chart panel
    widget(chartPanel, constraints:'span, grow')
    
    // our runs and time
    panel(layout: new MigLayout('fill'), border: titledBorder('Time'), constraints: 'grow 100 1') {
     scrollPane(constraints:'span 3 2, growx, h 75px') {
      runs = list(listData: model.mis.keySet().toArray())
     }
     label('Start:', constraints: 'right, gapbefore 50px')
     textField(id:"startDate", text: bind { model.startDate }, action: plotAction, constraints:'wrap, right, growx')
     label('End:', constraints: 'right, top')
     textField(id:"endDate", text: bind { model.endDate }, action: plotAction, constraints:'wrap, right, top, growx')
     label("+/-", constraints: 'right')
     textField(id:"padding", text: "30", constraints: 'growx')
     label("min")
     button(action: plotAction, constraints:'span 2, bottom, right')
    }
        
    // our plots panel
    panel(layout: new MigLayout(), border: titledBorder('Plots'), constraints: 'grow 100 1') {
     model.data.each { id, map ->
      checkBox(id: id, selected: false, action: plotAction, text: map.title, constraints:'wrap')
     }
    }
   }
}

SwingBuilder gets rid of all the boilerplate code and MigLayout makes it possible to code decent Java UIs by hand:


We've covered the Model and the View, now it's time to focus on the Controller. The controller mediates between the model and view. It contains all of the logic for handling events from the UI and manipulating the model.

One common pattern in the existing Griffon examples is the use of Swing Action objects to trigger actions from the UI. My UI was pretty simple so I could reuse a single action on all of the components to refresh the plot:

actions {
 action(id: 'plotAction', 
  name: 'Update', 
  closure: controller.plot)
}

I put this code in my DrillingAnalyticsView class, but it could just as easily be defined in its own file and imported into the view via the build() method. You'll notice that I give the action an id--plotAction--which I use to reference it from the components:

 button(action: plotAction, constraints:'span 2, bottom, right')

You can also see that the action just delegates to the controller.plot closure. This is convenient because it keeps all of the logic in one place and the controller has access to both the model and view. The actual code of the controller.plot is unremarkable. The big consideration is to properly manage your threading. Don't do long running actions in the EDT as it will freeze the UI, and don't update the UI from outside the EDT as Swing is not thread safe. Andres Almiray has a good description of how Griffon makes this easy.

Since my app is fairly niche (I doubt there's many of you visualizing drilling data), I'm not going to post the whole source code here. However, I want to point out that the source code consists of just 327 lines of code, and that's including blank lines and comments! The bulk of that code is the logic to query the database and update the JFreeChart plots. This truly demonstrates how simple and easy it is to build an app with Griffon.

If you're looking for more Griffon examples, check out the samples included in the samples directory of the Griffon distribution, and keep an eye on Griffon posts groovy.dzone.com

OSGi Command Line Applications

I'm a big fan of OSGi. One thing I always wanted to do but never got around to implementing until just recently was to be able to call services in an OSGi application from the command line. I've often wanted to be able to script PSICAT instead of having to fire it up and interact with the GUI. Turns out it's not all that difficult; you just need to sit down and do it. The only snag I ran into was that I couldn't find an implementation-agnostic way of accomplishing this, so the code I'm going to show is for the Equinox OSGi implementation. Though the same could easily be accomplished in Felix or likely other implementations with minor changes.

As with most things, there are multiple ways to skin a cat. The route I chose was to embed Equinox in a Java app and mediate command line access through this class. Fortunately, most of the work is already done for us via the EcliseStarter class (if you're on Felix, check out this). Assuming Equinox is on your classpath, simply calling EclipseStarter#startup() will fire up the Equinox runtime. More importantly, it will give you a BundleContext which you can use to interact with the OSGi framework. Once we have a BundleContext, we can do interesting things like install and start additional bundles:

 public static void main(final String[] args) throws Exception {
  // start the framework
  context = EclipseStarter.startup(new String[0], null);

  // install all bundles
  installAllPlugins();

  // start our platform bundles
  startPlugin("org.eclipse.core.runtime");

  // start plugins
  for (Bundle b : context.getBundles()) {
   startPlugin(b.getSymbolicName());
  }
...

The final piece is to do the command line interaction. For this, I created an interface that bundles can publish services under to make them available to the command line:

public interface ICommand {
 /**
  * Execute this command.
  * 
  * @param args
  *            the args.
  * @return the return value.
  */
 Object execute(String[] args) throws Exception;

 /**
  * Gets the help text that explains this command.
  * 
  * @return the help text.
  */
 String getHelp();
}

Unfortunately since there is a lot of classloader magic going on, we can't just get these ICommand classes from the service registry and invoke them directly (like we would do from inside OSGi). The OSGi classes are on a different classloader than the one we started things on. At first this may seem annoying but its actually a good thing--it means fools can't crash the OSGi implementation. So we either can specify some classloader chicanery (osgi.parentClassloader=app) or we can invoke the commands via reflection. I opted for this route because I was always taught not to mess with things you don't understand and the ClassLoader hierarchy under OSGi is definitely something I don't understand. Here's the two applicable methods:

 private static Object invokeCommand(final String name, final String[] args)
   throws Exception {
  String filter = "(&(" + Constants.OBJECTCLASS + "="
    + ICommand.class.getName() + ")(name=" + name + "))";
  ServiceReference[] services = context.getAllServiceReferences(
    ICommand.class.getName(), filter);
  if ((services != null) && (services.length != 0)) {
   Object c = context.getService(services[0]);
   if (c != null) {
    Method m = c.getClass().getMethod("execute", String[].class);
    return m.invoke(c, (Object) args);
   }
  }
  return "Command not found: " + name;

private static Map getAllCommands() {
  Map commands = new LinkedHashMap();
  try {
   ServiceReference[] services = context.getAllServiceReferences(
     ICommand.class.getName(), null);
   if (services != null) {
    for (ServiceReference r : services) {
     Object c = context.getService(r);
     if (c != null) {
      try {
       Method m = c.getClass().getMethod("getHelp");
       commands.put((String) r.getProperty("name"),
         (String) m.invoke(c));
      } catch (SecurityException e) {
       // ignore
      } catch (IllegalArgumentException e) {
       // ignore
      } catch (NoSuchMethodException e) {
       // ignore
      } catch (IllegalAccessException e) {
       // ignore
      } catch (InvocationTargetException e) {
       // ignore
      }
     }
    }
   }
  } catch (InvalidSyntaxException e) {
   // should never happen
  }
  return commands;
 }

Not my finest hour, throwing Exception, but it should get you on your way. It works like a charm in my app.

Cheers,
Josh

AT&T Update

Well, since I bitched about AT&T last time, I suppose I should post something with some technical merit. It'll be in the next post, so folks that want to read it don't have to read through this post. For those of you interested, things aren't fully resolved with AT&T but Elizabeth's mom got on the phone with AT&T and put them in their place. She took it to the AT&T National level and has direct lines to folks there that can actually get stuff done. Supposedly everything is almost sorted, I just need to bring my iPhone in and get it re-programmed to my new number. I say 'supposedly' because until the deal is actually done and it's been a month or two, I have absolutely no faith in AT&T. It was a bit comical, though, because Elizabeth's mom got things sorted in like 20 minutes. Both Elizabeth and I are dumbfounded after the numerous interactions with AT&T, both on the phone and in person, as to how she could be so persuasive.

AT&T == Lying, Deceitful, and Fraudulent

So it's been a long time since I blogged and I really hate to be so negative but I had an absolute nightmare of a day dealing with AT&T today. My birthday is coming up and Elizabeth thought it would be nice to get me an iPhone because I had been asking about them. So begins the saga. We previously had cell phones on Elizabeth's parent's AT&T Family Talk plan, which costs about $20/month for 2 lines. Elizabeth called up AT&T to figure out what we had to do to get me an iPhone. They informed us that all we needed to do was sign up for a new account and transfer our existing numbers to this account. However, despite mentioning numerous times that we were doing this to purchase an iPhone and asking explicitly about the costs, we were told "pay the transfer fee of $18/line and sign up for a Family Talk plan @ $69.99 and then go into the Apple Store and they'll set up the iPhone data plan" and you'll be good to go. (And I make it sound easy, but this really entailed 2 hours on the phone and talking to several different people at AT&T). Looking online we had a ballpark figure of around $150/month for the voice and data. This is far more than the $20/month we had been paying but we were willing to shell out the money for our own account and for the iPhone plan.

Fast forward to an hour later when we were in an Apple store trying to activate one of the few remaining iPhones. Activation failed! Apparently the new account got created as a business account instead of a domestic account and Apple couldn't activate the phone. WTF? Well we called up AT&T while at the mall and after another 30 minutes on the phone, got the new account switched to a domestic rather than business account. We go back for a second time to activate the phone and they said we're not eligible. So what everyone from AT&T neglected to mention was that despite signing up for a new account and a significant additional cost, by transferring our numbers we were still bound under the original contract. Nevermind the fact that we 1) weren't switching companies and 2) we were actually bringing MORE money in for AT&T since we were going from paying $20/month for the next year to paying $150/month for the next 2 years.

That's all fine and dandy but what I don't understand is how they can sign us up for a new contract under different terms but hold us to the original contract. They were all too happy to charge us $36 to transfer our numbers and commit us to paying $150/month for the next 2 years but when we want to get the iPhone at the discount all of a sudden the story is that we're still under the other contract and are not eligible for the phone at the reduced price. Now, I can understand the transfer logic and I can understand the new account logic. What I can't fathom is how they think they can enforce two contracts, with conflicting terms, at the same time? Either the new account comes under the old contract terms, and I pay $20/month through September 2009 and no iPhone upgrade (not really a new account then) or the new account is treated like a new account at the new rate for the new time period and I'm eligible for the iPhone upgrade. One or the other, but you can't have both!

But it doesn't end there. I went ahead and signed up a new single account under my name to purchase an iPhone. This meant having to get a new number. After the Apple store, we walked down the hall in the mall and went into the AT&T corporate store thinking it might be a refreshing change from spending hours on the phone. We explained the situation to the customer service rep there and he was all to happy to try and rectify the situation. He said "sure, we can just transfer those numbers back to the original account and close the new account". We were like, that's fine even though my old number/phone would basically go unused now that I had a new number. It would save Elizabeth from having to change her number. The AT&T rep couldn't do the transfer from his system because of that stupid business vs. domestic error so he called the corporate office to get things fixed. He almost got it but then needed permission from Elizabeth's mom to add us back onto the Family Talk plan. This makes sense, so I don't fault them for that. Unfortunately we couldn't get Elizabeth's mom on the phone so we couldn't continue.

It was at that point that the rep informed us that it would cost an additional $18/line to transfer the numbers back! So we had to pay to transfer the lines to a new account, despite the fact that no one informed us that we wouldn't be able to purchase the iPhone at the reduced rate and now they wanted to charge us an additional $18/line to transfer back. All that after spending the whole afternoon, from noon to 5PM either on the phone or in stores dealing with AT&T! The rep, Andrew "Drew" at the Southdale AT&T store then proceeded to get in our face about the charges and be quite rude. "Well it's not like we just went in and changed it without permission." No, but you also were deceitful when you said that all we had to do was sign up for a new account and transfer our numbers and then we'd be good to go.

But the best is yet to come. So we leave the store and Elizabeth immediately gets on the phone again with AT&T. Once she actually gets to a live person, she explains the situation for the umpteenth time and then gets flak when she asks for a manager after the person on the other end won't help her. After explaining the situation yet again, the manager seems sympathetic and is willing to waive the transfer fees. She begins the process of transferring back and then magically says "we can't transfer back because lines in a new account can't be transferred for 60 days". So the only thing you can do is transfer the 3 lines on her parents account to our account for 2 months and then transfer them to another account after that. And guess what, that's $18/line for each line and then another $18/line to transfer off our account. All because transferring back wasn't possible. Gee, well our buddy Drew in the store seemed to think it was possible. So yet again, AT&T comes up with these convenient rules.

So let's re-cap. When you sign up a new account with AT&T and try to transfer your lines, beware that despite them taking significantly more than what you were paying before and binding you to an additional 2 years, they can and will choose to enforce the previous contract when it suits them. So basically you're bound to 2 contracts and they've got you over a barrel by using whichever suits them at the time. You should also not expect to be informed of absolutely anything, especially not contract terms when you sign up for your new old account. Furthermore, what you can and can't do changes from person to person and from minute to minute. Our buddy Drew was going to transfer us back and the manager on the phone was going to transfer us back but then randomly came up with this no-transfer during 60 days rule which conveniently nets AT&T an additional $108 in transfer charges. The best part is, and what no one at AT&T seemed to grasp, was that it was in their interest to just give us a new account and let us sign up for the iPhone because it meant we went from paying $20/month to paying $150/month AND they had us for 2 full years! I hope some AT&T investors stumble across this and realize how poorly managed the company is that they are throwing away money and souring customers.

So at this point, there's not much we can do. We're going to let Elizabeth's mom talk to them and see if she can make any headway. Tomorrow I'm going to file a complaint with the MN Attorney General and the Better Business Bureau for deceitful and fraudulent practices. If Elizabeth's mom doesn't make any headway, I think we'll be contesting the charges with our credit card company and we'll have to see if it is worth filing in small claims court. After that, I'm out of ideas. The only advice I have is: steer clear of AT&T if you can.

Through it all, the Apple Store employees were helpful and pleasant to work with, even going so far as to try and cover for AT&T. It was all too obvious, though, that AT&T was in the wrong. They were truly apologetic that we had to go through such a mess, and didn't want this experience to sour our opinion of Apple and the iPhone. No worries, though, as we got nothing but top notch service from Apple.

Time for bed, it's been a long day.

St. Petersburg, Russia

Sorry for the lack of recent updates. It seems like I've been on a tour for work: Lincoln, Potsdam, and now St. Petersburg, Russia all in the last month or so. And if

St. Petersburg is like no place I've ever been. The diversity and contrast between buildings is amazing. You'll be walking down the street and see buildings with huge golden domes and intricate architecture next to a no-nonsense, utilitarian building that looks like it has been abandoned.

Overall, I've had good luck with the people. Most have been friendly and helpful. The rest have been largely indifferent to my butchering the pronunciation of the few Russian words I've picked up via osmosis.

The biggest adjustment for me is the lack of smoking bans in public areas. I was shocked when we arrived and I saw someone lighting a cigarette in the hotel lobby. It's completely different from the US and is something I don't think I'd want to get used to.

I've had a hard time adjusting to the timezone. It is 9 hours different from home but for whatever reason I haven't been sleeping very much and not on a regular schedule. Part of the problem may be that there's very little darkness at night during the summer. It usually gets dark around midnight and stays dark for 2 hours or so. It's almost like my first weeks on the ice in Antarctica.

I'm looking forward to returning home on Saturday. My flight is early Saturday morning and I arrive back in Minneapolis around 3:30PM if all goes to plan (though I'm not holding my breath with the state of air travel these days). I have to quick rush home from the airport, change, and go to a wedding reception. I doubt I'll make much of a party guest, but I should put in an appearance. After that, I think I'll take a few days to settle back in and get on a normal schedule. I think I'm home for all of a week before I have to pop over to DC for a quick meeting. Then I think I'm going to do everything in my power to spend a full month at home in my new house. Though we'll see what comes up.

Dasvidania.