Showing posts with label osgi. Show all posts
Showing posts with label osgi. Show all posts

Wednesday, May 21, 2008

Visualizer, Part 3: Poor Man's PDE Build

This is the third in my series (Part 1, Part 2) of posts about Visualizer. In this post I'll be talking about how to create a simplified PDE build.

As I mentioned in the previous post, Visualizer is built on OSGi. My preferred development environment for doing any Java development, but especially OSGi development, is Eclipse because of its wonderful JDT and PDE tooling. The PDE team has created an awesome environment for developing and managing OSGi bundles. However, one of the requirements that I had for Visualizer was that anyone could download the source code and build it, regardless of their IDE or environment preferences. PDE includes the ability to perform a headless build, but I didn't really want to expect the user to download Eclipse or to include a stripped down version of Eclipse in the Visualizer distribution just so the user could build it from the commandline. So I set out to create a "Poor Man's PDE Build" using just Ant.

Actually building the plugins with Ant is relatively simple. This short Ant file will build the plugin:

<?xml version="1.0" encoding="UTF-8"?>
<project default="build-plugin">
<property name="src.dir" value="src"/>
<property name="classes.dir" value="bin"/>
<property file="META-INF/MANIFEST.MF"/>
<property file="build.properties"/>

<path id="classpath">
<fileset dir="${dist.dir}">
<include name="*.jar"/>
</fileset>
<fileset file="${osgi.framework}"/>
<pathelement path="${java.class.path}"/>
</path>

<target name="init">
<mkdir dir="${classes.dir}"/>
</target>

<target name="compile" depends="init">
<echo message="Compiling the ${Bundle-SymbolicName} plugin"/>
<javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="classpath" debug="true"/>
</target>

<target name="copy-resources">
<echo message="Copying resources"/>
<copy todir="${classes.dir}">
<fileset dir="." includes="${bin.includes}"/>
</copy>
</target>

<target name="build-plugin" depends="compile, copy-resources">
<jar jarfile="${dist.dir}/${Bundle-SymbolicName}_${Bundle-Version}.jar" basedir="${classes.dir}" manifest="META-INF/MANIFEST.MF"/>
</target>

<target name="clean">
<delete includeemptydirs="true">
<fileset dir="${classes.dir}" includes="**/*"/>
</delete>
</target>
</project>


As you can see here, there really isn't much to the actual build. The best part is we can use the META-INF/MANIFEST.MF and build.properties created when we're working in PDE to control the build.

For a single bundle with no dependencies, this effectively duplicates the PDE build process. The difficulties comes in when you start having dependencies. If you use the headless PDE build, it will sort out all the dependencies for you and build your bundles in the proper order.

Implementing proper dependency resolution seemed awfully complicated, especially since PDE build already implements it. Fortunately, Visualizer doesn't require complicated dependency resolution because I've structured the bundles in a logical order. There are three levels of bundles: "core" which implement the main functionality, "ui" which implement the user interface to the core bundles, and "application" bundles that build on both the core and ui bundles.

Armed with this knowledge, we can structure a three stage build process where we first build all of the core bundles then all of the ui bundles and then all of the application bundles. To accomplish this, we have a master build.xml that calls out to the template build-plugin.xml file listed above using a subant task.


<target name="build-framework" depends="init">
<!-- build org.andrill.visualizer, org.andrill.visualizer.services* -->
<subant target="build-plugin" genericantfile="build-plugin.xml" failonerror="false">
<property name="dist.dir" value="../${build.dir}"/>
<property name="osgi.framework" value="../framework.jar"/>
<dirset dir=".">
<include name="org.andrill.visualizer"/>
<include name="org.andrill.visualizer.services*"/>
</dirset>
</subant>
</target>


Here you can see we build first the org.andrill.visualizer bundle and then all of the org.andrill.visualizer.services bundles. As we build each bundle, we copy the bundled JAR file to our dist.dir. Each time a bundle is built, it creates its classpath from all of the JARs in dist.dar. So even though there are dependencies among bundles, we are progressively fulfilling those dependencies by collecting the built bundles in dist.dir.

Once all of the "core" bundles are built, we can kick off the build of the ui bundles:


<target name="build-ui" depends="build-framework">
<!-- build org.andrill.visualizer.ui* -->
<subant target="build-plugin" genericantfile="build-plugin.xml" failonerror="false">
<property name="dist.dir" value="../${build.dir}"/>
<property name="osgi.framework" value="../framework.jar"/>
<dirset dir=".">
<include name="org.andrill.visualizer.ui*"/>
</dirset>
</subant>
</target>


Finally we can build all of the "application" bundles by excluding everything we've already built:

<target name="build-apps" depends="build-framework, build-ui">
<subant target="build-plugin" genericantfile="build-plugin.xml" failonerror="false">
<property name="dist.dir" value="../${build.dir}"/>
<property name="osgi.framework" value="../framework.jar"/>
<dirset dir=".">
<include name="*.*"/>
<exclude name="org.andrill.visualizer"/>
<exclude name="org.andrill.visualizer.services*"/>
<exclude name="org.andrill.visualizer.ui*"/>
<exclude name="${build.dir}"/>
<exclude name="${dist.dir}"/>
</dirset>
</subant>
</target>


You can check out the full build file at: build.xml and build-plugin.xml.

It's not nearly as neat as just kicking off a PDE build and letting it do all of the hard work of figuring out the dependencies for you. However, I'm rather fond of my approach because it keeps me honest. If I create a new bundle and it starts breaking the build, then I know I need to go back and make sure I've thought through the dependencies and am not trying to mix "core" code with "ui" code and such. And there's no need to bundle Eclipse with the source to build the thing.

Tuesday, May 06, 2008

Visualizer, Part 2: OSGi & Native Libraries

This is the second in my series (Part 1) of posts about Visualizer. In this post I'll be talking about packaging an OSGi bundle that includes native libraries.

Visualizer uses the OpenGL bindings provided by JOGL project to display images and data. JOGL provides a series of platform-specific downloads that include a standard JAR file of Java classes and a set of native libraries for variety of platforms. This complicates the deployment of Visualizer because we need to install the appropriate version of JOGL for the user's platform. One option is to mimic JOGL and provide platform-specific builds of Visualizer that includes the appropriate version of JOGL. This isn't ideal because it adds extra steps to the build process and can introduce confusion for users trying to figure out which version of Visualizer they should download.

Fortunately, there's another option: OSGi. In a nutshell, OSGi is a component framework specification that allows you to assemble and manage applications as a collection of components (bundles). I'm not really doing OSGi justice so if you don't know what it is, you owe it to yourself to check it out. And odds are you've probably already used something built on OSGi because it seems to be everywhere these days.

Anyhow, OSGi elegantly solves our Visualizer deployment problem by allowing us to provide a single download. We simply combine the Java classes and all of the platform-specific native libraries provided into a single bundle and OSGi will detect and extract the appropriate set of native libraries based on the user's platform.

The first step was to download all of the JOGL packages for the platforms you want to support. I have users on Linux (32 & 64 bit), Mac OS X, and Windows (32 & 64 bit), so I downloaded all of these. From these downloads, I kept one copy of the JOGL JAR files and collected all of the native libraries.

The next step was to use Eclipse's excellent PDE tooling to create a new "Plugin Project from existing JAR files". I called it 'jogl' and pointed it at the JOGL JAR files. It sucked in all of the class files and spat out an OSGi bundle. If there were no native libraries, we'd be done.

Since we have native libraries, I copied them into the jogl bundle directory using a straightforward directory structure:


The 'native' directory structure is not required; you can use whatever makes sense to you. As you can see from the screenshot, I've got a set of libraries for 5 platform/processor combinations.

The final step is to make the OSGi framework aware of the libraries so it can extract the appropriate libraries when it starts up. This requires using the Bundle-NativeCode header in your bundle manifest:

Bundle-NativeCode: native/macosx/libgluegen-rt.jnilib;
native/macosx/libjogl_cg.jnilib;
native/macosx/libjogl_awt.jnilib;
native/macosx/libjogl.jnilib;
osname=mac os x;
processor=x86;
processor=ppc,
native/linux/x86/libgluegen-rt.so;
native/linux/x86/libjogl_cg.so;
native/linux/x86/libjogl_awt.so;
native/linux/x86/libjogl.so;
osname=linux;
processor=x86,
native/linux/x86-64/libgluegen-rt.so;
native/linux/x86-64/libjogl_cg.so;
native/linux/x86-64/libjogl_awt.so;
native/linux/x86-64/libjogl.so;
osname=linux;
processor=x86-64,
native/windows/x86/gluegen-rt.dll;
native/windows/x86/jogl_cg.dll;
native/windows/x86/jogl_awt.dll;
native/windows/x86/jogl.dll;
osname=win32;
processor=x86,
native/windows/x86-64/gluegen-rt.dll;
native/windows/x86-64/jogl_cg.dll;
native/windows/x86-64/jogl_awt.dll;
native/windows/x86-64/jogl.dll;
osname=win32;
processor=x86-64


One quick note: watch the whitespace when editing the bundle manifest. The OSGi specification is explicit about where whitespace is allowed and where it is required.

With this final piece we can JAR up our class files, native libraries, and bundle manifest and we should be able to use it in any OSGi implementation. When the OSGi implementation loads our bundle, it will extract the appropriate set of native libraries based on the user's osname and processor properties and make sure those libraries are available on the classpath.

I've tested this JOGL bundle on the Equinox implementation of OSGi across Mac, Windows, and Linux and it works great. If anyone is interested, I can make the pre-built bundle of JOGL available for download.

Thursday, April 10, 2008

Visualizer, Part 1: Introduction

On and off over the past couple of months, I've been working on an OpenGL visualization application, called Visualizer (original, I know), for viewing high resolution core imagery and data. Visualizer is built on and with numerous technologies: Java, OSGi, JOGL, Eclipse, Ant, etc. Along the way, I've collected a fair number of tips, tricks, and tidbits that I'm going to share here. I hope to make Visualizer posts a regular feature of the blog, at least until the tidbits run out. If you like it, please let me know by commenting.

A quick aside here: Those of you in geosciences might recognize this concept as Corelyzer. My goal was to create a simplified, streamlined version of Corelyzer that would be easy to deploy on laptops/desktops; a sort of Corelyzer "lite" edition if you will. Visualizer doesn't have nearly as many features as the current version of Corelyzer, but it also doesn't suffer some of Corelyzer's warts.

I thought I'd start off with quick introduction of what Visualizer is for folks who aren't familiar with Corelyzer. During scientific geological drilling, hundreds or thousands of core samples are taken, imaged, and analyzed. This results in several (tens of) gigabytes of data and imagery. Collecting all of this data is of little use unless you have a way to view it. This is where tools like Corelyzer and Visualizer come in. They provide a way to view the images and data in context:


This picture illustrates Corelyzer in action. Stretched across the two 30" cinema screens is a high resolution image of core drilled in Antarctica. The user can zoom in or out and pan left or right to view the all of the imagery collected on the expedition.

So now that you know a bit of the background, let's look at the requirements (in no particular order) I had for Visualizer:
  • Cross platform - I have users on Mac, Windows, and Linux so I need a solution that will work for all of them.
  • Easy to deploy - Visualizer should work out of the box with no complicated setup for most users.
  • Easy to update - Needs a fairly automated mechanism to distribute updates and new features.
  • Open, extensible platform - Building on an open, extensible platform will allow me or others to quickly add new features or customize to a specific groups needs.
  • Reasonable performance - Visualizer should be able to handle 100m or so of imagery and data with reasonable performance on commodity hardware.
In the next blog post, I'll talk about the overall design of Visualizer and specific technologies employed. Future blog entries will include: "OSGi & Native Libraries", "Poor Man's Eclipse PDE Build", "Nifty Ant Tricks", "Automated Update Checks", and anything else that I can think of.

Cheers.

Tuesday, April 08, 2008

Antarctica Service Medal

Not too much technical content tonight. I spent most of the afternoon trying to get OSGi working from a Java Webstart application without much luck. It may be something that I revisit down the road, but for the time being distributing a zip with a set of shell scripts should work. As part of the process, I also learned how to sign a Jar file, which will be useful in an upcoming project.

This morning I pushed out the first update to the Grails app I have in production. Just before performing the actual update, I remembered to double check that my production DataSource was set to 'update' instead of 'create' or 'create-drop' the database. It was lucky I checked because it was set to 'create' from when I was wrangling with MySQL. I had already made a dump of the database, so I wouldn't have lost any data, but I was glad that I checked.

On the non-technical front, I received an Antarctica Service Medal in the mail for the time I spent in Antarctica:


I probably won't have too many occasions to actually wear it, but the medal is a nice memento of my time down there.

And the countdown has begun: 9 days until I'm in the Bahamas for my wedding and honeymoon. Pretty crazy!