Your Documents Under the Magnifying Glass

A few years ago I moved my household administrivia to a paperless system. Instead of stacking file folders deep with bills and statements, everything would be scanned & shredded. This greatly helped with storage space - but in a couple of years I ended up with a network drive filled with over 3,000 PDFs, images and documents. Bear in mind the majority of these are scanned documents - so the contents are images instead of machine-readable text. Everything was dumped into a single directory and files were named based on the timestamp of when they were scanned, taking hours to organize documents into folders and sub-folders.

Instead of burning hours sorting documents I started burning hours building a simple set of applications that would read document metadata, attempt to convert the images to text, group documents by common letterhead and then provide a simple search interface over all of it. Since optical character recognition is hit-and-miss, any full-text search should permit proximate indexing and searching to allow for fuzzy matches.

In the end I created two apps: DocMag and DocIndex. DocMag serves as the search front-end and allows users to perform full-text searches on scanned documents, label them with tags and automagically group other documents with the same letterhead or logo. The interface is pretty spartan and uses Spring Boot to build a straightforward integration into Elasticsearch. DocIndex is the batch process that crawls a filesystem and parses the documents using OCR, generates thumbnails, tags similar documents using computer vision-based template matching, and stores document metadata within Elasticsearch.

DocMag was created in Groovy using Spring Boot (Spring Web, Spring Data, etc). I did this mainly to understand how Spring Boot's conventions translated over to the Groovy world... it had been quite a while since I had worked with Grails. It turns out that Groovy, Spring Boot and Thymeleaf complemented each other quite well and make for fairly simple web development.

DocIndex was created with Spring Boot and Java 9 initially. I griped in an earlier post about my problems with Java 9's dependency management, so instead I fell back to the lambda expressions and work queue management within Java 8. This permits multithreaded parsing of discovered files, which then allows for vertically scaling document indexing by adding cores. Horizontal scaling should be possible by replacing the in-memory work queue with a proper shared message broker. There is a "reminder" issue I've already filed to migrate to a proper broker so this can be done sometime in the future.

Both DocMag and DocIndex are deployed as containers within DockerHub. This was especially necessary with DocIndex, as it relied heavily on native libraries for Tesseract OCR and OpenCV. OpenCV was the most contentious - each Linux distribution has a different version of OpenCV, and the version changes quite rapidly. Building containers for distribution allowed me to ensure users got the correct version of native libraries that worked well with their Java bindings.

Another nice feature of the containerized deployment model was composition - I was able to pair the correct revision of Elasticsearch, conditionally include Kibana, and provide a simple web application firewall by placing DocMag behind modsecurity and Apache. Network connections could be maintained between Elasticsearch, modsecurity, and DocMag without any of these interconnects leaking to the "outside" world, allowing me to do things such as only expose modsecurity to outside traffic and only permitting DocMag to receive requests through modsecurity. Elasticsearch could be hidden as well, only available on the internal network managed by Docker Compose.

Deployment can be relatively straightforward; since everything is deployed to Docker Hub as a container, one should just need to download the docker-compose.yml file and issue export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d. This should provision a single-node Elasticsearch instance, start DocMag behind modsecurity, and begin indexing with DocIndex.

If you are stuck digging through mountains of scanned documents, give DocMag a try. Ease of installation is one of its primary goals - so let me know if you find any issues getting it running!

Java Jigsaw Puzzles DevOps

Oh man that's a catchy blog title.

For the past couple o' weeks, my after-hours project has been trying out building webapps and batch jobs using the combo of Java 9, Spring Boot 2 milestone releases, Elasticsearch 6.1 and Docker Edge with Docker Compose. Just because I was in a WAF frame of mind I added modsecurity as a web application firewall in front of the app so I could learn a bit more about building WAF rules with Apache 2.

It was a fun lil' exercise, but in the end I found that all the cutting edge releases simply wouldn't play nicely with each other.

One painful exercise was trying to get Java 9 distributions to work within a Docker container just as it would within my desktop environment. Project Jigsaw is an oft-cited future feature of Java that build engineers have been asking for to end the myriad of JavaEE / Java ME / Java Desktop / Java Server distributions. It should help containerization by allowing svelte JRE installations to bootstrap within a minimal OS. However... this new way of distributing JREs with modular components creates yet another dependency management headache for builds. Once you begin writing manifest elements for Jigsaw + Java 9, every library and its mother now needs to be managed by your manifest as well. Its enough to drive you nuts.

Let's say you don't want to jump into building modular JARs yet and just build traditional JARs that don't use Jigsaw dependency management. Well... Ubuntu's OpenJRE 9 distribution doesn't automatically inject some Java 9 foundation libraries (such as javax.image), while Oracle's JDK does. If you use an Oracle JDK locally to develop things may appear just fine, but then you need to perform some command-line overrides for things to work on an OpenJRE 9 build. To make things more hairy, it seems that OpenJDK and Oracle have built implementations that might be runtime compatible but are NOT compatible from a build & deployment standpoint. Command-line arguments are vastly different, even though manifest formats are the same. That makes building standard build & deployment scripts a pain, as well as local testing. Distributing Oracle's JRE within a container is just to fraught for me to attempt - so I stick to distribution with OpenJDK instead.

I ended up burning too much time trying to get a consistent build between my streamlined Ubuntu-powered Docker container and my local MacOS development environment, so I punted back to Java 8. While Java 9 had some nice memory management features and some syntactic sugar, what I really needed was Lambda and Stream support. Java 8 was sufficient for this in both Oracle and OpenJDK-land.

The combo of Spring Boot 2 (milestone 7) and Elasticsearch 6.1.0 was another mix that simply didn't pan out. The Java libraries for Elasticsearch 6 had a few signature changes across the API which were entirely incompatible with Spring Data Elasticsearch, and the protocol between ES 5 and 6 did not appear to be compatible. I'm sure this will get patched up in short order within the Spring project, however until then I had to fall back to Elasticsearch 5.6.4. I wanted to stick with Spring Boot conventions as closely as possible, so I did not go native just for ES 6 support.

In the end... I do have a fully containerized solution using Spring Boot 2, Java 8, Elasticsearch 5.6.4, and modsecurity. Getting WAF protection, a single-node ES cluster, a web front-end and a indexing batch process running in the background all happens with:

export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d

...and that's it! Containers are also available at Docker Hub and require thankfully LITTLE dependency management.

Telling a Tale for Ten Years

Exactly ten years ago I started this ridiculous blog as a way to collaborate with other game developers. At the time I thought I would dig deep and push out at least one title. This eventually led to the "Desktop Distractions" studio concept, and the nascent title Deskblocks. I also worked within the CrystalSpace engine as an entrant to the PlaneShift team. I just couldn't give these projects traction however, so I gave it up and moved on to tinkering.

Even though the driving force behind the blog had faded away, and even though no one else reads this blog (aside from the State of .NET Integration Frameworks post), I kept updating it. Writing - even if it exists only for your own edification - really does help with communication and critical thinking regardless of what you write about. Even though my day job has nothing to do with garage door openers, my posts on my garage door security system helped me organize the build in a way that informed the Hack Clock project. Back in 2006 I began investigating vector processing, and the resulting frameworks have helped me think about and design microservice architectures. Of course, there was plenty of venting as well with my favorite software companies being dissolved or SuSE Linux winning and failing and winning and failing again. All of this writing helped me when performing comparative analysis at work, or designing parallel architectures, or watching trends in software development.

It is hard to believe a decade has slipped by. It doesn't even seem real. I don't think I've evolved much since that one cram session in a crystal chair, but I'm glad to have my collected ramblings to reflect back on.

Re-Examining Development Platforms

I've been granted an opportunity for perspective lately. I've done my best to steer away from .NET development up until recently - not because I had any particular gripes, but the technology platform just never seemed to be a great fit. C# as a language is pretty great... delegates, tail closure and nullable references are a welcome respite. VisualStudio isn't bad. If you judge IDEs by how many times they make you swear in a given workday, I'd say my four-letter word tally is comparable to that of me using Eclipse. In fact, once my development environment was up and running I thought I might grow to enjoy .NET development.

And then I started to use the foundation classes. First off... I'll acknowledge that no platform has ever been able to get dates and times "right." This is ever more apparent with .NET... who for some inexplicable reason have no real sense of this "epoch" thing that EVERY OTHER PLATFORM USES. Milliseconds since year zero are expressed as... integers? On top of that, time span arithmetic is only accurate when math is done using "ticks," which themselves are not really accurate to 1/10000 of a millisecond. I didn't go for the caesium clock upgrade in my current laptop.

Java developers often complain that anonymous classes are inelegant or verbose within Java. C# has abandoned anonymous classes in lieu of their delegate-based event handling system. However, C# developers exposed to anonymous inner classes actually seem to like them. A common gripe actually turns out to be a nifty feature when you're talking about event handling. The C#/.NET event handling mechanism isn't that fantastic... it's largely just a loose convention for using delegates. No extras or nice GoF listener patterns provided like a PropertyChangeListener.

Zooming out from design patterns and looking at .NET from an enterprise integration pattern perspective, the .NET platform is definitely at a major disadvantage when compared to JVM-based platforms. I've already covered the state of .NET integration frameworks but to recap: it's still nearly five years behind. My time with EasyNetQ has been great, but I still find myself wishing I could use Apache Camel to construct bigger things using common EIP components.

Despite the current rant, I've been fairly complacent with my new development platform. What really stirred things back up for me was when I cracked open the JMeter-Rabbit-AMQP plugin so that I could do RPC-based load testing of EasyNetQ services. Being a JMeter extension, JMeter-Rabbit-AMQP was a Java app that required me to fire up NetBeans once again and do some Java hacking. Once I did... damn. Until I made that sudden switch back I didn't see the huge gap that existed between .NET and Java development. While C# has some advantage over the Java language, the JVM platform is still leaps ahead.

Once you begin talking about instrumentation the gap grows even wider. I have grown accustomed to the fantastic introspection and profiling offered by Java Management Extensions and VisualVM; by contrast Microsoft's laughable implementation of Performance Counters has caused me more problems than it has solved. If it works (and it often doesn't due to permissions issues or outright registry corruption) there is no instrumentation that allows for live modification of managed objects or details on garbage collection. The actual .NET API to create and maintain performance counters is actually not bad, but the Performance Counter UI is so clunky and ill-conceived that it is often difficult to make use of it.

In the end... it doesn't matter. You do the best you can with the tenured development stack because ultimately it's not about the underlying technology - it's about the squishy, business-logicy brain inside of it. Keeping that squishy brain... err... braining is the most important thing.

Unbridled IDE Anger

It's midnight as I type this. I have spent the past four hours attempting edit a single freaking Grails project within an IDE. Any IDE.

I switched to NetBeans almost nine months ago and I've been a very happy NetBean'er. Everything worked out of the box with minimum futz and complied quite nicely with accepted standards. Maven 2, Subversion, minimalist Java runtimes and Spring hummed along without complaint. Looking to accelerate development of simple maintenance web applications I turned to Grails... and then things started to fall apart.

Grails actually has some great tooling within NetBeans, but one bug has been a huge thorn in my side. An acknowledged bug causes Grails projects to not notice any external libraries, even if they are your own and open within NetBeans. This is beyond annoying because features like syntax highlighting, error detection and auto-completion go completely awry. The bug is reportedly resolved in the upcoming version 7 of NetBeans, however I installed Beta 7 and the problem continues to occur.

Not being beyond switching back to Eclipse, I decided to try Spring's own IDE. Spring's IDE is a slightly customized version of Eclipse, with a bit nicer front-end dashboard for plugins and Spring product support built-in. Since Grails is a Spring project I figured it would work well.... and indeed it did. My other projects... well...

SpringSource Tool Suite uses the latest beta of Maven 3 for dependency resolution... and you cannot change that. Yes, you can specify alternate Maven installations, but STS completely ignores those settings for dependency resolution. Maven 3 will also refuse to load transient dependencies if it doesn't like your POM as well - which means that you simply cannot load your application with all required libraries if your Maven 2 pom.xml is rejected by Maven 3.

I could go back to my old Eclipse installation... and then I need to find a way to load the Grails plugins into the old Eclipse install. That sounds like loads of fun.

So now I have three choices:

1. Use NetBeans and have absolutely no auto-complete or strongly typed variables
   
2. Use SpringSource Tool Suite and re-do all my Maven 2 POMs
   
3. Use my old Eclipse installation and try to install the necessary Spring plugins
   

Now, bear in mind I've burned four hours trying to STS to work. Another four to get NetBeans to work. How interested do you think I am at trying to get MORE plugins shoved into my old Eclipse installation?

I'm cheezed because NetBeans Grails support is rendered nearly useless by a fairly blatant bug. And I'm cheezed because Eclipse continues to be an albatross to every normal project I have. IntelliJ IDEA - you might just be next.

Until then, I've got a Grails app to blindly maintain while NetBeans continues to yell about imports not being found.

Candyland Gets Paved. Again. Repeatedly.

I'm getting a bit tired of being so curmudgeonly. I can't help it tho. All my toys are being taken away.

First Novell acquires SuSE Linux, which I have been fairly skeptical of since the purchase was announced. Then Nokia ate Trolltech, the place where my favorite Qt was grown. And then Oracle ate Sun Microsystems, the ones who kept expanding the boundaries of software engineering and releasing such leaps to the public (usually (sometimes)). Now Oracle is smacking around robust and growing projects that have served the software engineering good for many years now.

And now Novell is bought by Attachmate, with 882 patents being absorbed by a Microsoft subsidiary. The openSUSE team says everything is "business as usual,", but there are very strong indications that this will not always be so.

Maybe this is just the ebb and flow that is technology capitalism... but all my power tools seem to be disappearing. If Apache and JBoss are bought by Philip Morris, I'm going to freakin' lose it.

Is Oracle Doing what Sun Couldn't? I Can't Bring Myself To Think So.

I'm staring at my mug with the image James Gosling created - Tux standing in front of Sun's gravestone, consoling Duke's loss. Sun left us too early... it was always an amazing playground of tech, giving us cool stuff like Jini and JMX. I was ill at ease with Oracle before their acquisition of Sun, and now I'm even more so.

Still... I noticed that Oracle recently has done something that Sun hasn't been able to do. While Sun tried to govern Java with some sort of loose democracy known as the Java Community Process, it never was able to get consensus with the group and move forward. Perhaps Oracle's benevolent dictatorship is what the JCP needed - for now they have been able to get IBM to join forces with them and work on development of the OpenJDK.

This is a pretty big deal, as Gosling notes. Not only does this collaboration accelerate open Java development but this also possibly reduces fragmentation since IBM will be scaling down their work on the Apache Harmony JVM. RedHat's IcedTea implementation is already married to the OpenJDK codebase, and without IBM's backing of Harmony the OpenJDK implementation quickly becomes a de facto standard.

If Oracle can light a fire under the JCP and put significant engineering efforts around an Open Java development tookit... I dunno. Maybe things will turn out all right after all. I'm still pretty hesitant.

Eclipsed by a Bean Maven

I've long been a fan of NetBeans, but for the past several years I have had to use Eclipse. Nearly all of the projects or organizations I've walked into have had pre-existing standards and ways of doin' stuff, and the worst thing you can do as a greenhorn in a company is try to switch horses mid-stream.

Wow... I get a triple combo score for clichés there.

Now I get to bootstrap a development team and decide what we use. The awesome thing is that there are now something like elebenty kabillion IDEs, RIA frameworks and rapid application environments. JSF no longer blows. Everyone uses the same build files now. XML hell is now in the rear-view mirror. Vendor lock-in isn't what it used to be thankfully enough.

When I started building out projects I went for an IDE with tight Maven 2 integration, JSF 2.0 support, Java EE 6 interwoven and nice Subversion tools. I imagined that I'd be heavily leveraging plugins from the Spring IDE project and plugins from JBoss, notably the Drools Workbench. All this plugin support caused me to initially lean towards Eclipse.

As I went further along I also started using GlassFish v3 instead of Tomcat, exploring JavaFX and increasingly using Maven 2 for continuous integration and artifact generation. I didn't really use the features of Spring IDE at all, and while I did use the Drools Workbench I ended up doing more decision-table type stuff, making drl's easier to manage. Ultimately a deciding moment came when I needed to do a very quick-and-dirty desktop application with a simple user interface... something that just needed a logo, URL text field, username and password. For the life of me I couldn't find any free/OSS plugins for Eclipse that let me create a quick JDesktop or Swing application... at least within the 45 minutes that I had available. After pulling out my hair I had a sudden flashback to the good ole' ConsultComm days and reminisced "boy, creating UIs in NetBeans sure was swell." It was at that point when I tried to think of any reasons why I was sticking with Eclilpse... and came up empty.

I've switched to NetBeans, which the IDE itself made extremely easy with it's Eclipse workspace synchronization. Read that line again... synchronization. I know! Mind blowing! It doesn't import the project, it allows your project to remain in the Eclipse workspace while also being mantained in NetBeans. Easy!

All the UI goodness I remembered was still there. In no time flat I had created the entire UI and help menus. Not once did I consult a HowTo or documentation, instead it all immediately made sense. The entire project was done before lunch.

I have shifted all of my development to NetBeans now, despite the fact that I can't be sure where its future lies. I don't care. I'll take advantage of an easy-to-use IDE as long as I possibly can. If I switch back to Eclipse in the future it will like be because I was forced to... just like every other time in the past.

As for standardizing on a development environment: I've learned that no one IDE fits all. Every developer has a preference, and with Maven 2 reaching near-ubiquity it doesn't matter nearly as much as it used to. I say let every developer chose his or her IDE, just pick the one you work most rapidly and naturally within. As long as I can build it with mvn on the command line I just don't care.

Java EE 6 - Gee, 6?

Java Enterprise Edition 6 was just released, even with Oracle's possible acquisition of Sun looming on the horizon. In the past I've seen J2EE and Java 5 EE releases come and go whilst I went on not caring. It wasn't because I didn't need anything more than Java's Standard Edition... I do... but the Java Enterprise Edition releases always seemed to be released with features I needed nine months ago and found in other frameworks.

I decided to take more of a look within this release and give it a fair shake. This release had a lot of nice stuff but was still playing catch-up with other tech: JPA starts catching up with the functionality offered by Hibernate, JAX-RS (RESTful Web Services) is finally produced as an answer to the myriad of other REST frameworks and dependency injection finally gets introduced into Java proper. All of these additions are features that were already offered by alternate libraries over a year ago (some of them more than five years ago) and are just now becoming standard.

Inversion of Control isn't the only elder pattern that finally debuts in Java EE 6. Asynchronous responses appear throughout the new Enterprise Edition, finally allowing threads to stop blocking subsequent operations. Asynchronous processing is finally available in the Servlet 3.0 spec so inbound HTTP requests don't have to hang forever while they await all the info to formulate a response. Session Beans can now be invoked asynchronously so EJB execution doesn't consume a client thread and eat up threadpools. JSF now has AJAX support for asynchronous webapp calls. REST-based Web Services allow for Web Service exposure with much less resource utilization and much higher concurrency through more streamlined asynchronous HTTP calls. All of these features not only have been present in alternate libraries for some time, but entire protologisms and design patterns have been honed to allow for just such asynchronicity. Comet and callback methods have been a predominant pattern for some time now, although Servlet 3.0 is the first time a Java Enterprise Edition has allowed for asynchronous Servlet processing. Now that WebSockets are becoming more prevalent Java Servlets are going to need to catch up to everyone else yet again and in short order.

Am I even tempted by this release of Java Enterprise Edition? Maybe kinda. I like annotation-based models, as I've seen how deep XML hell truly goes. JAX-RS annotations are a compelling alternative to JAX-RPC, and the new Bean Validation annotations would likely be very helpful. Asynchronous EJB invocation, along with a more minimalistic and annotation-based EJB spec, may even convince me to finally give Enterprise Java Beans a try once more.

I already use JPA, JAXB, JMS, JMX, JAAS, JAX-RPC, JavaSpaces and Servlets/JSPs/JSTL but in more of an a la carte sorta mechanism, intermingled with Spring, DWR, Hibernate and Camel. Now that I've typed the previous line... I realize that even though I've never intentionally downloaded the Java 5 EE package from Sun it seems I've been using Java Enterprise Edition all along. Weird. I guess it has slowly seeped into the crevasses of my frameworks, unintentionally filling out the rest of the libraries I use. Now that we have REST support and asynchronous context... I might walk the final yard and intentionally download it this time.

Candyland Gets Paved

I've been a big fan of Java for quite a long time, using it for nearly all my enterprise software development. Looks like I'm going to have to find something else now.

Today Oracle announced it would purchase Sun
Microsystems, the company that had previously played host to a myriad of great technologies such as the Solaris OS, the Java programming language, the NetBeans integrated development environment, the MySQL database, VirtualBox for desktop virtualization, the GlassFish application server (not to mention an emerging JMS server) and OpenOffice.org for open-source enterprise office software. With Oracle's purchase pretty much a done deal, you can now expect most, if not all, of these technologies to wither on the vine.

I rely on NetBeans, Java, Solaris, OpenOffice and VirtualBox so that I can do my job on a daily basis. With those removed, I'm pretty much screwed.

Think I'm being alarmist? Maybe. I did play Chicken Little when Novell bought SuSE in 2003, and that deal appears to be working out somewhat better than anyone expected - even amidst poorly considered kinships with Microsoft. SuSE is slowly recovering, and Novell seems to attempt to be a good steward. But Oracle? Lessee... what's their track record for aquired technologies? They've turned Tangosol Coherence from a must-have element in a distributed software stack into a minuscule trinket tucked away in their closet. InnoDB has not progressed well and has caused continue enterprise issues. And WebLogic? It used to be the fastest Web service platform out there, now it remains largely ignored.

Tell me... what strategic value is Oracle going to find in VirtualBox? Or OpenOffice.org? Do you really think Oracle would have allowed projects such as Hibernate to exist when they want to make Toplink ubiquitous? Oracle will continue to neglect these projects, just like they've ignored previous projects they've acquired, until they decompose.

Seriously, are you going to trust a company who's had the same impossible-to-navigate site for fifteen years? A company who attempts to license its products using terms that require a slide rule and burnt offerings to figure out? Just look at the difference of how each company announced the acquisition: Sun created a micro-site that explains the deal and attempts to sell a bright side. Oracle could hardly be bothered to post a statement, showing their indifference to the acquisition that will most likely just mean a reduction in competition, not an enhancement to their portfolio.

Right now it seems there are two possible outs: hope that Apache Harmony can deliver on its goal of releasing its own open Java platform, or abandon Java as a platform and move to something like Qt.

Maybe I'm prematurely freaking out. Maybe I'm wrong about Oracle's apathy destroying the projects acquired from Sun. I certainly hope so. Still, I would wager that the capitalist will continue to decimate good ideas, digesting Sun's properties into a discarded pile alongside acquisitions of olde.

Someone to Give Me the Time

It's been really interesting to see the responses from Blitz, Fly Object Space and GigaSpaces concerning state management as well as Newton and Rio concerning service discovery. I'm definitely learning as I go, but the good thing is that it seems like there are many in the community eager to help.

Now I'm working on another issue with enterprise service development - scheduled services. There are some services out there who may want a have an event fire in 1000 milliseconds, or five minutes, or an hour, or somewhere in between. This would appear to be an easy thing to solve at first blush - until you consider volume, quality of service and scalability. It's a steep drop into complexity at that point.

Here's the thing: you could easily just do a scheduled executor in J2SE, but once your VM dies then your pending events die too. You could submit a scheduled job to something like clustered Quartz instances, but then you must have a reliable back-end database to write to (no native replication). You could use something like Moab Cluster Suite, but it seems to live outside the muuuuuuuuch more simple realm of event scheduling.

So let's think outside the box and use some replicated object store that isn't necessarily meant for scheduling. How about we slap a time to live (TTL) on a JMS message, throw it on a queue and wait for it to hit the dead letter queue? That might work at times, but TTLs are really intended for quality of service and not for scheduled events. Unless you have a consumer attached to the former queue constantly polling for messages you're not guaranteed to land in the latter dead letter queue.

How about using Camel's Delayer Enterprise Integration Pattern? Nope - that's just a Thread.sleep on the local VM. Doesn't do you much good once the VM dies. How about a delayed message using JBoss Messaging? I've heard tell that it exists, but I can't find much reference to it in the documentation.

This isn't a new problem - there's even JSR 236 that is intended to address this problem. But it's been hanging around since 2004 with very little activity of note, so I doubt it's going to have much hope of working by Monday.

Until JSR 236 is addressed I'll likely have to just find a way to deal with this on my own. Maybe create a JobStore for Quartz that's backed by a JMS topic? Or just suck it up and build a clustered Quartz instance with a fault-tolerant database?

Gah. Sticky wicket.

Where Are the Java Physics APIs?

If I'm going to try and port Deskblocks to Java then I'll need to find a native Java physics API. If my only choices are native libraries then it makes more sense just to stick to Qt 4 - I'd rather keep this entirely in the "compile once run anywhere domain." Managing native libraries with wrappers functions is just a pain in the butt.

I was surprised to see slim pickins for physics API. For 2D physics I just found Phys2D and for 3D physics I found JBullet, a port of the Bullet physics API. Both seem to be great projects, and both seem to be currently active. Indeed, I'll probably give Phys2D a try and see if I can use it. I guess I just expected an ODE port by now.

Java DeskBlocks

Right before I started my latest hacking-for-cash endeavor I was working on DeskBlocks, a physics sandbox rendered directly on the desktop. I was using Qt 4 for development, mostly so I could use ODE and refresh my C++ coding. One problem in development was that things would work perfectly fine in X11 - rounded edges, nice circles bouncing around, render speed was great - then things would work horribly in Windows. Or vice-versa. I could never get things to behave properly cross-platform.

With Java's big update, it's windowing toolkit now allows for translucent and shaped windows on platforms that support it. That means the rendering issues I had with Qt 4 may be solved with Java 6u10.

It makes me start thinking if I should move the project to Java instead. It appears there are Java bindings for ODE... so it just might work.

Java 6.9

Someone just directed my attention to the Java 6 update 10 intro on Sun's site. What the living...?

First off, this isn't a minor update. This is taking a backhoe to the foundations of Java, hitting a water line, but digging a basement anyway. Why this wasn't released in 7 I don't know... I guess it's because the update is largely centered around deployment of Java as a platform and not adding any functionality to the underlying API. But damn, it's an overhaul.

First, Java is now chopped neatly into libraries, so you only download what you need. That means Java installations can be one-third of what they were in the previous release. Java can now be downloaded and installed more efficiently as well, thanks to some much easier-to-use JavaScript and HTML-fu.

Konqueror has already done this for a while now, but applets will now execute within a full JVM instead of a half-baked nsplugin. This allows for more robust applets and, from my experience with Konqueror, plugins that are more crash-resistant.

Finally the fairly... blech... look of Java has been completely overhauled with Nibus, long at last. Previously I've had to use javootoo's Look and Feel libraries to make things look remotely presentable. Now Nimbus should be able to fill that gap nicely by adding more modern window decorations and UI components.

These were all desperately needed improvements to have Java make inroads into the desktop space. Let's hope it isn't too late.

UML Hell

I've been searching for a UML editor for a while that I like. So you don't have to, I installed (or tried to) several UML editors and took each for a spin. I needed some diagramming mainly for collaboration and presentations... and my choices usually broke down into a) crappy but usable or b) pretty but unusable.

Poseidon for UML Community Edition is "free," but you have to register the product. I dislike typing. Didn't install it.

Gaphor wouldn't even install or run with my Python setup. Tried for 15 minutes then threw in the towel.

Umbrello I've actually used for some time now and consider it my favorite UML editor. When it doesn't crash. Which it does. A lot. I used both the KDE 3.5 and KDE 4 versions, both enjoy the segfault.

UMLet remained up, but the UI just didn't do it for me. It was more a random collection of widgets than an enforced UML diagramming tool.

Violet was one I really, really like. It was simple to the point of minimalism, which I like. However it had some serious UI bugs that made all elements change their text attribute at the same time.

Dia isn't a strict UML editing tool - it's more of a casual diagramming tool. It works really, really well when you want to brainstorm or braindump ideas. But I was looking for something that strictly enforced UML patterns and let me define attributes, methods, classes, sequence diagrams, etc.

ArgoUML, once again, is the only one that can make the cut. This is the open source relative of Poseidon and includes a ton of functionality. ArgoUML has been under active development for years and years, and continues to be the only big player on the block. And with Java WebStart deployment it's exceptionally easy to get cross-platform installation on everyone's machine.

So ArgoUML is still the hands-down cross-platform favorite, with Dia playing a different role yet still the only other contender. At least I finally freakin' settled on one for good.

Trying to Work (Together)

I'm currently bashing my head against a wall. Both physically and metaphysically.

Nothing is cooler than Jini. However, unfortunately nothing is more obscure than Jini. It is something that exists etherally as a specification, not necessarily (although occasionally) as an implementation. It's the how, not the what. It's an adverb. Or something. My brain is lightly fried.

See, you can start with Jini's reference implementation via the starter kit, which exists as a concrete example of the specification. However, the last release was at the end of 2005. This is probably due to this implementation being picked up by the Apache Incubator and turned into Apache River. Apache River does seem to be under active development but has yet to issue an official release and has distributed just one release candidate.

Alright... so the Sun/Apache standard implementation is in transition. Who else has Jini services ready? Rio is a java.net project that appears to be both flexible as well as standards-compliant, so it appears to be a contender. There even appears to be moderate development traffic. However the stable binaries don't seem to match the current on-line documentation, and I haven't been able to download the latest milestone release. It does appear to have Spring integration however - and may yet be a contender. But until the documentation syncs up with the latest stable release, it's a bit hard to follow.

Cheiron also has an implementation with Seven that is Jini-compliant, and appears to be receiving some good development traffic as well. I'm still researching how to go about implementation, and their docs currently match up with their releases. I'm trying to (still) read their documentation and see what I need to do to get up to speed. It appears that most people discussing Jini in forums use Seven Suite as their implementation of choice, although Rio has a strong following as well due to its ease of Spring integration and nice administrative tools.

But for me this means I'm writing a helluva lot of "Hello World" and "Echo" applications, reading until my eyes bleed and trying to figure out how to get this all to work under a local development environment. Jini has been around forever... maybe I'm just having a hard time catching up with the rest of the world.

I'm hoping Spring Integration makes this all a bit more straight-forward for complete knobs like myself. Please oh please tell me they're adding Jini into the next milestone. Having ubiquitous services over a self-healing network is just too good. I'd love to be able to scale just by plugging in a server and walking away.

Jini in the Skull

Now that I have my uber-huge project out the door, I'm trying to think smarter about development in general. I kept thinking that being smarter about development meant thinking bigger - so I initially tried to get more involved in the infrastructure of things. But it wasn't a good fit in my brain - you can't lose yourself in a PowerEdge like you can lose yourself in a stream of bytecode.

This hit home when I was talking to a previously C++ programmer at work. Because I'm a) lazy and b) often muddle explanations, I often tell new Java developers... whilst stammering and mealy-mouthed... that Java passes objects by reference. This quickly helps people understand(ish) why setting a value on an Object inside a method doesn't necessitate a return value. However, when I say that I'm perpetuating a distinctly wrong concept. One I should be ashamed about.

I try to avoid the long conversation that ensues, but I should tell developers the truth. Java always passes by value, never by reference. In fact, it passes references by value. Most people look at me like a Buick is growing out of my head when I say "passes references by value." But the fact is that variables only hold primitives and references, and are always passed by value.

Take a look at Java is Pass-By-Value, Dammit! It all boils down to this fact: C and Java always pass by value... that's why you can't do the standard "swap" function. For example, in C++ you might do:

Object hello = new("hello");
Object world = new("world");

swap(hello, world);

printf("%s", hello);
printf("%s", world);

void swap(Object obj1, Object obj2) {
Object swap = obj1;
obj1 = obj2;
obj2 = swap;
}

This transposes the value in obj1 with the one in obj2. You should see "world hello" from running this craptastic pseudocode. Try this same code in Java and nothing happens - you've locally overwritten some values, but you didn't swap your references:

Object hello = new Object("hello");
Object world = new Object("world");

swap(hello, world);

System.out.println(hello);
System.out.println(world);

void swap(Object obj1, Object obj2) {
Object swap = obj1;
obj1 = obj2;
obj2 = swap;
}

Here you'll just get "hello world" out - your references remained intact, because you passed by value.

This is ultimately a cleaner way to develop... so it's a major plus for both C and Java. And my New Year's resolution is to stop telling people that Java passes by reference just so I can end the conversation sooner.

On the flip side of the complexity coin, I've been reading the the Jini specification by the Jini community as well as Jini and JavaSpaces Application Development by Robert Flenner. Jini has evidently been around forever, but I've only recently become interested in it. Remember the old pre-turn-of-the-century adage that Java would be running on everything in your house, from your kitchen toaster to the fridge? Evidently around 2000 or so Jini was sold as becoming the premier way Jini could allow your toaster to auto-discover your refrigerator and... er... do... heating and cooling... stuff. Who the hell knows. The idea of automagically connected and integrated micro-device clusters communicating across a mesh network is cool, but practical consumer applications are pretty much nil. Then once EJB's started incorporating RMI, Jini came back to the forefront as an easy way to do the heavy lifting of RMI without the thick refried JavaBean layer.

Once you get Jini up and running, it is wicked cool. Start up your Jini registrar and then poof, services get published across a network. Look for a remote service and poof you can discover it and invoke it - no need for stubs or manual marshalling. Once you get the Jini infrastructure up, you don't have to teach developers how to use it... they just implement an interface and the rest is done for them. You can have a mesh network of peer-to-peer nodes up and running within seconds, and the actual node developers don't even know they've done it.

No crappy WSDL's. No UDDI. No thick & slow SOAP XML transport over HTTP. Bytecode and serialized objects all the way. We're not just talking a faster mode of transport, we're talking about delving down two entire network layers.

The application for such technology is mind-boggling... which maybe is why people aren't using it as much as they could. Damn shame, too.

ConsultComm 4 Prototype

I just finished working on the first UI prototype for ConsultComm 4. I'm going to try releasing the pre-alphas as a Java WebStart app - available at http://prdownloads.sourceforge.net/consultcomm/ConsultComm.jnlp .

The JTable/JTree mosh-up is courtesy of SwingX. The code has switched from being very state-driven to being more procedurally driven from events generated by embedded components and JavaBeans. Hopefully this procedurally driven approach will mean less bugs, since actions will only take place at one part of the code. An update to one Bean's property will automatically announce itself to all the other Objects that use said Bean, which means you don't have to force each and every component to maintain the Bean's state.

It's hard to explain. But if you look at the code you'll see a ton of event handlers and action listeners, each of which call a single method. Even small changes make ripples across the codebase, so even distant code can hear when a part of the system has been altered.

I'm not making any sense. To any point, even though the demo is fairly simple there was a lot of work that went into it. And hopefully this work will mean accelerated development afterwards.

Tree on my Table

While I'm deciding on if I should just trash this whole "indy developer" routine I made such a horrible attempt at, I decided to dedicate more time to ConsultComm development.

Desktop integration with Java has been, and continues to be, horrible. It is slowly getting better, but even basic elements are absent or work terribly. The system tray icon works, but isn't any more polished than when it was an incubator project with the JDIC. Now, I'm not trying to be critical of the original System Tray author - he did a fine job. I just expected Sun to make it actually 100% usable.

One thing people have always expected to have was a tree layout for the windowing toolkit with multiple columns for each row. It's not a new concept - TableTrees are in nearly every file browser out there. But for some reason Swing just hasn't had it. Sure, there have been Sun-created articles and tutorials, but no official components. I made one for ConsultComm and have since attempted to make it into a stand-alone component, but it's a good deal of work.

Supposedly the table/tree is making it's way into JSR 296, but those can take a while. I doubt we'll see it soon.

You may notice that I wrote a JDIC component back in the day - SystemInfo. I initially just wanted to donate the code and be done, but was encouraged to make an incubator project out of it. I did so, but quickly became frustrated with Sun's collaboration & project repository Web application and just gave up. Hence the current unusable state the project is in. It appears Sun has now made a separate... something alongside the JDIC mess called "SwingLabs." All the links seem to run in a circle between the JDIC project repository and the SwingLabs site... so I'm not precisely clear on their relationship to each other. But it appears SwingLabs is trying to "productize" or at least "centralize" the disparate desktop components and APIs into a single, coherent package.

In SwingLabs' main package called SwingX they have a new component called JXTreeTable that appears to offer the basic functionality originally put forth in way back in 2003 (probably earlier) with Philip Milne's article. It's fairly basic, but I'm attempting to integrate it into ConsultComm. Relying on a (hopefully) more stable and independent UI codebase should accelerate development. Who knows? I may just fix up SystemInfo if I am able to glean the time. Then again... maybe not. Doing desktop integration via JNI is an insane headache.