Your Documents Under the Magnifying Glass

A few years ago I moved my household administrivia to a paperless system. Instead of stacking file folders deep with bills and statements, everything would be scanned & shredded. This greatly helped with storage space - but in a couple of years I ended up with a network drive filled with over 3,000 PDFs, images and documents. Bear in mind the majority of these are scanned documents - so the contents are images instead of machine-readable text. Everything was dumped into a single directory and files were named based on the timestamp of when they were scanned, taking hours to organize documents into folders and sub-folders.

Instead of burning hours sorting documents I started burning hours building a simple set of applications that would read document metadata, attempt to convert the images to text, group documents by common letterhead and then provide a simple search interface over all of it. Since optical character recognition is hit-and-miss, any full-text search should permit proximate indexing and searching to allow for fuzzy matches.

In the end I created two apps: DocMag and DocIndex. DocMag serves as the search front-end and allows users to perform full-text searches on scanned documents, label them with tags and automagically group other documents with the same letterhead or logo. The interface is pretty spartan and uses Spring Boot to build a straightforward integration into Elasticsearch. DocIndex is the batch process that crawls a filesystem and parses the documents using OCR, generates thumbnails, tags similar documents using computer vision-based template matching, and stores document metadata within Elasticsearch.

DocMag was created in Groovy using Spring Boot (Spring Web, Spring Data, etc). I did this mainly to understand how Spring Boot's conventions translated over to the Groovy world... it had been quite a while since I had worked with Grails. It turns out that Groovy, Spring Boot and Thymeleaf complemented each other quite well and make for fairly simple web development.

DocIndex was created with Spring Boot and Java 9 initially. I griped in an earlier post about my problems with Java 9's dependency management, so instead I fell back to the lambda expressions and work queue management within Java 8. This permits multithreaded parsing of discovered files, which then allows for vertically scaling document indexing by adding cores. Horizontal scaling should be possible by replacing the in-memory work queue with a proper shared message broker. There is a "reminder" issue I've already filed to migrate to a proper broker so this can be done sometime in the future.

Both DocMag and DocIndex are deployed as containers within DockerHub. This was especially necessary with DocIndex, as it relied heavily on native libraries for Tesseract OCR and OpenCV. OpenCV was the most contentious - each Linux distribution has a different version of OpenCV, and the version changes quite rapidly. Building containers for distribution allowed me to ensure users got the correct version of native libraries that worked well with their Java bindings.

Another nice feature of the containerized deployment model was composition - I was able to pair the correct revision of Elasticsearch, conditionally include Kibana, and provide a simple web application firewall by placing DocMag behind modsecurity and Apache. Network connections could be maintained between Elasticsearch, modsecurity, and DocMag without any of these interconnects leaking to the "outside" world, allowing me to do things such as only expose modsecurity to outside traffic and only permitting DocMag to receive requests through modsecurity. Elasticsearch could be hidden as well, only available on the internal network managed by Docker Compose.

Deployment can be relatively straightforward; since everything is deployed to Docker Hub as a container, one should just need to download the docker-compose.yml file and issue export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d. This should provision a single-node Elasticsearch instance, start DocMag behind modsecurity, and begin indexing with DocIndex.

If you are stuck digging through mountains of scanned documents, give DocMag a try. Ease of installation is one of its primary goals - so let me know if you find any issues getting it running!

Java Jigsaw Puzzles DevOps

Oh man that's a catchy blog title.

For the past couple o' weeks, my after-hours project has been trying out building webapps and batch jobs using the combo of Java 9, Spring Boot 2 milestone releases, Elasticsearch 6.1 and Docker Edge with Docker Compose. Just because I was in a WAF frame of mind I added modsecurity as a web application firewall in front of the app so I could learn a bit more about building WAF rules with Apache 2.

It was a fun lil' exercise, but in the end I found that all the cutting edge releases simply wouldn't play nicely with each other.

One painful exercise was trying to get Java 9 distributions to work within a Docker container just as it would within my desktop environment. Project Jigsaw is an oft-cited future feature of Java that build engineers have been asking for to end the myriad of JavaEE / Java ME / Java Desktop / Java Server distributions. It should help containerization by allowing svelte JRE installations to bootstrap within a minimal OS. However... this new way of distributing JREs with modular components creates yet another dependency management headache for builds. Once you begin writing manifest elements for Jigsaw + Java 9, every library and its mother now needs to be managed by your manifest as well. Its enough to drive you nuts.

Let's say you don't want to jump into building modular JARs yet and just build traditional JARs that don't use Jigsaw dependency management. Well... Ubuntu's OpenJRE 9 distribution doesn't automatically inject some Java 9 foundation libraries (such as javax.image), while Oracle's JDK does. If you use an Oracle JDK locally to develop things may appear just fine, but then you need to perform some command-line overrides for things to work on an OpenJRE 9 build. To make things more hairy, it seems that OpenJDK and Oracle have built implementations that might be runtime compatible but are NOT compatible from a build & deployment standpoint. Command-line arguments are vastly different, even though manifest formats are the same. That makes building standard build & deployment scripts a pain, as well as local testing. Distributing Oracle's JRE within a container is just to fraught for me to attempt - so I stick to distribution with OpenJDK instead.

I ended up burning too much time trying to get a consistent build between my streamlined Ubuntu-powered Docker container and my local MacOS development environment, so I punted back to Java 8. While Java 9 had some nice memory management features and some syntactic sugar, what I really needed was Lambda and Stream support. Java 8 was sufficient for this in both Oracle and OpenJDK-land.

The combo of Spring Boot 2 (milestone 7) and Elasticsearch 6.1.0 was another mix that simply didn't pan out. The Java libraries for Elasticsearch 6 had a few signature changes across the API which were entirely incompatible with Spring Data Elasticsearch, and the protocol between ES 5 and 6 did not appear to be compatible. I'm sure this will get patched up in short order within the Spring project, however until then I had to fall back to Elasticsearch 5.6.4. I wanted to stick with Spring Boot conventions as closely as possible, so I did not go native just for ES 6 support.

In the end... I do have a fully containerized solution using Spring Boot 2, Java 8, Elasticsearch 5.6.4, and modsecurity. Getting WAF protection, a single-node ES cluster, a web front-end and a indexing batch process running in the background all happens with:

export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d

...and that's it! Containers are also available at Docker Hub and require thankfully LITTLE dependency management.

Why Organizations Should Open-Source Projects

I can understand if traditional businesses often struggle with the concept of contributing to open source projects or maintaining OSS projects of their own. If a business manager looks at software like a physical, inherently valuable object it is often hard to make that same object freely available. After all, if the business spent $5,000 on development of the software, why should it give the result away?

I believe the more appropriate view is that the value of an application isn't realized upon publication like a book or a movie. While each may be products of creative labor, software is never really "finished." A well-kept application is always in a state of flux, adapting to new use cases and fixing defects. The only time you are actually done with an application's codebase is when you abandon it.

When you realize software is never completely done, the big question is how a business effectively maintains the codebase. How are bugs found and then resolved? How are new features prioritized and implemented? How do you keep things going without draining your existing engineering staff? Once those questions come to light, releasing software to an open source community makes a lot more sense.

I'm not saying that all applications a business writes should be publicly released as open source. Apps laden with business logic, code that epitomize your core business, will likely not be re-usable for others in the community and may disclose sensitive business practices. However "glue" libraries such as utilities, messaging or scalability frameworks can be highly re-usable and can be isolated so as not to disclose any core business use cases.

If an application is re-usable and adopted by others in the open-source community, they begin to rely on your app and apply their own critical thinking to its codebase. The larger community may conceive of use cases you haven't yet encountered, or find esoteric bugs that you haven't run into yet. Even better, OSS developers will often contribute code or bug patches to resolve issues or add much desired features. At this point the maintenance efforts for your codebase are distributed among a large and very knowledgeable public, significantly reducing the expense in maintaining the code. By adding multiple points of view new ideas and a pool of developers an app may become more reliable than if you attempted to maintain it on your own. Both Netflix and Twitter have released such projects as open source with great success and community support.

Recently I helped foster a similar initative to release a Spring AMQP component for Apache Camel. There was absolutely no business logic within the codebase, it was a glue component used for message transport and so it needed to be as rock-solid and dependable as possible. Not only that, engineering resources were scarce and AMQP best practices were still new to the team. The more eyes that could review the component and the codebase the better it would be.

After some evaluation we chose GitHub to host the source and Sonatype's OSS repository to host the resulting Maven targets. After the intital import into GitHub was performed, we signed up for Sonatype's OSS repository access and began publishing snapshots. Jenkins, our continuous integration server, would check out the codebase from GitHub and then publish snapshots to Sonatype on demand. Internally we started using the snapshot builds, ensuring we pushed our local changes into GitHub whenever we were ready to distribute another snapshot build.

One question that arose early was what the company's "sponsorship" of the project should be. The company managers and directors wanted to ensure the project didn't carry any organization artifacts with it - for example, packages or classes that carry the company name. However it does make sense to have a single steward of the project, one that monitors submissions and maintains the pipeline. To that end we used GitHub's "team" concept to create a team repository where software engineers were added as owners of the team code. The core team would merge pull requests, monitor issues being submitted and ultimately be responsible for pushing artifacts to the Sonatype repository. If an employee would leave the company they could possibly be removed from the team itself and no longer be granted rights to publish to the Maven repository, however they could still create forks and provide pull requests. This was an added benefit - engineers could continue contributing to the project long after they left the company itself. By open sourcing the project we could both open maintenance to a sea of new developers as well as prevent losing the historical knowledge of old ones.

While the code itself was freely available and the binaries were actively published, the project couldn't necessarily be considered "released" to the community until we began promoting it. The project spanned many other popular projects including Spring, Apache Camel and RabbitMQ. Once the component was in a stable-ish state that could be tested by other developers, posts were submitted to mailing lists for each project. There were varying levels of response, but a few individuals began to express interest and even volunteered to write How-To documents and include it in peer presentations. At the same time I also started to see if I could share lessons learned with the StackOverflow user base, and offered snippets from the component's codebase if I though it could be useful. As a result the GitHub project began to get ranked higher with search ranks, which also helped greatly with visibility.

Once the component started to be used within production we released the initial one-dot-oh release. The Maven release was promoted to the Sonatype stable repository, tweets were tweeted, posts were submitted to mailing lists and we tried to invite as many people as necessary to kick the tires. Once it became easy to include the component in Ivy and Maven dependencies, adoption greatly increased and more people tried the component. As a result we started to see an increase in pull requests, suggestions and bug fixes. There were a few deviations from the AMQP specification that we wouldn't have noticed had not the community taken a critical look at the component and provided patches. Use cases for asynchronous production made for very helpful unit tests and helped prioritize new features. The 1.1.0 release of the component was markedly more robust than the 1.0 release but required less engineering effort on behalf of the team.

To my mind everyone won through the open-sourcing of the camel-spring-amqp project. The company was able to deliver high-quality software hardened through peer review and a global pool of developers were able to re-use a collaborative codebase. Overall cost went down, business value went up and high-fives were copiously distributed to all involved.

Who Can You Count On During Crunch Time? Turns Out... Nobody.

I've long depended on the software community to save my butt in times of need. And it used to.

It stopped helping this week, and instead started wrecking havoc.

You'll notice I did not say the open source community. And I did not say the Java community. Even tho these two communities are the ones that my latest rant is aimed at. No... this issue has already burned me big time with commercial companies, which is why I left the likes of IBM, Microsoft and Oracle. But now I'm not sitting any better... everyone has sunk to the same level of mediocrity.

Bugs now are being reported, exhaustively, patched and submitted to release managers. And yet months, even years go by without so much as a cursory review. A few good examples come to mind... there were fairly blatant bugs, even typos in a Hibernate dialect for the H2 RDBMS. The author of H2 reported the bug, patched it and even made unit tests for the project. Has the fix even seen daylight? No. It has been open since July of 2008.

Here's an even worse example: thousands (if not millions) of people rely on Apache's Commons Codec library. It's used for string matching, BASE64 encoding and a slew of other things. One of the speech codecs suffers from an ArrayIndexOutOfBounds exception during encoding. A simple mistake to remedy, and one that was remedied and committed to their source repository. Was such an obvious bug ever fixed in a production release? No. In fact, a new release hasn't been made in five years.

And some of the bugs are bad because the maintainers refuse to fix them and label them as a feature. For example, does Spring's Hibernate DAO framework actually begin a transaction when you call... say... beginTransaction()? Nope, beginTransaction is a do-nothing operation. Wow, that makes things easy to troubleshoot and fix.

Okay, so far I've described problems that all have ready work-arounds. That's the only saving grace in these instances - the projects are open-source and so fixes can be applied and binaries re-built. But do you really want patched, out-of-band libraries going into your production system? And what about when you hit the really big problems nary days before the "big release," like finding a fatal, obvious and unfixed bug in your JMS broker? It's been crunch time for two weeks, you're already sleep deprived, your code is absolutely going out in two days... are you going to make a gentle post on the dev list after unit testing a thoroughly researched patch for an obvious bug the maintainers missed? No. You're going to punch the laptop.

Basically I've succumbed to the entropy and decay of all the frameworks I used to depend on. Hibernate Core has over 1500 bugs that have yet to be assigned a release or triaged and doesn't even appear to be actively maintained anymore. Commons Codec hasn't seen a release since July of 2004... kids born during their last release are headed towards elementary school. And the instability of ActiveMQ 5.1 continues to plague its 5.2 release.

The standard reaction to this kind of rant is "if you don't like it, why don't you submit patches?" "Why don't you join the project and help out?" "Stop complaining and contribute!" Yet contributions have been made, entire bugs have been fixed by others MONTHS ago, and yet there addition to the project has netted nothing. What hope is there for a sleep-deprived guy like myself to contribute before his project goes down in flames and the powers that be bail on these frameworks for the rest of their collective careers?