Your Documents Under the Magnifying Glass

A few years ago I moved my household administrivia to a paperless system. Instead of stacking file folders deep with bills and statements, everything would be scanned & shredded. This greatly helped with storage space - but in a couple of years I ended up with a network drive filled with over 3,000 PDFs, images and documents. Bear in mind the majority of these are scanned documents - so the contents are images instead of machine-readable text. Everything was dumped into a single directory and files were named based on the timestamp of when they were scanned, taking hours to organize documents into folders and sub-folders.

Instead of burning hours sorting documents I started burning hours building a simple set of applications that would read document metadata, attempt to convert the images to text, group documents by common letterhead and then provide a simple search interface over all of it. Since optical character recognition is hit-and-miss, any full-text search should permit proximate indexing and searching to allow for fuzzy matches.

In the end I created two apps: DocMag and DocIndex. DocMag serves as the search front-end and allows users to perform full-text searches on scanned documents, label them with tags and automagically group other documents with the same letterhead or logo. The interface is pretty spartan and uses Spring Boot to build a straightforward integration into Elasticsearch. DocIndex is the batch process that crawls a filesystem and parses the documents using OCR, generates thumbnails, tags similar documents using computer vision-based template matching, and stores document metadata within Elasticsearch.

DocMag was created in Groovy using Spring Boot (Spring Web, Spring Data, etc). I did this mainly to understand how Spring Boot's conventions translated over to the Groovy world... it had been quite a while since I had worked with Grails. It turns out that Groovy, Spring Boot and Thymeleaf complemented each other quite well and make for fairly simple web development.

DocIndex was created with Spring Boot and Java 9 initially. I griped in an earlier post about my problems with Java 9's dependency management, so instead I fell back to the lambda expressions and work queue management within Java 8. This permits multithreaded parsing of discovered files, which then allows for vertically scaling document indexing by adding cores. Horizontal scaling should be possible by replacing the in-memory work queue with a proper shared message broker. There is a "reminder" issue I've already filed to migrate to a proper broker so this can be done sometime in the future.

Both DocMag and DocIndex are deployed as containers within DockerHub. This was especially necessary with DocIndex, as it relied heavily on native libraries for Tesseract OCR and OpenCV. OpenCV was the most contentious - each Linux distribution has a different version of OpenCV, and the version changes quite rapidly. Building containers for distribution allowed me to ensure users got the correct version of native libraries that worked well with their Java bindings.

Another nice feature of the containerized deployment model was composition - I was able to pair the correct revision of Elasticsearch, conditionally include Kibana, and provide a simple web application firewall by placing DocMag behind modsecurity and Apache. Network connections could be maintained between Elasticsearch, modsecurity, and DocMag without any of these interconnects leaking to the "outside" world, allowing me to do things such as only expose modsecurity to outside traffic and only permitting DocMag to receive requests through modsecurity. Elasticsearch could be hidden as well, only available on the internal network managed by Docker Compose.

Deployment can be relatively straightforward; since everything is deployed to Docker Hub as a container, one should just need to download the docker-compose.yml file and issue export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d. This should provision a single-node Elasticsearch instance, start DocMag behind modsecurity, and begin indexing with DocIndex.

If you are stuck digging through mountains of scanned documents, give DocMag a try. Ease of installation is one of its primary goals - so let me know if you find any issues getting it running!

Alarm Clock Hacking by Blocks

A little over two years ago I built an alarm clock intended for hacking by kids, using a web-based Python IDE. When I tested the lessons, I found that kids didn't like messing with Python and only learned enough to get things barely working. Yet, when it came to Scratch Jr or the desktop version of Scratch, they would spend hours at a time. I needed to find a more approachable way to code.

Recently I discovered Blockly, a product from Google for Education. With that framework you can code by blocks and use its transcoder to output JavaScript, Python, Lua, Dart or (ugh) PHP. The transcoder runs entirely client-side, and the output is human-readable - well indented and even commented.

Writing custom blocks turned out to be an easy thing, so I created blocks to modify the LED display, send audio out to a speaker, or react to button presses. Now you can use blocks to program the clock, while retaining all the functionality present in the older Python interface.

If I was going to redo the Hack Clock, this time I wanted to have a presentable site with full hardware and software lessons, for both Python and Blockly. I revamped the Hack Clock website, completed the Python lessons that I left incomplete last time, wrote new Blockly lessons for the new IDE, and completely re-did the hardware how-tos. Lesson writing took up the lion's share of time, since they all needed new images and better testing.

Another bit o' feedback I had received was that installing the Hack Clock software was too much of a pain. I tried to make this a bit easier this time by offering releases within a Debian pkg, although you still needed to use apt to install dependencies. Still, this cuts down installation from over an hour to about ten minutes... and most of those ten minutes is spent twiddling your thumbs while you want for packages to download and install.

The hardware needed tweaking as well. It turns out the Raspberry Pi headphone jack is just a PWM pin hack and it seemed that GStreamer sometimes just couldn't grok it. The headphone jack was never a complete solution either - it required a discrete amplifier to power speakers, and soldering wires onto a 1/8" jack is a GIGANTIC pain. To make the audio hardware easier to cope with, I moved away from the headphone jack to Adafruit's I2S decoder and amplifier. It provided better audio and cleaner installation without increasing my part count or price. It has proven out to be easier for everyone so far.

The old Hack Clock had another embarrassing flaw: it could only handle one button input and couldn't manage output at all. That drove me nuts and was probably the second biggest thing I wanted to fix. With the latest release the Hack Clock can handle as many buttons as you have GPIO pins, and you can also drive output pins as "switches" in code. The code-by-blocks IDE could deal with buttons and switches as simple function blocks - which meant reacting to user input became much easier to code.

Once things were ready, I installed the Hack Clock software in a mission-critical environment: kids' rooms. So far things have gone well; audio has been more reliable than with the headphone jack, and they have been able to tweak the software more easily than with Python. One bit I noticed this round however: kids don't like looking down to read something, then looking back to code it. The next generation Hack Clock should have an interactive demo to guide through the lessons so they never have to glance away from the IDE.

I'd love to hear what other people experience when they try to get the Hack Clock running as well. A hardware list is posted on Hackaday, and all the instructions are at http://hackclock.deckerego.net/. Let me know what you think!

Hack Your Alarm Clock

Kids, teens and adults are more inspired to tinker after they learn the basic framework of what they are tinkering with. Electronics and circuit sets are great, but it really helps to jump-start learning with demos or projects that are heavily documented in the box. After you get your bearings on what works and what breaks, you are more free to experiment and add your own ideas.

To that effort I'm working on a hackable alarm clock based on the Raspberry Pi platform along with several Adafruit/Sparkfun components. The idea is that we can build a customized, tricked-out bedroom alarm clock with some general off-the-shelf components to see how hardware and software work in concert to create something useful. After the project is done, I'm hoping its creators will feel inspired to rip it apart or build new features using the pieces.

I've already run these lessons with a few kids between the ages of 5 and 11, and it was pretty interesting to see what they found interesting. They all wanted to start playing music, but they celebrated being able to have whatever number they wished show up on the LED display. To simply type "77," hit "Save" and refresh the display was enough to send them clapping. They also spent a ton of time building their own enclosures out of Lego, designing the perfect case for their clock.

Right now we have made it to Lesson 6 (the AM/PM indicator), however I have only published up to Lesson 4. More are to come, and I have already found some refinements that need to be made in the previous lessons. The lessons are available at http://hackclock.deckerego.net/, the hardware project is documented on HackADay at http://hackaday.io/hacker/5116-deckerego, and the Python source code for the clock driver is posted on GitHub via https://github.com/deckerego/hack-clock. I will keep these sites updated as the project grows.

Let me know if you try out the project yourself. We are already making some fun hacks, such as using the LED display to show the answers to math problems. I'd love to hear what other people devise!

Why Organizations Should Open-Source Projects

I can understand if traditional businesses often struggle with the concept of contributing to open source projects or maintaining OSS projects of their own. If a business manager looks at software like a physical, inherently valuable object it is often hard to make that same object freely available. After all, if the business spent $5,000 on development of the software, why should it give the result away?

I believe the more appropriate view is that the value of an application isn't realized upon publication like a book or a movie. While each may be products of creative labor, software is never really "finished." A well-kept application is always in a state of flux, adapting to new use cases and fixing defects. The only time you are actually done with an application's codebase is when you abandon it.

When you realize software is never completely done, the big question is how a business effectively maintains the codebase. How are bugs found and then resolved? How are new features prioritized and implemented? How do you keep things going without draining your existing engineering staff? Once those questions come to light, releasing software to an open source community makes a lot more sense.

I'm not saying that all applications a business writes should be publicly released as open source. Apps laden with business logic, code that epitomize your core business, will likely not be re-usable for others in the community and may disclose sensitive business practices. However "glue" libraries such as utilities, messaging or scalability frameworks can be highly re-usable and can be isolated so as not to disclose any core business use cases.

If an application is re-usable and adopted by others in the open-source community, they begin to rely on your app and apply their own critical thinking to its codebase. The larger community may conceive of use cases you haven't yet encountered, or find esoteric bugs that you haven't run into yet. Even better, OSS developers will often contribute code or bug patches to resolve issues or add much desired features. At this point the maintenance efforts for your codebase are distributed among a large and very knowledgeable public, significantly reducing the expense in maintaining the code. By adding multiple points of view new ideas and a pool of developers an app may become more reliable than if you attempted to maintain it on your own. Both Netflix and Twitter have released such projects as open source with great success and community support.

Recently I helped foster a similar initative to release a Spring AMQP component for Apache Camel. There was absolutely no business logic within the codebase, it was a glue component used for message transport and so it needed to be as rock-solid and dependable as possible. Not only that, engineering resources were scarce and AMQP best practices were still new to the team. The more eyes that could review the component and the codebase the better it would be.

After some evaluation we chose GitHub to host the source and Sonatype's OSS repository to host the resulting Maven targets. After the intital import into GitHub was performed, we signed up for Sonatype's OSS repository access and began publishing snapshots. Jenkins, our continuous integration server, would check out the codebase from GitHub and then publish snapshots to Sonatype on demand. Internally we started using the snapshot builds, ensuring we pushed our local changes into GitHub whenever we were ready to distribute another snapshot build.

One question that arose early was what the company's "sponsorship" of the project should be. The company managers and directors wanted to ensure the project didn't carry any organization artifacts with it - for example, packages or classes that carry the company name. However it does make sense to have a single steward of the project, one that monitors submissions and maintains the pipeline. To that end we used GitHub's "team" concept to create a team repository where software engineers were added as owners of the team code. The core team would merge pull requests, monitor issues being submitted and ultimately be responsible for pushing artifacts to the Sonatype repository. If an employee would leave the company they could possibly be removed from the team itself and no longer be granted rights to publish to the Maven repository, however they could still create forks and provide pull requests. This was an added benefit - engineers could continue contributing to the project long after they left the company itself. By open sourcing the project we could both open maintenance to a sea of new developers as well as prevent losing the historical knowledge of old ones.

While the code itself was freely available and the binaries were actively published, the project couldn't necessarily be considered "released" to the community until we began promoting it. The project spanned many other popular projects including Spring, Apache Camel and RabbitMQ. Once the component was in a stable-ish state that could be tested by other developers, posts were submitted to mailing lists for each project. There were varying levels of response, but a few individuals began to express interest and even volunteered to write How-To documents and include it in peer presentations. At the same time I also started to see if I could share lessons learned with the StackOverflow user base, and offered snippets from the component's codebase if I though it could be useful. As a result the GitHub project began to get ranked higher with search ranks, which also helped greatly with visibility.

Once the component started to be used within production we released the initial one-dot-oh release. The Maven release was promoted to the Sonatype stable repository, tweets were tweeted, posts were submitted to mailing lists and we tried to invite as many people as necessary to kick the tires. Once it became easy to include the component in Ivy and Maven dependencies, adoption greatly increased and more people tried the component. As a result we started to see an increase in pull requests, suggestions and bug fixes. There were a few deviations from the AMQP specification that we wouldn't have noticed had not the community taken a critical look at the component and provided patches. Use cases for asynchronous production made for very helpful unit tests and helped prioritize new features. The 1.1.0 release of the component was markedly more robust than the 1.0 release but required less engineering effort on behalf of the team.

To my mind everyone won through the open-sourcing of the camel-spring-amqp project. The company was able to deliver high-quality software hardened through peer review and a global pool of developers were able to re-use a collaborative codebase. Overall cost went down, business value went up and high-fives were copiously distributed to all involved.

Counting on Trolls Under Your Bridge

It seems no matter if you're in a huge corporate project or a smaller cadre of independent developers, the rules of managing people and code remain the same. And open-source projects tend to manage people much better than their corporate brethren.

In projects with multiple people involved you have to continually worry about people leaving, contributing crappy code or going on a complete tangent. Usually in corporate life people just want the thing to work, and if it randomly happens to hit production and not have complete show-stopping errors, great. But the code could be complete cruft and no one would care.

With an open-source project, however, you must have complete transparency. Brian Fitzpatrick and Ben Collins-Sussman recently gave a presentation on managing people in OSS projects effectively. Code reviews, definite goals and communicating "just enough" were all key. Hrm... how many commercial software developers do the same thing?

It's not without its difficulties, however. Especially with the kernel, contributions can be a big mixed bag. Sifting the good contributions from the pointless ones can be time-consuming and tedious. So projects are split into more minute portions, and newbies are either mentored or given a sandbox to play in.

It seems like there is a role model out there for taking a wide variety of people with an even wider variety of backgrounds and time commitments and having them all contribute to a well-developed end product. It's a shame it isn't emulated more often.

Nokia Acquires Trolltech & Qt

Trolltech, maker of my favorite development platform Qt 4, just announced they're being purchased by Nokia. Damn.

History has pretty much shown that the independent, open-source shops being purchased by mega-corps have largely resulted in a slovenly product. Novell's purchase of SuSE has resulted in a distro that kernel dumps... kernel dumps occasionally on bootup. I don't hold fantastic hopes for Qt 4 unfortunately.

It appears that others share my scepticism. A good deal of comments to LWN - which still boasts a pretty coherent readership - seems to also have the sort of timidity stemming from being burned before. The Register, which prints an overall positive article, still feels it needs to assert Nokia has made claims of continuing Qt development.

It feels at the same time like the OSS world is shrinking and expanding. While awareness and adoption is at an all-time high, the high-profile projects are starting to be absorbed into the same machine they raged against.

A Package On Your Doorstep

I first started working with CrystalSpace in 2003. Back then compatibility would change from version to version, building CS was a regular task and CEL was still an up-and-coming project.

Then this January the team announced the first stable release of CrystalSpace; one feature-complete and ready to be production ready. It's amazing how mature the project has become since then... documentation has flourished, CEL has become a fantastic development framework, titles can be developed with little to no coding and Blender integration is now ready for prime time and sponsored by the Blender project itself.

Not only that, CrystalSpace has finally hit the prime time: binary packages are now widely available on most Linux distros. Debian has been carrying packages for a while, but now native SuSE RPM's are available via PackMan.

No, you shut up.

Now people can develop entire titles while only focusing on content creation and scripting, allowing people to focus on what makes a game a game. Not only that, there are copious templates and examples for how to build projects. And you no longer have to build packages every night. And it's open source and readily available. It blows my mind.

Running at 100%

I'm knee-deep in the depths of development. I have so much to tell you, good and faithful reader, but time is a fickle mistress. I already feel extreme pangs of guilt having fallen behind on ConsultComm bugfixes and feature releases.

A huge cheer and huzzah to CrystalSpace for releasing 1.0. This is an amazingly huge accomplishment. If you've ever needed a 3D game engine, look no further now. It's now feature-complete and stable.

Item! Linspire's (formerly Lindows) Click 'N Run software is now being distributed for Linux distributions at large. Now every Linux user can take advantage of it's clickishness and runocity. Some may think that this is just a overly verbose apt-get, but it's not. Since it also can provision commercial software, this may turn out to be the Steam of desktop applications. That, my friend, would be a killer app.

Microsoft's new intellectual property blitz is hilarious. Their understanding of how creators "feel" about copyright law is laughable.

SecondLife's client software is now available under the GPL. Icculus had already been working on a Linux port, but now the whole community gets to jump in and help. Better yet, Linden Labs has just added a staff of several thousand developers that offer bugfixes, ports and ideas free of charge. I'm hoping this works out well for them - all eyes are on the project now.

If you hadn't guessed already, I'll buy anything Carmack makes.