Building a Reclaimed Kubernetes Cluster

Thanks to family hand-me-downs, over the passing years I have become a repository of unwanted laptops. Some of them barely boot anymore, but three-quarters of them have two cores and 2 GB of RAM or more. One actually had four cores and 8 GB of RAM... making it a veritable workhorse. I could make them disposable workstations, but instead I wove them together and created a personal Kubernetes cluster.

The base OS for the nodes is Ubuntu's latest LTS release. Rather than using Conjure on MaaS to set up the cluster (which required an isolated network for bootp and DNS and meh), I leveraged Kubespray's flurry of Ansible scripts to prep an inventory of machines over SSH. This ended up being surprisingly low impact and worked perfectly for the use case of building a test lab with piecemeal hardware.

Laptops work just fine as server nodes with a few tweaks:

• Even if you use the server distribution of Ubuntu, laptop events such as closing the lid will still result in a suspend/hibernate/resume action. Edit /etc/systemd/logind.conf to make sure the laptop keeps running when closed:
  sudo vi /etc/systemd/logind.conf
HandleLidSwitch=ignore
sudo service systemd-logind restart

• The display will remain on once you start ignoring LidSwitch events - run a script at startup to turn the display off and save energy.
• Even if you are running in console mode, NVIDIA Optimus laptops will go nuts and seemingly run the discrete and on-chip GPUs nonstop, overheating the machine. Install Ubuntu's Bumblebee packages to prevent this:
  sudo apt-get install bumblebee bumblebee-nvidia primus linux-headers-generic
• As with all Kubernetes nodes, disable swap by commenting out the partition in /etc/fstab. Since you will no longer need to resume from hibernate mode on the laptop, it can be safely disabled

Once you have the laptops prepped and the latest updates applied, you will need to make sure each node has a copy of python-netaddr installed: sudo apt-get install python-netaddr Ansible issues its commands over SSH, so ensure you have keyfile-based authentication set up from the machine you will be running Kubespray on to each of the nodes. If you don't already have an SSH key generated (for example, if you will run Kubespray on the master node), then you can generate a passwordless one via ssh-keygen. After that, copy the public key to each node with:

ssh-copy-id node1
ssh-copy-id node2
...


After that, the machine you are running Kubespray on will need Ansible installed. I ran Kubespray on the master node to keep things simple - so on that Ubuntu box I issued:

sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible
git clone https://github.com/kubernetes-incubator/kubespray.git
cp -rfp inventory/sample inventory/mycluster

This will:

1. Install Ansible on the box
2. Download the Ansible scripts from Kubespray
3. Creates a new Ansible inventory called "mycluster" that is a clone of the Kubespray sample

An important thing to remember is that you address nodes by straight IP address - not by hostname. This is especially important with Ansible scripts because the node's hostname may well change as part of the installation process. If your nodes are fetching their IP address via a DHCP server, ensure the DHCP server has static IP allocations for your nodes.

Once you have all the IP addresses for your nodes, set them in your inventory file. An easy way to do this at the command line is:

declare -a IPS=(192.168.1.32 192.168.1.36 192.168.1.40)
CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}


Verify the inventory is correct by cracking open inventory/mycluster/hosts.ini - if you want to change hostnames, now is the time.

I would recommend having Kubespray build a kubectl configuration file automagically for you. To have this generated as an artifact, change inventory/mycluster/group_vars/k8s-cluster.yml to have the following entry set: kubeconfig_localhost: true After these tweaks you should be ready to launch Kubespray's Ansible playbook. Note that Ubuntu's convention is to have you operate as a normal user and sudo all of your commands, so you will need to use Ansible's --become parameter:

ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml --ask-become-pass --become


At this point Kubespray will try its best to get a cluster up and running on the nodes specified in your inventory file. At the very end Kubespray will provide you with a kubectl configuration file in artifacts/admin.conf, which you can then copy or merge into another workstation's ~/.kube/config file.

Once you have the Kubernets configuration file set on your workstation, you can use it to fetch an authtoken to get into the Kubernetes Dashboard. The proper way to do this is to generate a new system secret that has the appropriate permissions to interrogate the running cluster... but the lazy way is to just steal the token used by Kubernetes' namespace controller.

I'm lazy, so first I list all the secretes in the kube-system namespace:

kubectl -n kube-system get secrets

And then fetch the token for the namespace controller:

kubectl -n kube-system describe secret namespace-controller-token-???

So that I can use it to login to the web dashboard:

kubectl proxy &
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

Now you should have a working cluster you can mess with!

So that the laptops were properly ventilated, I placed each vertically into a metal document sorter from an office supply store. This gives me a nifty vertical rack for the laptops that has plenty of air circulation and allows me to route cables out of the way.

I've constructed one weird frakencluster - but it works!

Your Documents Under the Magnifying Glass

A few years ago I moved my household administrivia to a paperless system. Instead of stacking file folders deep with bills and statements, everything would be scanned & shredded. This greatly helped with storage space - but in a couple of years I ended up with a network drive filled with over 3,000 PDFs, images and documents. Bear in mind the majority of these are scanned documents - so the contents are images instead of machine-readable text. Everything was dumped into a single directory and files were named based on the timestamp of when they were scanned, taking hours to organize documents into folders and sub-folders.

Instead of burning hours sorting documents I started burning hours building a simple set of applications that would read document metadata, attempt to convert the images to text, group documents by common letterhead and then provide a simple search interface over all of it. Since optical character recognition is hit-and-miss, any full-text search should permit proximate indexing and searching to allow for fuzzy matches.

In the end I created two apps: DocMag and DocIndex. DocMag serves as the search front-end and allows users to perform full-text searches on scanned documents, label them with tags and automagically group other documents with the same letterhead or logo. The interface is pretty spartan and uses Spring Boot to build a straightforward integration into Elasticsearch. DocIndex is the batch process that crawls a filesystem and parses the documents using OCR, generates thumbnails, tags similar documents using computer vision-based template matching, and stores document metadata within Elasticsearch.

DocMag was created in Groovy using Spring Boot (Spring Web, Spring Data, etc). I did this mainly to understand how Spring Boot's conventions translated over to the Groovy world... it had been quite a while since I had worked with Grails. It turns out that Groovy, Spring Boot and Thymeleaf complemented each other quite well and make for fairly simple web development.

DocIndex was created with Spring Boot and Java 9 initially. I griped in an earlier post about my problems with Java 9's dependency management, so instead I fell back to the lambda expressions and work queue management within Java 8. This permits multithreaded parsing of discovered files, which then allows for vertically scaling document indexing by adding cores. Horizontal scaling should be possible by replacing the in-memory work queue with a proper shared message broker. There is a "reminder" issue I've already filed to migrate to a proper broker so this can be done sometime in the future.

Both DocMag and DocIndex are deployed as containers within DockerHub. This was especially necessary with DocIndex, as it relied heavily on native libraries for Tesseract OCR and OpenCV. OpenCV was the most contentious - each Linux distribution has a different version of OpenCV, and the version changes quite rapidly. Building containers for distribution allowed me to ensure users got the correct version of native libraries that worked well with their Java bindings.

Another nice feature of the containerized deployment model was composition - I was able to pair the correct revision of Elasticsearch, conditionally include Kibana, and provide a simple web application firewall by placing DocMag behind modsecurity and Apache. Network connections could be maintained between Elasticsearch, modsecurity, and DocMag without any of these interconnects leaking to the "outside" world, allowing me to do things such as only expose modsecurity to outside traffic and only permitting DocMag to receive requests through modsecurity. Elasticsearch could be hidden as well, only available on the internal network managed by Docker Compose.

Deployment can be relatively straightforward; since everything is deployed to Docker Hub as a container, one should just need to download the docker-compose.yml file and issue export DOCUMENT_HOST_DIR=/mnt/documents && docker-compose up -d. This should provision a single-node Elasticsearch instance, start DocMag behind modsecurity, and begin indexing with DocIndex.

If you are stuck digging through mountains of scanned documents, give DocMag a try. Ease of installation is one of its primary goals - so let me know if you find any issues getting it running!

Alarm Clock Hacking by Blocks

A little over two years ago I built an alarm clock intended for hacking by kids, using a web-based Python IDE. When I tested the lessons, I found that kids didn't like messing with Python and only learned enough to get things barely working. Yet, when it came to Scratch Jr or the desktop version of Scratch, they would spend hours at a time. I needed to find a more approachable way to code.

Recently I discovered Blockly, a product from Google for Education. With that framework you can code by blocks and use its transcoder to output JavaScript, Python, Lua, Dart or (ugh) PHP. The transcoder runs entirely client-side, and the output is human-readable - well indented and even commented.

Writing custom blocks turned out to be an easy thing, so I created blocks to modify the LED display, send audio out to a speaker, or react to button presses. Now you can use blocks to program the clock, while retaining all the functionality present in the older Python interface.

If I was going to redo the Hack Clock, this time I wanted to have a presentable site with full hardware and software lessons, for both Python and Blockly. I revamped the Hack Clock website, completed the Python lessons that I left incomplete last time, wrote new Blockly lessons for the new IDE, and completely re-did the hardware how-tos. Lesson writing took up the lion's share of time, since they all needed new images and better testing.

Another bit o' feedback I had received was that installing the Hack Clock software was too much of a pain. I tried to make this a bit easier this time by offering releases within a Debian pkg, although you still needed to use apt to install dependencies. Still, this cuts down installation from over an hour to about ten minutes... and most of those ten minutes is spent twiddling your thumbs while you want for packages to download and install.

The hardware needed tweaking as well. It turns out the Raspberry Pi headphone jack is just a PWM pin hack and it seemed that GStreamer sometimes just couldn't grok it. The headphone jack was never a complete solution either - it required a discrete amplifier to power speakers, and soldering wires onto a 1/8" jack is a GIGANTIC pain. To make the audio hardware easier to cope with, I moved away from the headphone jack to Adafruit's I2S decoder and amplifier. It provided better audio and cleaner installation without increasing my part count or price. It has proven out to be easier for everyone so far.

The old Hack Clock had another embarrassing flaw: it could only handle one button input and couldn't manage output at all. That drove me nuts and was probably the second biggest thing I wanted to fix. With the latest release the Hack Clock can handle as many buttons as you have GPIO pins, and you can also drive output pins as "switches" in code. The code-by-blocks IDE could deal with buttons and switches as simple function blocks - which meant reacting to user input became much easier to code.

Once things were ready, I installed the Hack Clock software in a mission-critical environment: kids' rooms. So far things have gone well; audio has been more reliable than with the headphone jack, and they have been able to tweak the software more easily than with Python. One bit I noticed this round however: kids don't like looking down to read something, then looking back to code it. The next generation Hack Clock should have an interactive demo to guide through the lessons so they never have to glance away from the IDE.

I'd love to hear what other people experience when they try to get the Hack Clock running as well. A hardware list is posted on Hackaday, and all the instructions are at http://hackclock.deckerego.net/. Let me know what you think!

Raspberry Pi Finally Conquers Userland

Raspberry Pi developers have had quite a coup on their hands this past few weeks. The "official" Raspberry Pi Linux distribution Raspian was just upgraded to Debian 8, or "Jessie." This provides a huge number of wins - the 4.1 release of the Linux kernel, latest glibc and build chain updates, more native packages (like Node.JS and wiringPi), and device trees. Oh, sweet device trees.

While the current Raspian distribution still relies on wiringPi 2.24, the most recent 2.29 version has a much nicer way of addressing GPIO in userspace by exposing the GPIO ports in /dev/gpiomem. All too often Raspberry Pi developers run GPIO apps as root to access the array of general purpose I/O pins, however this leads to all the lovely security holes and vulnerabilities that privileged access brings. You never want Apache or Python or any user-created apps running as root - so instead you must find a way to export these ports and allow unprivileged users to access them. Traditionally this has been done using wiringPi's export utility, however the latest gpiomem exposure seems to be much cleaner.

With Jessie I've been able to significantly cut the complexity of installing Garage Security and Sprinkler Switch. I don't need to manually install wiringPi, Node.JS, Video4Linux and a number of other packages. Things seem to largely "just work" as one might expect of a modern distro. One example is that Motion has been updated and appears to be pre-packaged on Raspian, and the necessary Video4Linux bcm2835-v4l2 kernel module properly creates a /dev/video0 device. CPU utilization appears to be much lower with the current stack, and it appears that I can just tweak Motion's configs to save videos in an HTML 5-friendly way rather than transcoding them with a script.

Garage Security and Sprinker Switch are being updated now for Jessie and testing is underway... the new Jessie builds are looking very promising so far.

Telling a Tale for Ten Years

Exactly ten years ago I started this ridiculous blog as a way to collaborate with other game developers. At the time I thought I would dig deep and push out at least one title. This eventually led to the "Desktop Distractions" studio concept, and the nascent title Deskblocks. I also worked within the CrystalSpace engine as an entrant to the PlaneShift team. I just couldn't give these projects traction however, so I gave it up and moved on to tinkering.

Even though the driving force behind the blog had faded away, and even though no one else reads this blog (aside from the State of .NET Integration Frameworks post), I kept updating it. Writing - even if it exists only for your own edification - really does help with communication and critical thinking regardless of what you write about. Even though my day job has nothing to do with garage door openers, my posts on my garage door security system helped me organize the build in a way that informed the Hack Clock project. Back in 2006 I began investigating vector processing, and the resulting frameworks have helped me think about and design microservice architectures. Of course, there was plenty of venting as well with my favorite software companies being dissolved or SuSE Linux winning and failing and winning and failing again. All of this writing helped me when performing comparative analysis at work, or designing parallel architectures, or watching trends in software development.

It is hard to believe a decade has slipped by. It doesn't even seem real. I don't think I've evolved much since that one cram session in a crystal chair, but I'm glad to have my collected ramblings to reflect back on.

A Drift Into Failure

I'm still working towards catching up on my Christmas ready. I already wrote my missives on Thinking in Systems and A Pattern Language; next up is the DevOps favorite Drift into Failure.

The basic premise of Drift is that failures, even massive ones, don't (usually) happen because of a vast conspiracy or from the deeds of evil people. Massive failures occur from behavior that is considered completely normal, even accepted, as part of a daily routine. These routines give our perspectives tunnel vision and often don't allow us to see the underlying issue. Production goals, scarce resources and pressure on performance causes drift in these routines that slowly erode safe practices.

Fatal aircraft crashes and space shuttle disasters are often quoted in the book, however every operations or software engineer in IT has seen this play out a gazillion times before. The site goes down on a regular basis... and no one knows quite why. After digging and pushing new code and re-pushing bug fixes for many sleepless nights, one often finds out that the outage was due to a routine maintenance task gone awry. Maybe a query optimization cache was manually flushed within the production RDBMS, causing the entire cluster to freak out and create a bad query plan. It seemed perfectly sane at the time and even if every single person knew this was going to happen the day before, it likely wouldn't have been caught.

Drift points out how remediation and "root cause" reporting is often fruitless. The concept of high-reliability organizations was pushed in the 1980's as an entire school of thought, focused on errors and incidents as the basic units of failure. Dekker demonstrates that "the past is no good basis for sustaining a belief in future safety," and such a focus on root-cause analysis often does not prevent future incidents. The traditional "Swiss Cheese Model" for determining cause has attempted to see where all of the holes within established safety procedures line up, so as to create a long gap through which problems can drive themselves through. This type of reductionist thinking where atomic failures create linear consequences has turned out not to be predictive after all - instead we need to look at things through the lens of probability.

One of the best practices that anyone, including those supporting enterprise software, can encourage to avoid failure is to be skeptical during the quiet times and always invite in a wide range of viewpoints and opinions. Overconfidence can be your downfall, and dissent is always a healthy way to get new perspective. Dekker quotes Managing the Unexpected to point out that "success... breeds overconfidence in the adequacy of current practices and reduces the acceptance of opposing points of view." Those that were not technically qualified to make decisions often were the ones that made them, or outside pressures (event subtle ones) caused trade-offs in accepted practices. Redundancies that were supposed to make things highly available often make systems more complicated and, in turn, actually make them more likely to fail.

The best way to avoid a drift into failure is to invite outside opinion, even bring in disparate practice groups. Take minority opinions seriously. Don't distill everything to a slideshow. Be wary of adding redundancies and failsafes - often the most simple solution will be the most reliable. The recent re-invigoration in microservices is a great example of this - by simplifying the pieces of a complex system, we can allow each component to work in isolation and ignore the remainder of the system. This allows the system to grow, adapt and evolve without support systems usually provided for monolithic software stacks.

Drifting into failure occurs when an organization can't adapt to an increasingly complex environment. Never settle, always embrace diversity and keep exploring new ways to evolve. A great quote from Dekker is "if you stop exploring, however, and just keep exploiting [by only taking advantage of what you already know], you might miss out on something much better, and your system may become brittle, or less adaptive."

A Systems Language

A Pattern Language is an interesting book to pick up, and that's not just a joke about the size of the volume. Its web site betrays how old the book actually is; it was published in 1977 based on research that had been ongoing for several years. It's scope is pretty large and covers everything from the layout of an office building to the composition of an entire town. Much attention is focused on how to build communities within these spaces, and a lot of research provides evidence on optimal ways of building and tearing down boundaries.

Of particular interest to me were chapters concerning self-governing workshops and offices. The book stresses that no one enjoys their work if they are a cog in a machine. Instead, "work is a form of living, with its own intrinsic rewards; any way of organizing work which is at odds with this idea, which treats work instrumentally, as a means only to other ends, is inhuman." This is a fairly strongly worded assertion that means that employees must feel empowered in order to construct meaningful product.

Just as Thinking in Patterns postulated that groups should autonomously self-organize in order to realize their greatest efficiency, A Pattern Language encourages the formation of self-governing workshops and offices of 5 to 20 workers. A chapter is dedicated to the federation of these workgroups to produce complex artifacts - such as several independent workshops working in concert to build much larger things.

A Pattern Language also encourages keeping service departments small (less than 12 members) and ensuring that they can work without having to fight red tape. This applies to many shared services departments in both government as well as public sector organizations; departments and public services don't work if they are too large as the human qualities vanish. One must fight the urge to make an "idiot-proof system," since this can cause the system to devolve to the point that only idiots will run it.

The book is largely about physical space of course, so it has many recommendations on how offices should be connected. The authors specifically studied what isolated groups within a company, and even what we might consider small physical distances amounted to big interruptions in communication. If two parts of an office are too far apart, people will not move between them as often as they need to. If they are a floor apart, they sometimes will not speak at all.

Ultimately A Pattern Language has a lot of common sense to offer up about how to build a work community, backed by a fair amount of research that bucked many trends in the 70's. It had points that should not be glossed over even now, including:

You spend 8 hours at work - there is no reason it should be any less of a community

Workplaces must not be too scattered, nor too agglomerated, but clustered in groups of 15

Workplaces should be decentralized, not reliant on a central hub

Mix manual jobs, desk jobs, craft jobs, selling, etc. within a community

There should always be a common piece of land (or a courtyard) within the work community which ties offices together

The work community is interlaced with the larger community they operate within

Workspace efficiency and community engagement is definitely not a new practice, however we always tend to think it is. If we can remember the lessons learned thirty-seven years ago, we may be in a better place to make a better workplace today.

Thinking in Patterns

Cognitive Hazard by Arenamontanus

I've finally started to look at some recommended reading that has been on my wish list for going on two years now. Two of the books, Thinking in Systems and A Pattern Language, have particularly resonated with me since they spoke directly to the practice of software engineering without mentioning it once.

Donella Meadows has left behind quite a legacy, and has great observations on how people work within overarching systems. Systems are everywhere and are often composed of yet other systems - just as it is with how people manage their workload every day. In particular, Donella notes the traps that systems can cause which cause things to go completely awry. Let's see if we can identify any of these traps within the context of enterprise software development:

• Policy Resistance (think of "The War on Drugs," where two sides are trying to leverage the same system)
• Tragedy of the Commons (exhausting a shared resource)
• Drift to Low Performance (goals are eroded because negative feedback has more resonance than positive feedback)
• Escalation (one side is attempting to out-produce the other, without a balance in between the two sides)
• Competitive Exclusion (success to the successful)
• Shifting the Burden to the Intervenor (an addiction has removed a system's ability to shoulder its own burdens)
• Rule Beating (finding loopholes)
• Seeking the Wrong Goal

Any of those sound familiar in your current software engineering practice? No matter if this is exhibited between the business and the engineers, or PM's and engineers, or between engineers - these are universal pratfalls.

There are ways to influence systems and avoid the traps we often fall into. These leverage points within the system can allow you to alter behavior and encourage positive results. A tricky point remains that some of the leverage points that are easiest to alter have the smallest impact, and some of the largest impact leverage points are very difficult to manipulate. If we look at an Agile software scrum, you might identify least impactful to most these leverage points as:

1. Numbers, Constants and Parameters. It often feels like you're changing things because you have the most control over these knobs and dials... but all too often reactions are delayed and are cushioned by buffers within the system. Sure, you can change your sprint velocity or begin estimating bugs, but those are just different views on the same result.
2. Buffers, or the sizes of stabilizing stocks that act as reservoirs of results. A buffer may delay or even out the consequence of a change within the system. Changing buffers would be like changing from a two to a four week development sprint in Agile - you may give yourself more time to recover, but more than likely you're just delaying an inevitable fail.
3. Failing that, you might try to alter the real, physical parts of the system and how they interact. This can happen, but they are often difficult to change and result in a game of whack-a-mole. This is more fondly called "re-arranging the chairs on the Titanic," and often is exhibited by swapping out team members but keeping the system the same.
4. The next leverage point might be to try and change how quickly you respond to changes by reducing delays, which in turn alters how quickly the system changes. However, Donella does demonstrate that shorter reaction time can very easily result in greater volatility, and things can become so volatile that they crash. This is what Agile is meant to guard against by locking down a sprint and ensuring priorities aren't changed on a day-by-day basis.
5. In order to get a grasp on things one may also overlook the balancing feedback loops - or safety measures - that safeguard the system in times of emergency. The excuse is generally that "the worst is unlikely to happen," however this drastically reduces one's survival range. Adaptability is important, and if you take away the ability to adapt you can crash even harder.
6. Monitoring for reinforcing feedback loops is something that becomes crucially important. This tasks requires one to watch for runaway chain reactions, which can cause a meltdown if not kept in check. Here bad decisions and bad reactions begat even more bad decisions and bad reactions, causing a runaway system. Look for balance instead of infinite feedback loops; if you can keep pushing your tasks to the next sprint, you're only encouraging a runaway backlog of tasks.
7. Information flows can save a system. If information is in your face and always available, it influences even small decisions. Look at the Nest thermostat or smoke detector - here are devices whose primary purpose is to give you a nonstop flow of info wherever you are. The more info you have (such as how many hours heat was pumped into your house), the more you make small alterations to find balance. This is another part of the Agile process in the form of burndowns/burnups/velocity graphs. This info is meant to be viewed and reflected upon often.
8. Rules (incentives, punishments, constraints) often have to take place to enforce all the above points. In order to kill feedback loops, ensure emergency systems are maintained and information is shared some rules of the game have to be put into place.
9. Self-organization, which is an odd juxtaposition of the above rule about rules, is something that Donella prizes most about not only the human condition but systems in general. Usually if you let the component pieces of a system find their role, they will find a way to work with other components in harmony. This is the proof against micro-management; the more you manage, the more you can threaten a system's success. Let developers go free within the confines of the sprint, and don't hover over them (aside from a daily standup).
10. Find the right goals to change a system. If you focus on GDP, you will focus on gross domestic product at the exclusion of other things. Picking the right goal is tougher than it sounds - you need to know what you want first. However if you can clearly identify and communicate a measurable goal, you can have a huge amount of control over the system. Define what the business actually wants to see - and involve them in the decision making process.
11. Change your mindset. This is effectively what EVERY project management methodology attempts to do - make you think about the same problem in a different way. If it gives you a renewed perspective, this can be helpful. However...
12. ...ultimately you should transcend paradigms and realize no paradigms are true. This is what supposed "anti-patterns" are meant to exhibit, and it can be helpful to realize that Agile, just like Waterfall, will ultimately come and go. Just ship early, and ship often.

Just as we have "Gang of Four" or "Enterprise Integration" patterns, the above are system patterns that can help us decompose and deal with a system. Look for the common traps that always happen - and then evaluate your leverage points to counteract them.

Re-Examining Development Platforms

I've been granted an opportunity for perspective lately. I've done my best to steer away from .NET development up until recently - not because I had any particular gripes, but the technology platform just never seemed to be a great fit. C# as a language is pretty great... delegates, tail closure and nullable references are a welcome respite. VisualStudio isn't bad. If you judge IDEs by how many times they make you swear in a given workday, I'd say my four-letter word tally is comparable to that of me using Eclipse. In fact, once my development environment was up and running I thought I might grow to enjoy .NET development.

And then I started to use the foundation classes. First off... I'll acknowledge that no platform has ever been able to get dates and times "right." This is ever more apparent with .NET... who for some inexplicable reason have no real sense of this "epoch" thing that EVERY OTHER PLATFORM USES. Milliseconds since year zero are expressed as... integers? On top of that, time span arithmetic is only accurate when math is done using "ticks," which themselves are not really accurate to 1/10000 of a millisecond. I didn't go for the caesium clock upgrade in my current laptop.

Java developers often complain that anonymous classes are inelegant or verbose within Java. C# has abandoned anonymous classes in lieu of their delegate-based event handling system. However, C# developers exposed to anonymous inner classes actually seem to like them. A common gripe actually turns out to be a nifty feature when you're talking about event handling. The C#/.NET event handling mechanism isn't that fantastic... it's largely just a loose convention for using delegates. No extras or nice GoF listener patterns provided like a PropertyChangeListener.

Zooming out from design patterns and looking at .NET from an enterprise integration pattern perspective, the .NET platform is definitely at a major disadvantage when compared to JVM-based platforms. I've already covered the state of .NET integration frameworks but to recap: it's still nearly five years behind. My time with EasyNetQ has been great, but I still find myself wishing I could use Apache Camel to construct bigger things using common EIP components.

Despite the current rant, I've been fairly complacent with my new development platform. What really stirred things back up for me was when I cracked open the JMeter-Rabbit-AMQP plugin so that I could do RPC-based load testing of EasyNetQ services. Being a JMeter extension, JMeter-Rabbit-AMQP was a Java app that required me to fire up NetBeans once again and do some Java hacking. Once I did... damn. Until I made that sudden switch back I didn't see the huge gap that existed between .NET and Java development. While C# has some advantage over the Java language, the JVM platform is still leaps ahead.

Once you begin talking about instrumentation the gap grows even wider. I have grown accustomed to the fantastic introspection and profiling offered by Java Management Extensions and VisualVM; by contrast Microsoft's laughable implementation of Performance Counters has caused me more problems than it has solved. If it works (and it often doesn't due to permissions issues or outright registry corruption) there is no instrumentation that allows for live modification of managed objects or details on garbage collection. The actual .NET API to create and maintain performance counters is actually not bad, but the Performance Counter UI is so clunky and ill-conceived that it is often difficult to make use of it.

In the end... it doesn't matter. You do the best you can with the tenured development stack because ultimately it's not about the underlying technology - it's about the squishy, business-logicy brain inside of it. Keeping that squishy brain... err... braining is the most important thing.

What I've Seen with Your Eyes

Eskil posted the video for his GDC presentations and they're abso-freakin-lutely amazing.

The gameplay of Love was interesting, but the video displaying the tools Eskil created are completely mind blowing. It's the stuff that actually gives you hope for the world again. The GDC tool video shows off several tools Eskil has released: Loq Ariou which allows you to create assets & models with the same ease as a pencil & scratch paper, Co On which provides scene mapping that's startlingly similar to how you might visualize things in your own mind, and Verse, a data transfer & protocol standard that allows such data to be shared instantaneously between applications.

Obviously Eskil had to create an intelligent set of tools to properly build Love within a decade, but I had no idea he had constructed such a cadre of tools that could be re-used by other developers. Not only does he speed content generation up and provide better interfaces - he goes one step further by breaking down human factor boundaries that plague every other asset generation tool to date. Just watch the video - especially the portion demonstrating shaders in Co On - and you'll see why I'm going completely nuts over these releases.

Eskil is giving back a huge amount to the community at large with these tools, and is likely opening the doors for many, many others to creatively express themselves in ways that were once prohibitively difficult. Love isn't just creating a fanbase... it's creating a legacy.

Trapped In The (Triganometry) Matrix

When everyone in intermediate school asked the math teacher "why the hell should we spend four months on matrix math," he should have looked directly at me and said "because, chuck, you'll need to know how to rotate sprites in both 2D and 3D space, otherwise your code in the future will suck." I would have listened then.

I was finally able to get Qt to properly apply a bitmask to my QWidget, so I can now have a window open that tilts at a 45 degree angle. I've got to start thinking Π/2... ODE works in radians, Qt in degrees.

There's a number of differences between Qt and ODE that can be a real pain in the butt. ODE's screen coordinate origin is at the bottom-left of the screen (or world)... which I guess kinda makes sense for physics coordinates that want to model themselves after the real world. But Qt uses what everyone else in computer-land uses: an origin at the top-left of the screen. It's whack. That means counter-clockwise rotations in ODE land are clockwise rotations in Qt land. Increasing your values of y send you up in ODE, down in Qt. Normals face one way with ODE, the other way with Qt. Blech.

A big tip o' the hat to Stefan Waner and Steven R. Costenoble for their Finite Mathematics page. Also big thanks to Columbia College of Missouri for reminding me how to reduce a matrix in my head. Without them I wouldn't have been able to remember how to invert the rotation matrix and easily convert ODE rotation matrixes to Qt rotation matrixes.