A Drift Into Failure

I'm still working towards catching up on my Christmas ready. I already wrote my missives on Thinking in Systems and A Pattern Language; next up is the DevOps favorite Drift into Failure.

The basic premise of Drift is that failures, even massive ones, don't (usually) happen because of a vast conspiracy or from the deeds of evil people. Massive failures occur from behavior that is considered completely normal, even accepted, as part of a daily routine. These routines give our perspectives tunnel vision and often don't allow us to see the underlying issue. Production goals, scarce resources and pressure on performance causes drift in these routines that slowly erode safe practices.

Fatal aircraft crashes and space shuttle disasters are often quoted in the book, however every operations or software engineer in IT has seen this play out a gazillion times before. The site goes down on a regular basis... and no one knows quite why. After digging and pushing new code and re-pushing bug fixes for many sleepless nights, one often finds out that the outage was due to a routine maintenance task gone awry. Maybe a query optimization cache was manually flushed within the production RDBMS, causing the entire cluster to freak out and create a bad query plan. It seemed perfectly sane at the time and even if every single person knew this was going to happen the day before, it likely wouldn't have been caught.

Drift points out how remediation and "root cause" reporting is often fruitless. The concept of high-reliability organizations was pushed in the 1980's as an entire school of thought, focused on errors and incidents as the basic units of failure. Dekker demonstrates that "the past is no good basis for sustaining a belief in future safety," and such a focus on root-cause analysis often does not prevent future incidents. The traditional "Swiss Cheese Model" for determining cause has attempted to see where all of the holes within established safety procedures line up, so as to create a long gap through which problems can drive themselves through. This type of reductionist thinking where atomic failures create linear consequences has turned out not to be predictive after all - instead we need to look at things through the lens of probability.

One of the best practices that anyone, including those supporting enterprise software, can encourage to avoid failure is to be skeptical during the quiet times and always invite in a wide range of viewpoints and opinions. Overconfidence can be your downfall, and dissent is always a healthy way to get new perspective. Dekker quotes Managing the Unexpected to point out that "success... breeds overconfidence in the adequacy of current practices and reduces the acceptance of opposing points of view." Those that were not technically qualified to make decisions often were the ones that made them, or outside pressures (event subtle ones) caused trade-offs in accepted practices. Redundancies that were supposed to make things highly available often make systems more complicated and, in turn, actually make them more likely to fail.

The best way to avoid a drift into failure is to invite outside opinion, even bring in disparate practice groups. Take minority opinions seriously. Don't distill everything to a slideshow. Be wary of adding redundancies and failsafes - often the most simple solution will be the most reliable. The recent re-invigoration in microservices is a great example of this - by simplifying the pieces of a complex system, we can allow each component to work in isolation and ignore the remainder of the system. This allows the system to grow, adapt and evolve without support systems usually provided for monolithic software stacks.

Drifting into failure occurs when an organization can't adapt to an increasingly complex environment. Never settle, always embrace diversity and keep exploring new ways to evolve. A great quote from Dekker is "if you stop exploring, however, and just keep exploiting [by only taking advantage of what you already know], you might miss out on something much better, and your system may become brittle, or less adaptive."

Someone to Give Me the Time

It's been really interesting to see the responses from Blitz, Fly Object Space and GigaSpaces concerning state management as well as Newton and Rio concerning service discovery. I'm definitely learning as I go, but the good thing is that it seems like there are many in the community eager to help.

Now I'm working on another issue with enterprise service development - scheduled services. There are some services out there who may want a have an event fire in 1000 milliseconds, or five minutes, or an hour, or somewhere in between. This would appear to be an easy thing to solve at first blush - until you consider volume, quality of service and scalability. It's a steep drop into complexity at that point.

Here's the thing: you could easily just do a scheduled executor in J2SE, but once your VM dies then your pending events die too. You could submit a scheduled job to something like clustered Quartz instances, but then you must have a reliable back-end database to write to (no native replication). You could use something like Moab Cluster Suite, but it seems to live outside the muuuuuuuuch more simple realm of event scheduling.

So let's think outside the box and use some replicated object store that isn't necessarily meant for scheduling. How about we slap a time to live (TTL) on a JMS message, throw it on a queue and wait for it to hit the dead letter queue? That might work at times, but TTLs are really intended for quality of service and not for scheduled events. Unless you have a consumer attached to the former queue constantly polling for messages you're not guaranteed to land in the latter dead letter queue.

How about using Camel's Delayer Enterprise Integration Pattern? Nope - that's just a Thread.sleep on the local VM. Doesn't do you much good once the VM dies. How about a delayed message using JBoss Messaging? I've heard tell that it exists, but I can't find much reference to it in the documentation.

This isn't a new problem - there's even JSR 236 that is intended to address this problem. But it's been hanging around since 2004 with very little activity of note, so I doubt it's going to have much hope of working by Monday.

Until JSR 236 is addressed I'll likely have to just find a way to deal with this on my own. Maybe create a JobStore for Quartz that's backed by a JMS topic? Or just suck it up and build a clustered Quartz instance with a fault-tolerant database?

Gah. Sticky wicket.

Empty Spaces

Yup, I'm still trying to work my way around Jini. This time it's JavaSpaces.

Apache River hasn't gone much farther from the last time I looked at it, but I liked the bare-bones reference implementation aspect. GigaSpaces seems a bit thick for my tastes and seems to be tightly coupled with their application server. I thought Blitz JavaSpaces might be a better fit, especially if I could use their fault tolerant edition.

I was able to get Blitz up and running then configured it to do unicast discovery to a pre-existing Jini registrar without a problem. I was having problems getting my client to connect in its security context, so I decided to dig a little deeper. As I did I also kept an eye towards fault-tolerance, but found that branch seemingly suspended. I later found a post from the author indicating he didn't really see a good motivation for moving forward with his fault-tolerance work:

In my spare moments I've been doing a re-implementation but the fact of the matter is that it's not a trivial problem to solve (though I believe I do have a solution). And here's the rub, this work doesn't pay the bills which means that it's going to take a long time to implement because I have to do a day's work first. For those who don't know, most of Blitz has been written during time between periods of employment - not over weekends and evenings as you might expect.
This presents me with a problem - users seem to want this feature but I'm struggling to see doing this as a good thing. Here's some of my reasons:

1. I'd be building a significant feature which will, judging by demand, make a lot of money for those who use it but zilch for me.
   
2. Not only do I earn nothing from this venture but I have to earn a significant amount of cash just to allow me time to develop the feature. Basically, I'd be financing everybody else's money making ventures.
   
3. One of GigaSpaces key value adds is the clustering/replication feature - they are fully commercial and need to earn a crust plus they're one of only a few credible companies that can provide corporate grade support for JavaSpaces. Were I to do this work for Blitz I'd maybe be damaging the market I've been helping to create.
   

Right now, I feel like the price of this piece of work is too high for me personally and for others in the commercial JINI world (and like it or not they are an important element in any future success for JINI). I can see why Blitz users might want this feature - they can avoid paying Gigaspaces a license for starters.

So... it seems like the development of an enterprise ready Blitz isn't in the cards. Casually strolling through Wikipedia's definition of a Tuple space brought up Fly Object Space, a tuple space that is not a JavaSpace implementation. While it doesn't fit into the Jini realm I know and love, it is a more minimalistic implementation of an object space that fits my desire of something smaller and to-the-point. It doesn't appear to support replication or fail-over on the non-commercial level, but I'm checking to see if there are plans to support it on a commercial level.

It's tough. I need an object space that has a minimalistic implementation, has a small footprint and can at least run active/passive for fault tolerance. Maybe I might have to dust off my old Terracotta instance and try out SemiSpace.

EDIT: Be sure and see Nati Shalom's comments following this post.

Massive Server Farms Can Equal Massive Failure

A quick trip over to the Amazon Web Service Status Page reveals that massive server resources don't exactly equate to massive uptime numbers. The S3 storage cloud has been down pretty much all day due to "an issue with the communication between several Amazon SQS components." This has affected both the EU and the US, causing some big headaches.

I'm not all that disturbed. I'm just happy that someone makes even bigger impacting gaffes than I do.

Trying to Work (Together)

I'm currently bashing my head against a wall. Both physically and metaphysically.

Nothing is cooler than Jini. However, unfortunately nothing is more obscure than Jini. It is something that exists etherally as a specification, not necessarily (although occasionally) as an implementation. It's the how, not the what. It's an adverb. Or something. My brain is lightly fried.

See, you can start with Jini's reference implementation via the starter kit, which exists as a concrete example of the specification. However, the last release was at the end of 2005. This is probably due to this implementation being picked up by the Apache Incubator and turned into Apache River. Apache River does seem to be under active development but has yet to issue an official release and has distributed just one release candidate.

Alright... so the Sun/Apache standard implementation is in transition. Who else has Jini services ready? Rio is a java.net project that appears to be both flexible as well as standards-compliant, so it appears to be a contender. There even appears to be moderate development traffic. However the stable binaries don't seem to match the current on-line documentation, and I haven't been able to download the latest milestone release. It does appear to have Spring integration however - and may yet be a contender. But until the documentation syncs up with the latest stable release, it's a bit hard to follow.

Cheiron also has an implementation with Seven that is Jini-compliant, and appears to be receiving some good development traffic as well. I'm still researching how to go about implementation, and their docs currently match up with their releases. I'm trying to (still) read their documentation and see what I need to do to get up to speed. It appears that most people discussing Jini in forums use Seven Suite as their implementation of choice, although Rio has a strong following as well due to its ease of Spring integration and nice administrative tools.

But for me this means I'm writing a helluva lot of "Hello World" and "Echo" applications, reading until my eyes bleed and trying to figure out how to get this all to work under a local development environment. Jini has been around forever... maybe I'm just having a hard time catching up with the rest of the world.

I'm hoping Spring Integration makes this all a bit more straight-forward for complete knobs like myself. Please oh please tell me they're adding Jini into the next milestone. Having ubiquitous services over a self-healing network is just too good. I'd love to be able to scale just by plugging in a server and walking away.