A Drift Into Failure

I'm still working towards catching up on my Christmas ready. I already wrote my missives on Thinking in Systems and A Pattern Language; next up is the DevOps favorite Drift into Failure.

The basic premise of Drift is that failures, even massive ones, don't (usually) happen because of a vast conspiracy or from the deeds of evil people. Massive failures occur from behavior that is considered completely normal, even accepted, as part of a daily routine. These routines give our perspectives tunnel vision and often don't allow us to see the underlying issue. Production goals, scarce resources and pressure on performance causes drift in these routines that slowly erode safe practices.

Fatal aircraft crashes and space shuttle disasters are often quoted in the book, however every operations or software engineer in IT has seen this play out a gazillion times before. The site goes down on a regular basis... and no one knows quite why. After digging and pushing new code and re-pushing bug fixes for many sleepless nights, one often finds out that the outage was due to a routine maintenance task gone awry. Maybe a query optimization cache was manually flushed within the production RDBMS, causing the entire cluster to freak out and create a bad query plan. It seemed perfectly sane at the time and even if every single person knew this was going to happen the day before, it likely wouldn't have been caught.

Drift points out how remediation and "root cause" reporting is often fruitless. The concept of high-reliability organizations was pushed in the 1980's as an entire school of thought, focused on errors and incidents as the basic units of failure. Dekker demonstrates that "the past is no good basis for sustaining a belief in future safety," and such a focus on root-cause analysis often does not prevent future incidents. The traditional "Swiss Cheese Model" for determining cause has attempted to see where all of the holes within established safety procedures line up, so as to create a long gap through which problems can drive themselves through. This type of reductionist thinking where atomic failures create linear consequences has turned out not to be predictive after all - instead we need to look at things through the lens of probability.

One of the best practices that anyone, including those supporting enterprise software, can encourage to avoid failure is to be skeptical during the quiet times and always invite in a wide range of viewpoints and opinions. Overconfidence can be your downfall, and dissent is always a healthy way to get new perspective. Dekker quotes Managing the Unexpected to point out that "success... breeds overconfidence in the adequacy of current practices and reduces the acceptance of opposing points of view." Those that were not technically qualified to make decisions often were the ones that made them, or outside pressures (event subtle ones) caused trade-offs in accepted practices. Redundancies that were supposed to make things highly available often make systems more complicated and, in turn, actually make them more likely to fail.

The best way to avoid a drift into failure is to invite outside opinion, even bring in disparate practice groups. Take minority opinions seriously. Don't distill everything to a slideshow. Be wary of adding redundancies and failsafes - often the most simple solution will be the most reliable. The recent re-invigoration in microservices is a great example of this - by simplifying the pieces of a complex system, we can allow each component to work in isolation and ignore the remainder of the system. This allows the system to grow, adapt and evolve without support systems usually provided for monolithic software stacks.

Drifting into failure occurs when an organization can't adapt to an increasingly complex environment. Never settle, always embrace diversity and keep exploring new ways to evolve. A great quote from Dekker is "if you stop exploring, however, and just keep exploiting [by only taking advantage of what you already know], you might miss out on something much better, and your system may become brittle, or less adaptive."

A Systems Language

A Pattern Language is an interesting book to pick up, and that's not just a joke about the size of the volume. Its web site betrays how old the book actually is; it was published in 1977 based on research that had been ongoing for several years. It's scope is pretty large and covers everything from the layout of an office building to the composition of an entire town. Much attention is focused on how to build communities within these spaces, and a lot of research provides evidence on optimal ways of building and tearing down boundaries.

Of particular interest to me were chapters concerning self-governing workshops and offices. The book stresses that no one enjoys their work if they are a cog in a machine. Instead, "work is a form of living, with its own intrinsic rewards; any way of organizing work which is at odds with this idea, which treats work instrumentally, as a means only to other ends, is inhuman." This is a fairly strongly worded assertion that means that employees must feel empowered in order to construct meaningful product.

Just as Thinking in Patterns postulated that groups should autonomously self-organize in order to realize their greatest efficiency, A Pattern Language encourages the formation of self-governing workshops and offices of 5 to 20 workers. A chapter is dedicated to the federation of these workgroups to produce complex artifacts - such as several independent workshops working in concert to build much larger things.

A Pattern Language also encourages keeping service departments small (less than 12 members) and ensuring that they can work without having to fight red tape. This applies to many shared services departments in both government as well as public sector organizations; departments and public services don't work if they are too large as the human qualities vanish. One must fight the urge to make an "idiot-proof system," since this can cause the system to devolve to the point that only idiots will run it.

The book is largely about physical space of course, so it has many recommendations on how offices should be connected. The authors specifically studied what isolated groups within a company, and even what we might consider small physical distances amounted to big interruptions in communication. If two parts of an office are too far apart, people will not move between them as often as they need to. If they are a floor apart, they sometimes will not speak at all.

Ultimately A Pattern Language has a lot of common sense to offer up about how to build a work community, backed by a fair amount of research that bucked many trends in the 70's. It had points that should not be glossed over even now, including:

You spend 8 hours at work - there is no reason it should be any less of a community

Workplaces must not be too scattered, nor too agglomerated, but clustered in groups of 15

Workplaces should be decentralized, not reliant on a central hub

Mix manual jobs, desk jobs, craft jobs, selling, etc. within a community

There should always be a common piece of land (or a courtyard) within the work community which ties offices together

The work community is interlaced with the larger community they operate within

Workspace efficiency and community engagement is definitely not a new practice, however we always tend to think it is. If we can remember the lessons learned thirty-seven years ago, we may be in a better place to make a better workplace today.

Thinking in Patterns

Cognitive Hazard by Arenamontanus

I've finally started to look at some recommended reading that has been on my wish list for going on two years now. Two of the books, Thinking in Systems and A Pattern Language, have particularly resonated with me since they spoke directly to the practice of software engineering without mentioning it once.

Donella Meadows has left behind quite a legacy, and has great observations on how people work within overarching systems. Systems are everywhere and are often composed of yet other systems - just as it is with how people manage their workload every day. In particular, Donella notes the traps that systems can cause which cause things to go completely awry. Let's see if we can identify any of these traps within the context of enterprise software development:

• Policy Resistance (think of "The War on Drugs," where two sides are trying to leverage the same system)
• Tragedy of the Commons (exhausting a shared resource)
• Drift to Low Performance (goals are eroded because negative feedback has more resonance than positive feedback)
• Escalation (one side is attempting to out-produce the other, without a balance in between the two sides)
• Competitive Exclusion (success to the successful)
• Shifting the Burden to the Intervenor (an addiction has removed a system's ability to shoulder its own burdens)
• Rule Beating (finding loopholes)
• Seeking the Wrong Goal

Any of those sound familiar in your current software engineering practice? No matter if this is exhibited between the business and the engineers, or PM's and engineers, or between engineers - these are universal pratfalls.

There are ways to influence systems and avoid the traps we often fall into. These leverage points within the system can allow you to alter behavior and encourage positive results. A tricky point remains that some of the leverage points that are easiest to alter have the smallest impact, and some of the largest impact leverage points are very difficult to manipulate. If we look at an Agile software scrum, you might identify least impactful to most these leverage points as:

1. Numbers, Constants and Parameters. It often feels like you're changing things because you have the most control over these knobs and dials... but all too often reactions are delayed and are cushioned by buffers within the system. Sure, you can change your sprint velocity or begin estimating bugs, but those are just different views on the same result.
2. Buffers, or the sizes of stabilizing stocks that act as reservoirs of results. A buffer may delay or even out the consequence of a change within the system. Changing buffers would be like changing from a two to a four week development sprint in Agile - you may give yourself more time to recover, but more than likely you're just delaying an inevitable fail.
3. Failing that, you might try to alter the real, physical parts of the system and how they interact. This can happen, but they are often difficult to change and result in a game of whack-a-mole. This is more fondly called "re-arranging the chairs on the Titanic," and often is exhibited by swapping out team members but keeping the system the same.
4. The next leverage point might be to try and change how quickly you respond to changes by reducing delays, which in turn alters how quickly the system changes. However, Donella does demonstrate that shorter reaction time can very easily result in greater volatility, and things can become so volatile that they crash. This is what Agile is meant to guard against by locking down a sprint and ensuring priorities aren't changed on a day-by-day basis.
5. In order to get a grasp on things one may also overlook the balancing feedback loops - or safety measures - that safeguard the system in times of emergency. The excuse is generally that "the worst is unlikely to happen," however this drastically reduces one's survival range. Adaptability is important, and if you take away the ability to adapt you can crash even harder.
6. Monitoring for reinforcing feedback loops is something that becomes crucially important. This tasks requires one to watch for runaway chain reactions, which can cause a meltdown if not kept in check. Here bad decisions and bad reactions begat even more bad decisions and bad reactions, causing a runaway system. Look for balance instead of infinite feedback loops; if you can keep pushing your tasks to the next sprint, you're only encouraging a runaway backlog of tasks.
7. Information flows can save a system. If information is in your face and always available, it influences even small decisions. Look at the Nest thermostat or smoke detector - here are devices whose primary purpose is to give you a nonstop flow of info wherever you are. The more info you have (such as how many hours heat was pumped into your house), the more you make small alterations to find balance. This is another part of the Agile process in the form of burndowns/burnups/velocity graphs. This info is meant to be viewed and reflected upon often.
8. Rules (incentives, punishments, constraints) often have to take place to enforce all the above points. In order to kill feedback loops, ensure emergency systems are maintained and information is shared some rules of the game have to be put into place.
9. Self-organization, which is an odd juxtaposition of the above rule about rules, is something that Donella prizes most about not only the human condition but systems in general. Usually if you let the component pieces of a system find their role, they will find a way to work with other components in harmony. This is the proof against micro-management; the more you manage, the more you can threaten a system's success. Let developers go free within the confines of the sprint, and don't hover over them (aside from a daily standup).
10. Find the right goals to change a system. If you focus on GDP, you will focus on gross domestic product at the exclusion of other things. Picking the right goal is tougher than it sounds - you need to know what you want first. However if you can clearly identify and communicate a measurable goal, you can have a huge amount of control over the system. Define what the business actually wants to see - and involve them in the decision making process.
11. Change your mindset. This is effectively what EVERY project management methodology attempts to do - make you think about the same problem in a different way. If it gives you a renewed perspective, this can be helpful. However...
12. ...ultimately you should transcend paradigms and realize no paradigms are true. This is what supposed "anti-patterns" are meant to exhibit, and it can be helpful to realize that Agile, just like Waterfall, will ultimately come and go. Just ship early, and ship often.

Just as we have "Gang of Four" or "Enterprise Integration" patterns, the above are system patterns that can help us decompose and deal with a system. Look for the common traps that always happen - and then evaluate your leverage points to counteract them.

The State of .NET Integration Frameworks

I have been increasingly working with teams that are largely operating within the .NET platform, so I’ve needed to abandon my old favorites in the Scala / Java / Python world. An interesting thing that strikes me (and many others I’ve discussed the topic with) about the .NET ecosystem is how there are no good analogs for .NET frameworks to Java or Python frameworks. Yes, there are projects such as NHibernate, Log4Net or even Spring.NET - but these often are exercises in trying to shave a square peg down so it can fit into a round hole. Even SpringSource seems to be pulling back from .NET support, as its support has becoming increasingly shallow when compared to Java.

Gone are the days of relying on integration frameworks such as Apache Camel. Instead I needed to survey the .NET landscape for integration frameworks that would allow for the development of loosely-coupled service components that can be horizontally scaled and messages intelligently routed. While there were a few frameworks that helped with RPC-style message passing to private queues, not a whole lot implemented the whole EIP enchilada.

Following is a brief survey of what I found for .NET frameworks or libraries that can help build a distributed message-based architecture. I would love any feedback - so much of this is subjective, and I’m sure I’m still only scratching the surface. I tried to weigh the weaknesses and strengths of each framework to see which would best provide implementations of Enterprise Integration Patterns.

Windows Communication Foundation

Windows Communication Foundation (WCF) was released by Microsoft as a way to unify different ways the .NET framework communicated with other services. WCF abstracts SOAP and REST API calls, queued messaging, .NET remoting and other technologies under a single framework for service-oriented development.

Strengths

WCF is integrated natively within .NET 3.5 and beyond as a supported tenant of the Microsoft .NET framework. Since WCF is a first-party solution it enjoys the full support of Microsoft, receives regular updates and is in tune with the .NET release cycle.

WCF is a straight-forward integration framework that provides a familiar interface for remote connectivity using well-known messages. Its endpoint design is analogous to the messaging endpoints defined by Enterprise Integration Patterns (EIP), and it appears message construction follows EIP as well.

Weaknesses

While WCF provides a working implementation of EIP messaging endpoints and message construction, it does not necessarily provide concrete implementations of the message routing, messaging channels or message transformation. Instead WCF provides bindings to other routing and channel implementations such as MSMQ, datagram sockets, REST and SOAP.

NIntegrate

NIntegrate appears to have been discontinued and is no longer eligible for evaluation.

NServiceBus

NServiceBus is a nearly decade-old SOA framework built to a custom set of distributed service patterns. In its own words: "If you design your system according to the principles laid out [as part of its design patterns], NServiceBus will make your life a lot easier. On the other hand, if you do not follow these principles, NServiceBus will probably make them harder."

The principles of NServiceBus are largely aligned with those of asynchronous messaging, which is referred to as the "one-way messaging" pattern. It also offers several stand-alone helper processes for distributing tasks, managing timeouts, and providing service gateways.

Strengths

NServiceBus does not appear to be as complex as an Enterprise Service Bus, but is more robust than WCF. While WCF does not provide a queue-based message bus by default, NServiceBus does dictate that queued messages be used.

Both Publish/Subscribe as well as Request/Reply messaging is supported by NServiceBus, which is another strength over WCF’s focus on remote process invocation. In addition NServiceBus appears to provide transaction support and durable messaging for guaranteed message exchanges. Transactional and durable messages are persisted via RavenDB (an ACID-compliant document database).

Messages within NServiceBus are well-defined constructs that include headers for routing message exchanges. Marshalling is provided by NServiceBus to automatically convert objects into documents for submission to message queues.

[update 2013-08-16: Udi Dahan has brought to my attention that, at the time of this blog post, NServiceBus also supported sagas as a way for workflows to be defined as long-lived transactions. Unit testing libraries were also provided. Both appear to have later inspired similar implementations within MassTransit.]

Weaknesses

NServiceBus is strongly coupled to its underlying implementation. MSMQ is currently the only supported messaging channel, albeit with a good deal of custom bindings for transaction support, durable messaging, quality of service and publish/subscribe support. Support for other channels such as AMQP has been considered as part of NServiceBus’ roadmap, but has not yet been implemented.

Durable messaging is only provided via RavenDB, which is a freely available open source project. A RavenDB instance must be separately maintained and managed.

MassTransit

MassTransit is "a lean service bus implementation for .NET," not unlike NServiceBus. Instead of working on top of WCF, MassTransit has created an endpoint and message abstraction of its own. In doing so, MassTransit allows for a wider breadth of implementations for messaging channels and document marshalling.

MassTransit message handling and channels are defined using Dependency Injection and Inversion of Control, which allows for better protoyping and unit testing of components.

Strengths

One of MassTransit’s primary strengths is its flexible channel architecture. Implementations can leverage MSMQ, ActiveMQ, RabbitMQ and TIBCO for message channels and JSON, XML or other document types for marshalling. MassTransit also supports a wide variety of IoC containers (or no container at all, in theory).

Messages can be simple POCOs without annotation or implementation of another interface. Likewise message handlers (i.e. message consumers) can be simple classes that require a minimum of instrumentation.

Transactional and persistent messaging is supported. This also has multiple methods of implementation depending on preference.

MassTransit has an interesting implementation of .NET state machines for managing workflow and transactional messaging, called “sagas.” Sagas provide a way for workflows to be succinctly defined and efficiently process long-lived transactions. This could possibly be an efficient way to implement an Aggregator or Scatter-Gather Enterprise Integration Pattern.

So that developers can more easily create unit tests over consumers and handlers, MassTransit offers a testing library that more easily allows a local message bus to be constructed within the context of a given unit test. This greatly eases the development of unit tests by providing a supported framework for local unit testing.

Weaknesses

The MassTransit project itself can give the appearance of not being well maintained and entirely documented. The project has shifted from Google Code to GitHub while documentation is hosted within a stand-alone site. On occasion one will find dead links, outdated examples and incomplete documentation.

The AMQP routing implementation (the wire protocol also used by RabbitMQ) used by MassTransit is sub-par. The documentation indicates that routing keys are often lost from the message headers, and it appears only fanout exchanges are supported. MassTransit’s routing engine appears to compensate for this, however this obviates the efficiencies often gained when using AMQP bindings. In particular this hurts interoperability, since reply queues can be difficult to create based upon a fanout exchange. While MassTransit can use AMQP as a transport layer, it does not fully take advantage of AMQP’s capabilities.

Concurrent consumption is determined based on the entire message bus, not on each message consumer. This does not allow developers or operations to throttle the precise number of concurrently running threads permitted to respond to incoming messages. These sorts of quality of service levels are determined for the entire message bus.

Rhino Service Bus

The authors of Rhino Service Bus are the organization Hibernating Rhinos, who are also responsible for the document store RavenDB. There are many similarities between NServiceBus and Rhino Service Bus, most notably the emphasis on "one-way" asynchronous messaging. Also not unlike NServiceBus the primary queueing mechanism is MSMQ, however Hibernating Rhinos’ own Rhino Queues service is also supported for durable and transactional messages.

Rhino Service Bus was designed to be architecturally very much like NServiceBus and MassTransit, but more lightweight and with a slightly different design for private subscription storage.

Strengths

Rhino Service Bus implements both request/reply as well as publish/subscribe message exchange patterns using "one-way" asynchronous messaging. Message handling within Rhino Service Bus is very succinct; processing an inbound message requires only the implementation of an interface with a known message type.

While most integration frameworks in this survey appear to favor programmatic configuration by defining configuration variables in code, Rhino Service Bus defines settings with an external configuration file. In environments already accustomed to using external configuration files as part of their build process, this can reduce implementation cost.

Since the Rhino Service Bus’ persistent messaging schemes are directly integrated with Rhino Queues, it appears that installation, maintenance and patching will be more straight-forward than with NServiceBus.

The method for the selective consumption of messages is interesting - Rhino Service Bus uses generics to match endpoints with inbound messages. Only inbound messages of a declared type will be dispatched to a local endpoint.

Weaknesses

Rhino Servie Bus can only use either MSMQ or Rhino Queues as the message channel. More advanced brokers such as RabbitMQ are currently not supported, and there appears to be no proposed support on Hibernating Rhinos’ roadmap.

Once again, Rhino Service Bus configuration is XML based. While this is also cited as a strength, I know several engineers who instead consider this a liability. Personal preference on behalf of many developers seems to favor programmatic configuration over static configuration files.

Rhino Service Bus appears to have limited quality of service properties, likely due to an emphasis on transactions and stateful messaging via MSMQ. The service bus itself seems to have a central focus of maintaining stateful messaging, as opposed to routing stateless exchanges.

Spring.NET Integration

The Spring framework was one of the premier IoC frameworks for the Java Platform, and the SpringSource team has released a similar IoC framework for the .NET platform as well. Current incarnations of Spring (including those on the .NET platform) span beyond just depdency injection; it also includes data abstraction layers, messaging frameworks and service integration components.

A large initiative released by the SpringSource team in 2008 was Spring Integration, an integration framework that was meant to simplify the development of service oriented application development. Just as the initial Spring container was meant to be a concrete implementation of the "Gang of Four" (GoF) Programming Design Patterns, Spring Integration was meant to be a strict implementation of Gregor Hohpe’s Enterprise Integration Patterns.

Strengths

Adherence to the Enterprise Integration Patterns is one of Spring Integration’s primary strengths. An integration framework with a focus on pattern-driven development helps designers and engineers develop applications with well-defined solutions to problems. Spring Integration relies heavily on the Spring IoC framework, which also enforces GoF Design Patterns. By matching both of these well-proven enterprise design patterns developer productivity can be increased and code can be more easily tested.

Spring Integration offers an abstraction of message routing and message channels that allows the integration of best-of-breed implementations to more easily be performed. Broker implementations can range from AMQP, JMS, flat files, peer-to-peer, etc.

Weaknesses

Currently Spring Integration for .NET is in “incubator” stage within SpringSource. This can be considered to an open beta release of a project, where development is heavily underway and the given framework is subject to change at any time. Judging by the activity stream on the project’s issue tracker, it appears that there is currently only one engineer assigned to the project who last committed changes in March of 2010.

Since Spring Integration is closely tied to the Spring Framework, no alternate dependency injection containers are supported. While many integration frameworks surveyed rely on Castle Windsor, Spring Integration instead relies on the Spring IoC framework.

Spring.NET AMQP

The Spring framework was one of the premier IoC frameworks for the Java Platform, and the SpringSource team has released a similar IoC framework for the .NET platform as well. Current incarnations of Spring (including those on the .NET platform) span beyond just depdency injection; it also includes data abstraction layers, messaging frameworks and service integration components.

Spring.NET AMQP is not an integration framework, but instead provides "templates" as a high-level abstraction for messaging. While this doesn’t provide message routing directly, Spring.NET AMQP allows developers to establish routing keys and bindings that permit an AMQP broker to properly route messages.

The scope of Spring.NET’s abilities is roughly analogous to that of Microsoft’s WCF, with the addition of dependency injection and inversion of control containers.

Strengths

Within the Spring.NET AMQP project, adherence to the AMQP feature set is the primary focus of SpringSource. This allows engineers to fully leverage brokers such as RabbitMQ and perform topic exchanges, content-based routing and multicasting.

Weaknesses

Spring.NET AMQP is a lightweight templating library which does not provide any adherence to Enterprise Integration Patterns. Routing and message transformation is not supplied by the framework, however it can more easily be implemented by the developer.

EasyNetQ

When users encounter difficulties when integrating .NET integration frameworks, a common alternative cited is EasyNetQ. While not necessarily an integration framework, EasyNetQ allows for messages to be more easily routed via RabbitMQ, not unlike the Spring AMQP project.

The EasyNetQ was inspired by MassTransit and created for use by the airline travel company 15below. The primary goals of the project are maintaining a very simple API and requiring minimal (if not zero) configuration. Code required for requests, replies and subscriptions are fairly minimal.

Strengths

EasyNetQ was created from the ground-up to take advantage of the AMQP specification’s message routing capabilities, and attempts to fully leverage the routing capabilities inherent to AMQP brokers. By adhering to the AMQP standard and expected behavior, interoperability between .NET and other platforms becomes an easier task. To accomplish the goal of zero or little configuration required to start using EasyNetQ, the framework accomplishes most setup tasks by convention. This allows for safe defaults to be easily chosen and reduced the amount of code that needs to be written.

Just as MassTransit implements "sagas" for long-running business processes, EasyNetQ provides a saga framework for workflows to be succinctly defined and efficiently process long-lived transactions. In a very similar way the EasyNetQ saga support could be an efficient way to implement an Aggregator or Scatter-Gather Enterprise Integration Pattern.

Connection management to the message broker has been implemented in a fail-safe way within EasyNetQ. Broker connections are performed using a "lazy connection" approach which by default assumes the broker will not be available at all times. If a broker is not available, messages are queued locally and the broker is polled until it comes online. Once the message broker is available, messages resume transmission. Bear in mind publishers of message are not aware of this outage; they continue without halting. This works in contrast to most other integration frameworks which throw exceptions or halt when the broker goes offline.

Weaknesses

Many integration frameworks under review directly integrate with another dependency injection framework such as Castle Windsor, however EasyNetQ does not have a direct integration with any such dependency injection framework. This has the benefit of requiring less dependencies, however it also makes the management of endpoint objects a bit more convoluted. This is not to say that a dependency injection framework cannot be used - indeed a IoC container can be very easily used alongside EasyNetQ. Other integration frameworks, however, effectively use dependency injection frameworks to auto-discover endpoints by type and greatly simplify the registration of message consumers. Such auto-discovery is not immediately available from EasyNetQ libraries.

Limiting the number of concurrent consumers is currently not something readily supported by EasyNetQ. While most integration frameworks allow you to rate limit consumption based on the number of concurrent threads, EasyNetQ does not currently offer this as a configurable property. Instead EasyNetQ provides the scaffolding for asynchronous responses backed by a blocking collection. Worker threads are added to the blocking collection, which them accept each request as they become available.

Time to Live settings are not easily accessible within the EasyNetQ framework, however facilities do exist to set message expiration settings directly within AMQP. Since this facility is not readily available, it makes quality of service a bit more difficult to establish for messages.

EasyNetQ does not offer a testing framework to assist with the construction of unit tests. Instead it is up to the developer to construct a mock message bus with which to test message production and consumption. This may not necessarily make the development of unit tests more difficult however, since endpoints are much more simply defined than with other frameworks.

Creating A New, Internal Framework

Creating a new integration framework is often regarded as a universally poor idea among developers. The YAGNI development assistant was created explicitly for monitoring an engineer’s behavior within an IDE and preventing the independent development of yet another framework for features "that aren’t necessary at the moment, but might be in the future."

If proper alternatives do not seem to exist as integration frameworks, it may be necessary to evaluate creating a new, first-party integration framework. If the sponsoring organization is committed to contributing to the open-source community, leveraging the community at large to develop an inter-operable solution may provide a unique framework that fills a need within the .NET ecosystem.

Designing Head Patterns

When I saw Jason McDonald's design patterns quick reference guide I marked out. I forwarded the link to everyone I knew who heard of the Gang of Four and printed out a nice, one-page color version to keep. I almost freakin' framed it. I liked how succinct it is... it gives you the class diagrams so your brain is instantly sparked, and just enough description that you still have to think analytically about the pattern.

A few people asked me about the original Design Patterns book and a few others mentioned how much they liked Head First Design Patterns. I have to sadly admit - I originally wrote off the Head First series as just another "X for Dummies" clone, but now that I read the RMI section of their Java book and read the first 100 pages of their Design Patterns book I have to admit I judged... sigh... a book by its cover.

The RMI section was actually fairly straight forward and illustrative and even included an informative chapter that included Jini. The Design Patterns book has done a good job of explaining the Decorator pattern, something that I can't do with a pencil and paper in front of someone. Both books have been good references for me to pass along to other developers.

If only there was a pattern to remove outdated cliches two paragraphs ago.

DeckerEgo's Razor

I changed the title of this blog from "Tales of an Indy Game Developer" to "Tales of an Indie Developer." Right now my game development has stagnated and may go to zero. But I'm still hacking away on a number of open source projects... as well as hacking the various other constructs I have to live with in my day.

One of the things I'm supposed to do in my paid gigs is architect the infrastructure of entire enterprises. If the title sounds nebulous and nonsensical, it is. Basically I try to coordinate data center sprawl or hack away bits when it gets out of hand.

Through my days I've moved progressively from very myopic fields to extremely broad ones. I started out in LISP working on Chez Scheme, moving to C++ & Motorola 68k assembly, moving to 3D rendering pipelines, moving to enterprise applications, moving to Web applications, moving to corporate networking, moving to corporate infrastructure, moving to enterprise Web application development & infrastructure, now moving to enterprise infrastructure. The uncanny thing is that all of these different genres follows the same Tao of Programming, no matter if it's even really programming anymore.

I'm working on ConsultComm 4 now and it's ridiculously slow-going. I have ripped out every bit of old code and have completely redone things (again), this time trying to follow a more... organic... layout. I'm trying to avoid corner cases and develop things as main-stream to Java's intended architecture as possible. This includes more modern event handling (more akin to Servlet chains or AWT events), better thread management and hopefully a better UI with less "surprises" and non-intuitive user interface choices. This has meant a crapton of prototyping, a lot of refactoring and a lot of brute force hacking. But I'm hoping it will be worthwhile in the end; the goal is to make ConsultComm 4 the last major version.

I've also been spending a lot more time in the debugger. The new codebase is loaded with assert statements meant to catch errors as they happen and plan for unexpected consequences. By doing some deep introspection and catching mental assumptions I make while I code things are error'ing out a ton more, but I'm catching more bugs on the front-end. Also I'm able to catch a lot of one-off circumstances and fail over to more appropriate states.

The same thing applies if you're trying to build a nationwide enterprise-ready datacenter tho. Prototype out the wazoo. Go as mainline as possible - don't try to outsmart existing architecture standards. No surprises. Plain for failure and develop for the exceptions. Document your assertions and watch for when they fail. Introspect and watch all the traffic routing back and forth... catch errors before they become problems.

Ultimately both software and enterprise application infrastructure need to pass the same meta-tests in order to become a lasting solution. You should be able to slice away any piece of the structure and have it remain consistent with the layout of the whole. You wouldn't cut a loaf of wheat bread and somehow end up with a single slice of rye in the middle. However, I'd wager if you went into a typical corporate data center and picked out a random server... heck, even an entire rack... you'd likely find out that it is managed or designed differently than the remainder of the servers in the room. Likewise take an object or source file from a typical software application and you'll find code patterns that appear completely different than the rest of the codebase.

All servers should be managed the same. All racks of servers should be organized the same. All types of servers (DBMS, SAN, storage, application server, development servers, load balancers, firewalls) should be updated, backed up and access controlled in the same way. All objects in an application should follow the same pattern. Exceptions should be handled the same way across all methods. Events, error handling and user prompts should all look and feel the exact same. Refactor like mad if you have to... but if you re-design your pattern, you should re-implement your codebase/servers/data center.

The Tao of Programming said it best:

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity.

A program should follow the `Law of Least Astonishment'. What is this law? It is simply that the program should always respond to the user in the way that astonishes him least.

A program, no matter how complex, should act as a single unit. The program should be directed by the logic within rather than by outward appearances.

If the program fails in these requirements, it will be in a state of disorder and confusion. The only way to correct this is to rewrite the program.