One of the things I'm supposed to do in my paid gigs is architect the infrastructure of entire enterprises. If the title sounds nebulous and nonsensical, it is. Basically I try to coordinate data center sprawl or hack away bits when it gets out of hand.
Through my days I've moved progressively from very myopic fields to extremely broad ones. I started out in LISP working on Chez Scheme, moving to C++ & Motorola 68k assembly, moving to 3D rendering pipelines, moving to enterprise applications, moving to Web applications, moving to corporate networking, moving to corporate infrastructure, moving to enterprise Web application development & infrastructure, now moving to enterprise infrastructure. The uncanny thing is that all of these different genres follows the same Tao of Programming, no matter if it's even really programming anymore.
I'm working on ConsultComm 4 now and it's ridiculously slow-going. I have ripped out every bit of old code and have completely redone things (again), this time trying to follow a more... organic... layout. I'm trying to avoid corner cases and develop things as main-stream to Java's intended architecture as possible. This includes more modern event handling (more akin to Servlet chains or AWT events), better thread management and hopefully a better UI with less "surprises" and non-intuitive user interface choices. This has meant a crapton of prototyping, a lot of refactoring and a lot of brute force hacking. But I'm hoping it will be worthwhile in the end; the goal is to make ConsultComm 4 the last major version.
I've also been spending a lot more time in the debugger. The new codebase is loaded with
assertstatements meant to catch errors as they happen and plan for unexpected consequences. By doing some deep introspection and catching mental assumptions I make while I code things are error'ing out a ton more, but I'm catching more bugs on the front-end. Also I'm able to catch a lot of one-off circumstances and fail over to more appropriate states.
The same thing applies if you're trying to build a nationwide enterprise-ready datacenter tho. Prototype out the wazoo. Go as mainline as possible - don't try to outsmart existing architecture standards. No surprises. Plain for failure and develop for the exceptions. Document your assertions and watch for when they fail. Introspect and watch all the traffic routing back and forth... catch errors before they become problems.
Ultimately both software and enterprise application infrastructure need to pass the same meta-tests in order to become a lasting solution. You should be able to slice away any piece of the structure and have it remain consistent with the layout of the whole. You wouldn't cut a loaf of wheat bread and somehow end up with a single slice of rye in the middle. However, I'd wager if you went into a typical corporate data center and picked out a random server... heck, even an entire rack... you'd likely find out that it is managed or designed differently than the remainder of the servers in the room. Likewise take an object or source file from a typical software application and you'll find code patterns that appear completely different than the rest of the codebase.
All servers should be managed the same. All racks of servers should be organized the same. All types of servers (DBMS, SAN, storage, application server, development servers, load balancers, firewalls) should be updated, backed up and access controlled in the same way. All objects in an application should follow the same pattern. Exceptions should be handled the same way across all methods. Events, error handling and user prompts should all look and feel the exact same. Refactor like mad if you have to... but if you re-design your pattern, you should re-implement your codebase/servers/data center.
The Tao of Programming said it best:
A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity.
A program should follow the `Law of Least Astonishment'. What is this law? It is simply that the program should always respond to the user in the way that astonishes him least.
A program, no matter how complex, should act as a single unit. The program should be directed by the logic within rather than by outward appearances.
If the program fails in these requirements, it will be in a state of disorder and confusion. The only way to correct this is to rewrite the program.