Sunday, August 24, 2014

DNS - The Internet's Phone Book

My earlier post about filtering Internet content for kids bringing home their school iPads may have created more questions than answers for some parents. The big confusion seems to step from what a Domain Name System (DNS) server is, and how it helps filter out objectionable content.

Let's go waaaaay back in time, back to the birth of the initial global network called ARPANET. Back in the day - and even now - you could reach a remote computer by using its numeric address. To connect to a remote computer, your machine may connect to "192.168.129.34" and send along some pretty data. Those numeric addresses could be a pain to remember however - so shortcuts were created that mapped a human-recognizable name (like "BubbaComp") to the numeric address (like "192.168.129.34"). Solutions were eventually engineered that let people share these lists... that way everyone could have this helpful list of shortcuts. This convention kept evolving as users continued to join the global network, up to today. Now when you type in "amazon.com" your computer is smart enough to look up this shortcut name and find out the numeric address is 176.32.98.166. Your computer always talks to 176.32.98.166, however you talk to your browser using https://amazon.com.

This operates just like a phone book. No one remembers people's phone numbers anymore... or at least I don't. Instead you look up a person's name in your personal address book or the big dead-tree phone book on your front stoop, then communicate using the phone number in the book. Connecting to sites over the Internet works in the very same way.

What if you didn't want your kids visiting certain sites? You could employ the same trick as you might to stop them from calling certain people over the phone - edit the phone book. If your kids can't look up a person's name and find their phone number, they can't call the person. If you edit the Internet's phone book and remove objectionable sites, your kids can't visit the objectionable site on their device of choice. That's exactly what OpenDNS allows you to do - use a phone book that only has acceptable web sites within it.

What if a kid memorizes a phone number tho? Your plan falls apart a bit in that case. DNS filtering has the same limitation - if your kids memorize the IP address of a site (or share an underground DNS server), then they can go directly to the site and bypass your sanctioned "phone book."

If your kids go to a site that has a wide variety of content (like YouTube), you can't filter out specific types of content within the site. Just like calling a party line on the phone... if you allow access to the party line, you can't control anything past the initial dial.

Hopefully that helps explain why OpenDNS is only your first line of filtering. Lemme know in the comments if I can clarify further!

Saturday, August 23, 2014

Web Filtering at Home

[Updated to include an OpenDNS how-to]

Now that iPads are standard issue for a lot of schools, a few parents have asked me how they can block inappropriate material at home. While the schools themselves filter at the network level, as soon as the student comes home the network is wide open.

In all honesty, you can't filter out 100% of all objectionable content. It's hard to have software determine if a YouTube stream is showing questionable video. However, you can audit, track and block some obviously adult sites. The traditional options to perform web filtering include:
  • Software applications or parental controls on the device itself
  • Filtering devices on the router or wireless access point
  • External Internet services that block DNS requests

Software applications give you the most control on a per-device level and can block errant applications as well (like anti-virus software), however you have to install them on each and every device. They also have the benefit of blocking things no matter what network they are attached to. They usually require a medium-level effort to circumvent, and it is sometimes hard to get a report on what the actual activity has been or if any sites had to be blocked.

Filtering devices provide filtering for the entire network and do not rely on software to be installed on the device, which is nice. This solution is the hardest to circumvent, so long as you properly lock down your wireless access point. This solution cares less about applications however, and can’t really tell how appropriate actual content on a site is. It also only controls those devices on your network, and often doesn’t have fine-grained controls.

External Internet services filter your entire network, just like a filtering device would, however it is hosted out on the Internet rather than being something installed or managed inside your house. This option often doesn’t give you much (if any) per-device controls, however they often do a great job of letting you pick what and how many sites to filter out. These solutions often provide reporting as well, letting you see what was viewed by devices on the network. This solution also can’t tell you about the actual content on the site, but just the URL that was visited. This solution is the easiest to circumvent, although this can be mitigated by locking users out of the administrative settings of a device (e.g. not letting users change network settings on an iPad).

What I chose for the house was an external Internet service via OpenDNS. This was easy to set up since I just had to create an account and make a few minor tweaks to our wireless access point, and it gives me some nice reporting on what was blocked. For example, lately I saw a lot of adult sites being blocked and found an iOS application was loading them in the background.

OpenDNS has a Getting Started Guide on their site, but here's an abbreviated version of the steps for setting up OpenDNS on your home network:
  1. First, load up the settings console for your wireless router. Check your user guide for how to do this - usually it involves loading up a web page at http://192.168.1.1 and entering a username and password.
  2. Next, find the "Internet" or "WAN" settings page within your wireless router. This is in your router's user guide as well. It may look something like:
  3. Change the DNS Servers from the automatic settings to the values "208.67.222.222" and "208.67.220.220"
  4. Click on "Apply" or "Save" or whatever floats your router's boat.
  5. Create a new account at http://www.opendns.com/
  6. Part of your account creation process will be linking your local network to your OpenDNS account. Once your local network joins OpenDNS, it will begin monitoring what sites are requested.
  7. After you create your account, you will be taken to the OpenDNS dashboard. At that point you can decide how much filtering you want to apply to your network - from sites that are obviously adult-only to sites that are adult in theme (fashion magazines, for example).

I'll post a subsequent entry on what OpenDNS actually does in hopes of helping explain why this kind of filtering is useful and its limitations. While this might seem like rocket surgery at first, hopefully this helps you learn how to be a steward of your Internet connection... just like you have to monitor & maintain your sump pump before the basement floods.

Sunday, June 22, 2014

A Drift Into Failure

I'm still working towards catching up on my Christmas ready. I already wrote my missives on Thinking in Systems and A Pattern Language; next up is the DevOps favorite Drift into Failure.

The basic premise of Drift is that failures, even massive ones, don't (usually) happen because of a vast conspiracy or from the deeds of evil people. Massive failures occur from behavior that is considered completely normal, even accepted, as part of a daily routine. These routines give our perspectives tunnel vision and often don't allow us to see the underlying issue. Production goals, scarce resources and pressure on performance causes drift in these routines that slowly erode safe practices.

Fatal aircraft crashes and space shuttle disasters are often quoted in the book, however every operations or software engineer in IT has seen this play out a gazillion times before. The site goes down on a regular basis... and no one knows quite why. After digging and pushing new code and re-pushing bug fixes for many sleepless nights, one often finds out that the outage was due to a routine maintenance task gone awry. Maybe a query optimization cache was manually flushed within the production RDBMS, causing the entire cluster to freak out and create a bad query plan. It seemed perfectly sane at the time and even if every single person knew this was going to happen the day before, it likely wouldn't have been caught.

Drift points out how remediation and "root cause" reporting is often fruitless. The concept of high-reliability organizations was pushed in the 1980's as an entire school of thought, focused on errors and incidents as the basic units of failure. Dekker demonstrates that "the past is no good basis for sustaining a belief in future safety," and such a focus on root-cause analysis often does not prevent future incidents. The traditional "Swiss Cheese Model" for determining cause has attempted to see where all of the holes within established safety procedures line up, so as to create a long gap through which problems can drive themselves through. This type of reductionist thinking where atomic failures create linear consequences has turned out not to be predictive after all - instead we need to look at things through the lens of probability.

One of the best practices that anyone, including those supporting enterprise software, can encourage to avoid failure is to be skeptical during the quiet times and always invite in a wide range of viewpoints and opinions. Overconfidence can be your downfall, and dissent is always a healthy way to get new perspective. Dekker quotes Managing the Unexpected to point out that "success... breeds overconfidence in the adequacy of current practices and reduces the acceptance of opposing points of view." Those that were not technically qualified to make decisions often were the ones that made them, or outside pressures (event subtle ones) caused trade-offs in accepted practices. Redundancies that were supposed to make things highly available often make systems more complicated and, in turn, actually make them more likely to fail.

The best way to avoid a drift into failure is to invite outside opinion, even bring in disparate practice groups. Take minority opinions seriously. Don't distill everything to a slideshow. Be wary of adding redundancies and failsafes - often the most simple solution will be the most reliable. The recent re-invigoration in microservices is a great example of this - by simplifying the pieces of a complex system, we can allow each component to work in isolation and ignore the remainder of the system. This allows the system to grow, adapt and evolve without support systems usually provided for monolithic software stacks.

Drifting into failure occurs when an organization can't adapt to an increasingly complex environment. Never settle, always embrace diversity and keep exploring new ways to evolve. A great quote from Dekker is "if you stop exploring, however, and just keep exploiting [by only taking advantage of what you already know], you might miss out on something much better, and your system may become brittle, or less adaptive."

Sunday, March 23, 2014

An Expensive Failure of Judgement

So remember when I precariously perched a moderately encased rangefinder above my sump pump well? It was kinda wedged in between the cover and the well wall, and I thought there wasn't enough play in the line leading to the rangefinder as to let it drop in. Well... all my hackery finally caught up to me and a very expensive sensor ended up taking a swim. Current remained running through it the entire time so for several hours it swam in well water, slowly accreting minerals. No amount of drying out would save it.

I wasn't going to replace it with another expensive sensor... so I went the completely opposite direction and built an unbelievably primitive water detector. Here two plates of aluminum foil were hot-glued to construction paper and the bare end of my infamous telephone wire, then isolated in electrical tape. If water bridged the two aluminum plates, a connection would be made - at least enough of a connection to be considered a "high" signal.

The other end of the two wires were sent to the NPN transistor that was originally intended to work as a UART logic inverter. Now it was a simple logic gate; once the water closed the circuit the NPN shut off the current headed to a GPIO pin. If the pin was live, no water was detected. If the pin was dead, you had a problem.


The web front-end that I created for this whole rigamarole was updated to reflect this hack, and now just reports the binary status of the water detector. I'm not thrilled with the setup, but I also wasn't too keen on the idea of plopping any more money down on a solution.

So... lesson learned. Don't dangle water sensitive components over a well of water.

I do have a need for another security camera, so this whole setup may just be ditched in favor of another Motion rig. I really dig the I2C temperature and humidity breakout board however, and I'd like to keep using it. Maybe I'll save up my allowance and get a CC3000 WiFi board and pair it up with the temp/humidity board... that would be a pretty nifty & tiny package.

Tuesday, March 11, 2014

It's a Basement, not a Swimming Pool

Second up on my paranoia list is my basement slowly filling with water. My paranoia is founded in a rich history of failed sump pumps, broken water mains and power outages. I can mitigate some of my worries by installing a backup, non-electric Venturi aspirator and a die-cast primary sump pump - however anything mechanical can break. I believe in nothing anymore.

A Raspberry Pi can help satiate most of my neurosis, including this one. Using a Honeywell HumidIcon Digital Humidity/Temperature Sensor and a Maxbotix Ultrasonic Range Finder I can monitor basement humidity, temperature and sump well levels.

My first component to integrate was the range finder. The Maxbotix LV-EZ4 can operate in one of two modes - either providing an ASCII representation of the range using RS232 serial communication or using an analog voltage. I dorked around with two possible ways of using this - feeding an analog signal through an Adafruit Trinket and have the values translated into an I2C signal. However - I had a 5v Trinket - and even with constructing voltage dividers I couldn't quite coordinate the right voltages to negotiate with the Pi. I punted and used the serial port from the LV-EZ4, however the Pi uses UART and so I had to create a logic inverter using a recycled NPN transistor. Once I inverted the signals from the range finder, the Pi was able to read the inbound ASCII representation of the range.

After I had the range finder working, I used Sparkfun's Honeywell breakout board via I2C to communicate temperature and humidity to the Pi. Both the range finder and the breakout board fit nicely on a mini breadboard, sharing voltage and ground while splitting out I2C data, clock and RS232 data feeds. Once permissions were correctly set and kernel modules loaded, things appeared to be working nicely.

I wanted to save the range finder from water splashes, or at least slow its eventual decay. I re-used the case from the SD card I purchased for the Raspberry Pi, cutting out holes for the extrusions in the range finder board. Corners were then covered in electrical tape, and the seams were covered in hot glue. No, it's not pretty. No, it may not add to the LV-EZ4's lifespan. It was at least worth a shot however, and I've added a bit of crush/drop protection.

Everything is hooked into a Raspberry Pi Model A, just to save a few bucks. For an enclosure I ripped apart an old Netgear wireless access point, which easily housed the mini breadboard and the Pi. I decided to try things out but stumbled upon an unsettling fact... there are no power outlets near the sump pump well. Undeterred, I went looking for any long length of wire and found twenty feet of RJ11 telephone cable. It had four total wires - which would be more than enough to carry voltage, ground and signal wires. I sloppily spliced the wire, soldered it onto three jumpers, attached one side to the breadboard and another to the range finder. To my surprise - it actually worked. I was able to string the range finder all the way across the room, which also made ambient humidity readings more accurate.

In much the same way as I created the Bottle application for the garage door security monitor, I created a Bottle app to host REST APIs and display the well depth, temperature and humidity as well as allow Jabber (e.g. Google Talk) clients to request the status of the well and the climate. It all is working well so far, however I still need to tweak the Honeywell I2C code to make sure the component re-samples conditions at every request. Right now it is just fetching the currently stored values.

Right now the range finder is resting atop the sump pump well and is just waiting for the upcoming rains. My eventual goal is to create a home dashboard that aggregates all sensor data from around the house: sump pump well depth, basement temperature and humidity from the Basement Monitor APIs, ground-level temperature and humidity from a Nest thermostat, garage door state and camera feeds from the Garage Security APIs and maybe even power data from an attached APC UPS. The Bottle apps would then work to expose sensor data as REST APIs, and a more powerful Play application would serve the user interface, archive historical data, provide alerts and indicate trends.

Saturday, March 01, 2014

A Systems Language

A Pattern Language is an interesting book to pick up, and that's not just a joke about the size of the volume. Its web site betrays how old the book actually is; it was published in 1977 based on research that had been ongoing for several years. It's scope is pretty large and covers everything from the layout of an office building to the composition of an entire town. Much attention is focused on how to build communities within these spaces, and a lot of research provides evidence on optimal ways of building and tearing down boundaries.

Of particular interest to me were chapters concerning self-governing workshops and offices. The book stresses that no one enjoys their work if they are a cog in a machine. Instead, "work is a form of living, with its own intrinsic rewards; any way of organizing work which is at odds with this idea, which treats work instrumentally, as a means only to other ends, is inhuman." This is a fairly strongly worded assertion that means that employees must feel empowered in order to construct meaningful product.

Just as Thinking in Patterns postulated that groups should autonomously self-organize in order to realize their greatest efficiency, A Pattern Language encourages the formation of self-governing workshops and offices of 5 to 20 workers. A chapter is dedicated to the federation of these workgroups to produce complex artifacts - such as several independent workshops working in concert to build much larger things.

A Pattern Language also encourages keeping service departments small (less than 12 members) and ensuring that they can work without having to fight red tape. This applies to many shared services departments in both government as well as public sector organizations; departments and public services don't work if they are too large as the human qualities vanish. One must fight the urge to make an "idiot-proof system," since this can cause the system to devolve to the point that only idiots will run it.

The book is largely about physical space of course, so it has many recommendations on how offices should be connected. The authors specifically studied what isolated groups within a company, and even what we might consider small physical distances amounted to big interruptions in communication. If two parts of an office are too far apart, people will not move between them as often as they need to. If they are a floor apart, they sometimes will not speak at all.

Ultimately A Pattern Language has a lot of common sense to offer up about how to build a work community, backed by a fair amount of research that bucked many trends in the 70's. It had points that should not be glossed over even now, including:
  • You spend 8 hours at work - there is no reason it should be any less of a community
  • Workplaces must not be too scattered, nor too agglomerated, but clustered in groups of 15
  • Workplaces should be decentralized, not reliant on a central hub
  • Mix manual jobs, desk jobs, craft jobs, selling, etc. within a community
  • There should always be a common piece of land (or a courtyard) within the work community which ties offices together
  • The work community is interlaced with the larger community they operate within

    Workspace efficiency and community engagement is definitely not a new practice, however we always tend to think it is. If we can remember the lessons learned thirty-seven years ago, we may be in a better place to make a better workplace today.
  • Wednesday, February 12, 2014

    Thinking in Patterns

    Cognitive Hazard by Arenamontanus
    I've finally started to look at some recommended reading that has been on my wish list for going on two years now. Two of the books, Thinking in Systems and A Pattern Language, have particularly resonated with me since they spoke directly to the practice of software engineering without mentioning it once.

    Donella Meadows has left behind quite a legacy, and has great observations on how people work within overarching systems. Systems are everywhere and are often composed of yet other systems - just as it is with how people manage their workload every day. In particular, Donella notes the traps that systems can cause which cause things to go completely awry. Let's see if we can identify any of these traps within the context of enterprise software development:
    • Policy Resistance (think of "The War on Drugs," where two sides are trying to leverage the same system)
    • Tragedy of the Commons (exhausting a shared resource)
    • Drift to Low Performance (goals are eroded because negative feedback has more resonance than positive feedback)
    • Escalation (one side is attempting to out-produce the other, without a balance in between the two sides)
    • Competitive Exclusion (success to the successful)
    • Shifting the Burden to the Intervenor (an addiction has removed a system's ability to shoulder its own burdens)
    • Rule Beating (finding loopholes)
    • Seeking the Wrong Goal

    Any of those sound familiar in your current software engineering practice? No matter if this is exhibited between the business and the engineers, or PM's and engineers, or between engineers - these are universal pratfalls.

    There are ways to influence systems and avoid the traps we often fall into. These leverage points within the system can allow you to alter behavior and encourage positive results. A tricky point remains that some of the leverage points that are easiest to alter have the smallest impact, and some of the largest impact leverage points are very difficult to manipulate. If we look at an Agile software scrum, you might identify least impactful to most these leverage points as:
    1. Numbers, Constants and Parameters. It often feels like you're changing things because you have the most control over these knobs and dials... but all too often reactions are delayed and are cushioned by buffers within the system. Sure, you can change your sprint velocity or begin estimating bugs, but those are just different views on the same result.
    2. Buffers, or the sizes of stabilizing stocks that act as reservoirs of results. A buffer may delay or even out the consequence of a change within the system. Changing buffers would be like changing from a two to a four week development sprint in Agile - you may give yourself more time to recover, but more than likely you're just delaying an inevitable fail.
    3. Failing that, you might try to alter the real, physical parts of the system and how they interact. This can happen, but they are often difficult to change and result in a game of whack-a-mole. This is more fondly called "re-arranging the chairs on the Titanic," and often is exhibited by swapping out team members but keeping the system the same.
    4. The next leverage point might be to try and change how quickly you respond to changes by reducing delays, which in turn alters how quickly the system changes. However, Donella does demonstrate that shorter reaction time can very easily result in greater volatility, and things can become so volatile that they crash. This is what Agile is meant to guard against by locking down a sprint and ensuring priorities aren't changed on a day-by-day basis.
    5. In order to get a grasp on things one may also overlook the balancing feedback loops - or safety measures - that safeguard the system in times of emergency. The excuse is generally that "the worst is unlikely to happen," however this drastically reduces one's survival range. Adaptability is important, and if you take away the ability to adapt you can crash even harder.
    6. Monitoring for reinforcing feedback loops is something that becomes crucially important. This tasks requires one to watch for runaway chain reactions, which can cause a meltdown if not kept in check. Here bad decisions and bad reactions begat even more bad decisions and bad reactions, causing a runaway system. Look for balance instead of infinite feedback loops; if you can keep pushing your tasks to the next sprint, you're only encouraging a runaway backlog of tasks.
    7. Information flows can save a system. If information is in your face and always available, it influences even small decisions. Look at the Nest thermostat or smoke detector - here are devices whose primary purpose is to give you a nonstop flow of info wherever you are. The more info you have (such as how many hours heat was pumped into your house), the more you make small alterations to find balance. This is another part of the Agile process in the form of burndowns/burnups/velocity graphs. This info is meant to be viewed and reflected upon often.
    8. Rules (incentives, punishments, constraints) often have to take place to enforce all the above points. In order to kill feedback loops, ensure emergency systems are maintained and information is shared some rules of the game have to be put into place.
    9. Self-organization, which is an odd juxtaposition of the above rule about rules, is something that Donella prizes most about not only the human condition but systems in general. Usually if you let the component pieces of a system find their role, they will find a way to work with other components in harmony. This is the proof against micro-management; the more you manage, the more you can threaten a system's success. Let developers go free within the confines of the sprint, and don't hover over them (aside from a daily standup).
    10. Find the right goals to change a system. If you focus on GDP, you will focus on gross domestic product at the exclusion of other things. Picking the right goal is tougher than it sounds - you need to know what you want first. However if you can clearly identify and communicate a measurable goal, you can have a huge amount of control over the system. Define what the business actually wants to see - and involve them in the decision making process.
    11. Change your mindset. This is effectively what EVERY project management methodology attempts to do - make you think about the same problem in a different way. If it gives you a renewed perspective, this can be helpful. However...
    12. ...ultimately you should transcend paradigms and realize no paradigms are true. This is what supposed "anti-patterns" are meant to exhibit, and it can be helpful to realize that Agile, just like Waterfall, will ultimately come and go. Just ship early, and ship often.

    Just as we have "Gang of Four" or "Enterprise Integration" patterns, the above are system patterns that can help us decompose and deal with a system. Look for the common traps that always happen - and then evaluate your leverage points to counteract them.