Building a Reclaimed Kubernetes Cluster

Thanks to family hand-me-downs, over the passing years I have become a repository of unwanted laptops. Some of them barely boot anymore, but three-quarters of them have two cores and 2 GB of RAM or more. One actually had four cores and 8 GB of RAM... making it a veritable workhorse. I could make them disposable workstations, but instead I wove them together and created a personal Kubernetes cluster.

The base OS for the nodes is Ubuntu's latest LTS release. Rather than using Conjure on MaaS to set up the cluster (which required an isolated network for bootp and DNS and meh), I leveraged Kubespray's flurry of Ansible scripts to prep an inventory of machines over SSH. This ended up being surprisingly low impact and worked perfectly for the use case of building a test lab with piecemeal hardware.

Laptops work just fine as server nodes with a few tweaks:

• Even if you use the server distribution of Ubuntu, laptop events such as closing the lid will still result in a suspend/hibernate/resume action. Edit /etc/systemd/logind.conf to make sure the laptop keeps running when closed:
  sudo vi /etc/systemd/logind.conf
HandleLidSwitch=ignore
sudo service systemd-logind restart

• The display will remain on once you start ignoring LidSwitch events - run a script at startup to turn the display off and save energy.
• Even if you are running in console mode, NVIDIA Optimus laptops will go nuts and seemingly run the discrete and on-chip GPUs nonstop, overheating the machine. Install Ubuntu's Bumblebee packages to prevent this:
  sudo apt-get install bumblebee bumblebee-nvidia primus linux-headers-generic
• As with all Kubernetes nodes, disable swap by commenting out the partition in /etc/fstab. Since you will no longer need to resume from hibernate mode on the laptop, it can be safely disabled

Once you have the laptops prepped and the latest updates applied, you will need to make sure each node has a copy of python-netaddr installed: sudo apt-get install python-netaddr Ansible issues its commands over SSH, so ensure you have keyfile-based authentication set up from the machine you will be running Kubespray on to each of the nodes. If you don't already have an SSH key generated (for example, if you will run Kubespray on the master node), then you can generate a passwordless one via ssh-keygen. After that, copy the public key to each node with:

ssh-copy-id node1
ssh-copy-id node2
...


After that, the machine you are running Kubespray on will need Ansible installed. I ran Kubespray on the master node to keep things simple - so on that Ubuntu box I issued:

sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible
git clone https://github.com/kubernetes-incubator/kubespray.git
cp -rfp inventory/sample inventory/mycluster

This will:

1. Install Ansible on the box
2. Download the Ansible scripts from Kubespray
3. Creates a new Ansible inventory called "mycluster" that is a clone of the Kubespray sample

An important thing to remember is that you address nodes by straight IP address - not by hostname. This is especially important with Ansible scripts because the node's hostname may well change as part of the installation process. If your nodes are fetching their IP address via a DHCP server, ensure the DHCP server has static IP allocations for your nodes.

Once you have all the IP addresses for your nodes, set them in your inventory file. An easy way to do this at the command line is:

declare -a IPS=(192.168.1.32 192.168.1.36 192.168.1.40)
CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}


Verify the inventory is correct by cracking open inventory/mycluster/hosts.ini - if you want to change hostnames, now is the time.

I would recommend having Kubespray build a kubectl configuration file automagically for you. To have this generated as an artifact, change inventory/mycluster/group_vars/k8s-cluster.yml to have the following entry set: kubeconfig_localhost: true After these tweaks you should be ready to launch Kubespray's Ansible playbook. Note that Ubuntu's convention is to have you operate as a normal user and sudo all of your commands, so you will need to use Ansible's --become parameter:

ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml --ask-become-pass --become


At this point Kubespray will try its best to get a cluster up and running on the nodes specified in your inventory file. At the very end Kubespray will provide you with a kubectl configuration file in artifacts/admin.conf, which you can then copy or merge into another workstation's ~/.kube/config file.

Once you have the Kubernets configuration file set on your workstation, you can use it to fetch an authtoken to get into the Kubernetes Dashboard. The proper way to do this is to generate a new system secret that has the appropriate permissions to interrogate the running cluster... but the lazy way is to just steal the token used by Kubernetes' namespace controller.

I'm lazy, so first I list all the secretes in the kube-system namespace:

kubectl -n kube-system get secrets

And then fetch the token for the namespace controller:

kubectl -n kube-system describe secret namespace-controller-token-???

So that I can use it to login to the web dashboard:

kubectl proxy &
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

Now you should have a working cluster you can mess with!

So that the laptops were properly ventilated, I placed each vertically into a metal document sorter from an office supply store. This gives me a nifty vertical rack for the laptops that has plenty of air circulation and allows me to route cables out of the way.

I've constructed one weird frakencluster - but it works!

Massively Parallel Compute as a Service

Back in the Spring of 2012, I asked several panelists at VMWorld to weigh in on vector processing with GPUs as a big data/big compute solution. The response was a resounding "not yet," as the infrastructure has not yet reached commodity level and GPU processing was greatly constrained by memory paging. It now seems like both obstacles are being removed.

Amazon Web Services is now offering EC2 instances that offer up virtualized instances of NVIDIA's Kepler GPUs as "G2" instances. This supports H.264 encoding, OpenCL, CUDA and OpenGL toolsets which allows for more mature toolsets to build apps targeted to these vector processing instances. This kind of support allows for commodity toolchains and commodity infrastructure to allow for massively parallel processing on demand.

Memory paging should soon be addressed by NVIDIA via CUDA 6, and should also be addressed by AMD with its upcoming Kaveri architecture. Once memory addressing is unified, the swapping of memory regions should become unnecessary and allow for memory to be addressed locally without pagination. This simplifies application development, virtualization and hardware architectures considerably.

I believe that very soon we will see vector processing at scale garner as much attention as map/reduce clusters currently do. Massive data parsing has been commoditized, and now we have an opportunity to commoditize massive algorithmic crunching.

Filling The Pipeline

My past few work engagements have been centered around cloud computing and big data - doing stuff from managing large data centers to machine learning to map/reduce clusters. When I was at VMworld 2011 last year I took the opportunity to ask the "Big Compute and Big Data Panel" about leveraging vector processing hardware such as NVIDIA's Tesla to do data processing. The five panelists (Amr Awadallah of Cloudera, Clint Green of Data Tactics, Luke Lonergan of EMC, Richard McDougall of VMware and Paul Kent of SAS) largely agreed on a few main sticking points in vector processing for massively parallel systems:

• The toolset is still relatively immature (maybe three years behind general CISC architectures)
  
• The infrastructure has not yet reached commodity level
  
• Big Compute works well with vector processing clusters, but not big data, since the latter is all about locality rather than in-memory processing
  
• Commodity GPU processing is greatly constrained by memory paging - there's too much latency in transferring large in-memory datasets to GPU memory.
  

AMD had a few interesting announcements over the past few weeks that may pave the way for making cloud and big data/compute clusters more efficient and more "commodity." The first is their acquisition of SeaMicro, whose emphasis is around massively parallel, low-power computing cores with high-speed interconnects. This addresses one big issue brought up during the panel - that interconnects on big data clusters are going to become a prevailing issue as data needs to be transferred across nodes more rapidly to keep otherwise idle compute resources busy. CPUs can't crunch data sets if the data takes forever to arrive over the wire.

The next big announcement, which may be a huge sleeper hit, is AMD's unified memory architecture that's supposed to arrive in 2012. The slide on AnandTech shows that in AMD's 2012 product line the "GPU can access CPU memory," which is a HUGE development in vector processing. Imagine a data set being loaded in 64 GB of main memory, having 8 CPU cores clean the data using branch-intensive algorithms and then that same in-memory dataset being transformed by 512 stream processors. That kind of compute power without the need to stream data across a PCI-E bus could be a really, really big deal.

Still, the issue that remains are the tools that are available to make this happen. Very likely a developer would need to write generic C code to do the branching and then launch a separate OpenCL app to transform it but still share memory pointers so that nothing has to be swapped or paged out. In a world full of enterprise software developers, this kind of software engineering agility isn't exactly easy to find. If Cloudera were able to unleash this kind of power, AMD would have a big hit on their hands. Maybe AMD needs to start looking towards Cloudera as the final stage in the pipeline - an open-source framework that unlocks the potential of their infrastructure.

Old is the new New

The open beta of Quake Live opened this Tuesday, and I made sure to jump on and register an account as soon as I could. Of course, like most other people online at Tuesday night, I was in queue with tens of thousands of other players. I finally had an opportunity to play last night for about 30 minutes, just to make sure my account worked and see how things were put together.

It is definitely the classic Quake III Arena in all its OpenGL goodness. When the title first launched over nine years ago its hardware requirements may have stopped it from becoming ubiquitous; it supported hardware rendering only and, unlike other titles shipping at the time, didn't have a software renderer. Ah, how times have changed. A quick look at Steam's hardware survey shows how the desktop tide has changed, and now tons of people have way more than enough horsepower. And it doesn't stop at the desktop - Q3 has hit every major console and is even being developed for the Nintendo DS. It has even been ported to the iPhone. Official Linux and Mac support of Quake Live is reportedly a priority after the beta, granting an increasing OS X demographic access. Ubiquity no longer is a problem; making the installation as easy as a multi-platform browser plugin lowers the barrier of entry to near nil.

The key factor that stops Quake Live from just being a Q3A port is the actual infrastructure it resides within. The game proper is the endpoint, but the content itself is driven from the ladders, achievements, matchmaking and map inventory system contained within the Quake Live web application. It is one of those blindingly obvious why-isn't-everone-doing-this moments when you see how the game is orchestrated with the Quake Live portal; the strengths of the browser as a platform is completely leveraged, while the strengths of your desktop are used to power the game itself. The creators didn't try to cram Quake III into the browser itself, thereby condoning it to some sort of Flash-based hell. Instead they let you use the browser just as you would normally use it: for networking, finding a game, chatting, browsing leaderboards, looking at achievements, bugging friends, strutting your profile and other... forgive me for saying this... "social networking" features. When it comes time to do the deathmatch an external application is launched in tandem, allowing a fully fledged and fast OpenGL app to run on its own.

An interesting effect of this split-brainness between an online presence and a desktop renderer is that it accomplishes exactly what Valve wants to do via the Steam Cloud, where preferences and saves are stored on a central network instead of client-side. Most desktop gamers don't like the idea of savegames or prefs being stored on a remote server pool, and I would agree. For single-player experiences I would much rather hack my own .ini files and not be stranded when someone's cloud goes down in flames (clouds do NOT equal uptime... see Google and Amazon themselves for examples). However for multiplayer games this is acceptable; if a server is dead or a line is cut you wouldn't be able to multiplayer anyway - so it doesn't matter where configs reside. As an added bonus when the configs reside remotely you can't have players hack them, resulting in reducing the map to a wireframe or performing some esoteric modification to give them a competitive edge. Again, hacks are fine in single player, but not in a multiplayer scenario.

Add into the mix the fact that the economy has positively tanked and people have completely eviscerated discretionary spending, meaning $60 titles are no longer in the budget. Quake Live brings a new title, albeit of an old game, to market for the price of absolutely free. While it is true that CPMs and CPCs for online advertising has completely dropped through the floor, hopefully Quake Live will be able to cash in on its unique presentation, dedicated fan base and sheer volume of eyeballs. If the advertising model works, and Quake Live continues to be free this will provide a huge edge over other FPS titles this year.

Quake Live is a game-changing title, even if they didn't change the game. But why bother? Quake III Arena was arguably one of the most well-rounded and polished multiplayer first-person shooters out there with textbook weapon balancing and gameplay mechanics that became a staple in the genre. Why change something that works? The only balance issue that the original Quake III Arena had was that, towards the end, veteran players became so good that it was no longer possible for a new player to have any fun on a map. Now with Quake Live's matchmaking mechanics and dynamic skill levels even that mismatch has been mitigated.

Yeah, I've already gone on too long about this. This approach just makes so much sense from an engineering perspective and a gaming perspective that I'm sure tons of titles are now going to flood into the market, ready to follow suit.