Docker, Kubernetes and everything else related to containers are gaining every day more traction and after a somewhat slow start this seems to be “the next big thing” and everybody is rushing to support it. Microsoft, a late adopter, is now throwing everything it has to it while Red Hat and Google already have a solid offerings with OpenShift and GKE.

On a smaller scale I’m using plain Docker “in production” on my VPS since 2015 and it really changes your workflow, thus making really easy having multiple applications running on the same VPS without interferences. At first it’s quite painful but once that everything runs smoothly your continuous deployment pipeline works like a charm and you don’t have to worry about strange interactions caused by sharing the same OS.

The step up from Docker is using it with an orchestrator to manage multiple nodes. I used my MSDN Azure credits to try Docker Swarm, OpenShift Origin and Kubernetes but I always felt that something was missing: it works, it works great but you can’t really understand what’s going on under the hood. At the Voxxed Lugano I watched Nicola Ferraro’s presentation on OpenShift (a must see if you haven’t seen it) and he was talking about, among other things, about a Raspberry Pi cluster. That’s when something clicked in my mind and I decided to build my own following a couple of very well done tutorials that I found around.

Buying the components

The first step is to get the components and I bought everything from ModMyPi because they have a huge stock at fair prices, the only thing that they don’t sell is the Wi-Fi router but it’s sold by Amazon so it’s even easier to get.

This was my shopping list:

Product Quantity Price (EUR) Total (EUR)
Anidées 6 Port 60W High Power USB Hub 1x 36.16 36.16
Raspberry Pi 3 - Model B & 32GB MicroSD Card Bundle 4x 44 .43 177.72
Multi-Pi Stackable Raspberry Pi Case 3x 13.38 40.14
USB to Micro USB Cable 0.5m 5x 1.55 7.75
TP-Link TL-WR802N N300 WLAN Nano Router 1x 25.83 25.83
TOTAL     287.60

This may seem a bit expensive for an experiment but at 71.90 EUR per node I think that there isn’t a cheaper alternative to play with Kubernetes at that level. There is also something fun that I can describe as a “LEGO experience” when you follow the instructions and install Kubernetes.

Assemble the cluster

Assembling the cluster is very straightforward, just remember to always align correctly the boards with the stackable case and in less than 30 minutes you should be ready to deploy the software. As you can see I put the Power Hub at the bottom, the 4 Raspberry Pi’s and the Wi-Fi router on the top. The router’s purpose is to provide connectivity to the Raspberries and at the same time connect to an “upstream” router so it can relay also an internet connection. I found very helpful the articles written by Roland Huß to automate the deployment with Ansible (Project31/ansible-kubernetes-openshift-pi3) and Sergio Sisternes for providing a guide on how to setup the WiFi router (Recipe – Kubernetes cluster with Raspberry Pi 3).

The final result should look like this:

Setup the hardware

Most of the installation is automated by using Ansible. Thanks to Hypriot images a complete headless setup is possible.

  1. Download the latest Hypriots image and store it as hypriot.zip:
    curl -L https://github.com/hypriot/image-builder-rpi/releases/download/v1.5.0/hypriotos-rpi-v1.5.0.img.zip
    
  2. Install Hypriot’s flash installer script. Follow the directions on the installation page.

  3. Insert you Micro-SD card in your Desktop computer (via an adapter possibly) and run
    flash --hostname n1 --ssid "mysid" --password "secret" hypriotos-rpi-v1.5.0.img.zip
    

    You will be asked to which device to write. Check this carefully, otherwise you could destroy your Desktop OS if selecting the the wrong device. Typically its something like /dev/disk2 on OS X, but depends on the number of hard drives you have.

  4. Repeat step 2. to 3. for each Micro SD card. Please adapt the hostname before each round to n2, n3, n4.

Configure the network

It is now time to configure your WLAN router. This of course depends on which router you use. The following instructions are based on a TP-Link TL-WR802N which is quite inexpensive but still absolutely ok for our purposes since it sits very close to the cluster and my notebook anyway.

First of all you need to setup the SSID and password. Use the same credentials with which you have configured your images.

The addresses I have chosen are:

IP Device
192.168.23.1 WLAN Router
192.168.23.181192.168.23.184 Raspberry Pis

You should be able to SSH into every Pi with user pirate and password hypriot. Internet access from the nodes is mandatory for setting up the nodes with Ansible.

After that you checked every Raspberry you can configure the router using these steps detailed by Sergio Sisternes:

  • Go to DHCP > Client list and take note of the MAC addresses
  • Go to DHCP > Address reservation, and set a static (Reserved) IP address to each Raspberry
  • Go to Quick Setup and configure the router in the hotspot mode so it can connect itself to another router and share Internet with the Raspberries

Configure the cluster using Ansible

After this initial setup is done, the next step is to initialize the base system with Ansible. You will need Ansible 2 installed on your desktop (e.g. brew install ansible when running on OS X) and a copy of my GitHub repository with the playbooks. The project is largely based on of a great repository made by Roland Huß: Project31/ansible-kubernetes-openshift-pi3.

The project’s main features are:

  • From zero to a running cluster with a single Ansible playbook
  • Kubernetes 1.7
  • WiFi networking and Flannel as CNI
  • Cluster aware storage available out of the box using GlusterFS (replicated on every node)
  • Dashboard and Heapster deployed by default along with Traefik (a reverse proxy that exposes your containers to the outside world)
  • Manual deployments for the Docker registry, MySQL, whoami and Lavagna.

Ansible configuration

  1. Checkout the Ansible playbooks:

     git clone https://github.com/SitoCH/rpi-kubernetes-cluster.git rpi-kubernetes-cluster
     cd rpi-kubernetes-cluster
    
  2. Copy over hosts.example and adapt it to your needs

     cp hosts.example hosts
     vi hosts
    

    There are three groups:

    • pis contains all members of your cluster where one is marked as “master” in the field host_extra. This group will be added to every node in its /etc/hosts. It is important that one host is marked as “master”, since the playbooks rely on this host alias for accessing the API server.
    • master IP address of the Master
    • nodes All nodes which are not Master
    • volumes Volumes to create on GlusterFS

Configure the cluster

Run the deployment with the command:

ansible-playbook -k -i hosts setup.yml

Deploy default applications

Once that the cluster is up and running this playbook will deploy GlusterFS endpoints, Heapster, Dashboard and Traefik:

ansible-playbook -k -i hosts deployments.yml
Traefik

Traefik is a reverse proxy that exposes your containers to the outside world. The management dashboard is available at http://master-node:8080/

Manual deployments

For convenience useful deployments can be found in the directory manual-deployments, please note that you must install the Registry in order to use other manual deployments because the images will be centrally cached.

Registry

This container installs a Docker registry that acts as a pass-trough cache, this way only the first node will download an image from Internet and all the other requests will be served from “inside” the cluster.

kubectl --kubeconfig run/admin.conf create -f manual-deployments/registry/registry.yml

Who am I

Simple container that deploys on 3 nodes a website that prints it’s container ID.

Thanks to Traefik you can access it on http://master-node/whoami/

kubectl --kubeconfig run/admin.conf create -f manual-deployments/whoami/whoami.yml

MySQL

MySQL database server with persistent storage.

kubectl --kubeconfig run/admin.conf create -f manual-deployments/mysql/mysql.yml

Lavagna

Lavagna is an open-source issue/project management tool designed for small teams (another open source project of mine made with 2 friends).

kubectl --kubeconfig run/admin.conf create -f manual-deployments/lavagna/lavagna.yml

Conclusions

Building a real, tangible cluster from the ground up has been an amazing experience. It really helps you understand the core mechanics that are always abstracted when you deploy to a cloud provider like Azure. Just to name a few key areas that can be understood only by this “hands-on” approach:

  • Storage: storage is the hardest part to get right when using containers and tinkering with GlusterFS, volumes, claims and bindings really give you a good insight.
  • Container networking: another “black magic” topic, especially when working with different CNI plugins.
  • Reverse proxy: Traefik helps you proxy multiple applications in an efficient way but, as usual, trying is the best way to learn.
  • ARM based Docker images: nobody cares about a container’s base image because it’s usually x86, Raspberry Pi are ARM based so you have to dig deeper, find a good base image and understand how to build an ARM image on OS X.

Least but not last let’s show some screenshots of the Kubernetes dashboard that prove that installing a Raspberry Pi cluster is challenging but rewarding: