I: Building kubernetes home lab with k3d — Cluster deployment

07.12.2020 - AYB - Reading time ~1 Minute

K3D Logo

Part I: Kubernetes homelab with k3d

Let’s assume that you’re a developer or junior DevOps engineer who got the task “This thing must run in Kubernetes tomorrow” but you have no clue what the hell Kubernetes is and have to learn it ASAP to have your job the day after tomorrow.

*Disclaimer: Kubernetes is a multilayered abstraction realm for running various stuff isolated in sandboxes and make all this stuff communicate and collaborate at the same time. User/admin has some control over the Kubernetes to ask it do things in a desired way. The deeper you’re getting into it, the more scary it becomes, but it’s ok - everyone feels himself completely pissed off by Kubernetes at least 9100 times while learning it.*

Getting your first Kubernetes cluster running

Prerequisites

  • Linux/MacOS
  • Docker

This guide may be used on Windows, but it will require some changes to the operations/command that will NOT be highlighted here.

Deploying the lightweight dockerized k8s cluster using k3d

K3d Homepage

There are several methods how to install k3d on your computer and by default it’s offered to use crazy popular method like wget https://something/install.sh | bash. But it’s insanely stupid to use that method for many quite obvious reasons. If you’re not aiming to become a somewhat good DevOps, then just remember that this method fucks up all your security unless you are the author of the script.

Go to the Releases page and download the appropriate release for your OS/arch. Then chmod +x on the binary and put it somewhere in your $PATH. Or put somewhere and add this path to $PATH.

Whatsoever installation method you choose, lets start.

Deploying a control node with two workers:

k3d cluster create kube -p "80:80@loadbalancer" -p "443:443@loadbalancer" --agents 2

Note: With the parameters -p "80:80@loadbalancer" and -p "443:443@loadbalancer" we’re tellind k3d to occupy local ports 80 and 443 and route traffic to the cluster loadbalancer. These parameters are vitally important because otherwise you’d be unable to reach your applications in the cluster over the network. If you’re planning to have any services on other ports — just add the corresponding ports to this command.

Output

INFO[0000] portmapping '80:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy] 
INFO[0000] portmapping '443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy] 
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-kube'
INFO[0000] Created image volume k3d-kube-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-kube-tools'
INFO[0001] Creating node 'k3d-kube-server-0'
INFO[0001] Creating node 'k3d-kube-agent-0'
INFO[0001] Creating node 'k3d-kube-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-kube-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] Starting new tools node...
INFO[0001] Starting Node 'k3d-kube-tools'
INFO[0002] Starting cluster 'kube'
INFO[0002] Starting servers...
INFO[0002] Starting Node 'k3d-kube-server-0'
INFO[0005] Starting agents...
INFO[0006] Starting Node 'k3d-kube-agent-1'
INFO[0006] Starting Node 'k3d-kube-agent-0'
INFO[0017] Starting helpers...
INFO[0018] Starting Node 'k3d-kube-serverlb'
INFO[0024] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap... 
INFO[0026] Cluster 'kube' created successfully!
INFO[0026] You can now use it like this:
kubectl cluster-info
$kubectl get nodes
NAME                STATUS   ROLES                  AGE   VERSION
k3d-kube-server-0   Ready    control-plane,master   44s   v1.22.7+k3s1
k3d-kube-agent-0    Ready    <none>                 35s   v1.22.7+k3s1
k3d-kube-agent-1    Ready    <none>                 35s   v1.22.7+k3s1

Here we see that we got control plane and master node. But we need a space to deploy stuff. Lets go for worker nodes; apparently we’ll create two of them.

Check it

$k3d node list
NAME                ROLE           CLUSTER   STATUS
k3d-kube-agent-0    agent          kube      running
k3d-kube-agent-1    agent          kube      running
k3d-kube-server-0   server         kube      running
k3d-kube-serverlb   loadbalancer   kube      running
k3d-kube-tools                     kube      running

Check docker containers status

$docker ps -a
CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS         PORTS                                                               NAMES
e2cdc91188d7   ghcr.io/k3d-io/k3d-tools:5.4.1   "/app/k3d-tools noop"    3 minutes ago   Up 3 minutes                                                                       k3d-kube-tools
f3c3cf1da08a   ghcr.io/k3d-io/k3d-proxy:5.4.1   "/bin/sh -c nginx-pr…"   3 minutes ago   Up 2 minutes   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:51772->6443/tcp   k3d-kube-serverlb
1bca176b8b44   rancher/k3s:v1.22.7-k3s1         "/bin/k3d-entrypoint…"   3 minutes ago   Up 3 minutes                                                                       k3d-kube-agent-1
a57669c52085   rancher/k3s:v1.22.7-k3s1         "/bin/k3d-entrypoint…"   3 minutes ago   Up 3 minutes                                                                       k3d-kube-agent-0
94d7133d3d5c   rancher/k3s:v1.22.7-k3s1         "/bin/k3d-entrypoint…"   3 minutes ago   Up 3 minutes                                                                       k3d-kube-server-0

Everything seems fine.

Tricking DNS

Now we need to make some trick with DNS to avoid future issues with addressing to localhost — edit the /etc/hosts file. Run sudo vim /etc/hosts and at the end of line with 127.0.0.1 localhost add k3d.local It should become 127.0.0.1 localhost k3d.local

Deploying our first service

Disclaimer: I got sick at some moment typing all these kubectl get ... so I made some aliases listed below and added them to my ~/.zshrc.

alias k=kubectl
alias kg="kubectl get"
alias kgp="kubectl get pods -o wide"
alias kgpa="kubectl get pods --all-namespaces"
alias kgs="kubectl get services -o wide"
alias kgsa="kubectl get services --all-namespaces -o wide"
alias kgi="kubectl get ingresses"
alias kgia="kubectl get ingresses --all-namespaces"
alias kd="kubectl describe"
alias kdp="kubectl describe pod"
alias kds="kubectl describe service"
alias kdd="kubectl describe deployment"
alias kdi="kubectl describe ingress"

The most popular test service is «Whoami» by Traefik labs. At the GitHub there is a very nice deployment example for that service for Kubernetes.

git clone https://github.com/pacroy/whoami && cd whoami

Now change whoami.yml in line 6 to have replicas: 2

Now deploy the service

kubectl create ns whoami
kubectl apply -f whoami.yml -n whoami

Output:

namespace/whoami created
deployment.apps/whoami created
service/whoami created

Checking out the result:

$kgp -n whoami
NAME                      READY   STATUS    RESTARTS   AGE   IP          NODE              NOMINATED NODE   READINESS GATES
whoami-75d5976d8d-k8jfq   1/1     Running   0          13s   10.42.1.3   k3d-kube-node-0   <none>           <none>
whoami-75d5976d8d-t92cj   1/1     Running   0          13s   10.42.2.3   k3d-kube-node-1   <none>           <none>

Now edit the ingressroute.yml, line 9 to look like

- match: Host(`k3d.local`) && PathPrefix(`/whoami`)

and apply the route:

kubectl apply -f ingressroute.yml -n whoami

Lets test our very new test deployment by opening http://k3d.local/whoami at the browser. The output should be very alike the following:

Hostname: whoami-75d5976d8d-zz4bf
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.5
IP: fe80::dc98:ffff:fea4:c55
RemoteAddr: 10.42.2.4:55158
GET /whoami HTTP/1.1
Host: k3d.local
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-CA,en-US;q=0.7,en;q=0.3
Dnt: 1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.42.2.1
X-Forwarded-Host: k3d.local
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-56c4b88c4b-qqqsx
X-Real-Ip: 10.42.2.1

And since we’re having two replicas of the whoami pod, we can refresh the page and we’ll see how responding IP address and Port are changing which means whe’re successfully reaching both of our pods with the service. Hooraaah!

Now you have a fully functional Kubernetes cluster on your local machine.

Some useful notes

Install OpenLens — this is an awesome tool to manage your clusters and environments. Can’t get overexcited about it.

This cluster has Traefik ingress controller deployed by default. Traefik is a lightweight yet powerful router/proxy for Kubernetes, though it has a quite steep learning path. https://docs.traefik.io/ is your best friend with that.

Next: Building kubernetes home lab with k3d — Monitoring