Overview
Today we are going to talk about the differences about each of some of the popular options out there (and the ones that I tested myself) article is inspired by https://phoenixnap.com/blog/kubernetes-vs-docker-swarm
In today computation we are going to evaluate the following points and my personal thoughts about why should we consider all the options (and more, ej podman)
Installation
Docker
Straight forward in all cases and distros, you probably won’t need any sort of instruction outside the documentation to land using docker.
Swarm
Since swarm comes with docker in most distros, you are already there!
Kubernetes
So here things get a little tricky, because kubernetes it’s a suite of components, when we talk about kubernetes we often refer to the cluster itself already working, but to get there we need things like kubeadm or kubectl, from there we can create our cluster with more things like minikube, microk8s, k3s. These provide functionality for the cluster, while the other tools mentioned provide a way to operate with them. You can read more about each one here https://www.linkedin.com/pulse/understanding-kubeadm-kubelet-kubectl-aloysius-pious-fqwsc/
And about the components here components are listed here https://kubernetes.io/docs/reference/command-line-tools-reference/
Learning Curve
Docker
For personal usage I’ll say is quite easy, especially if you are only going to use things that other built, then to build larger things or even try to use it in production (don’t, at least use swarm) it becomes quite a challenge, not because is hard but yes because you start to find the limitations of running docker standalone.
Swarm
After using docker itself, I’ll say transitioning into swarm is quite easy, you start seeing a few commands more and that’s all, what is not the same is that we start digging into container orchestrators. But compared to k8s trust me that is really easy!
Kubernetes
If you come from docker standalone, the concept of container orchestrators itself is going to be a challenge for you, coming from swarm I’ll say is hard but doable and healthy, looking into the additional components of kubernetes is up to you how hard it will become, you can start by trying things like minikube and say of wow this is not that hard, but once you start to think about doing a production ready cluster which is the point of kubernetes (if you are not aiming for that is clear that swarm is enough) challenges for your knowledge start to appear
GUI
Docker
Nothing pre-installed, we can use things docker desktop for personal or team usage, portainer, etc
Swarm
Same case as Docker standalone, dozzle is quite nice since it comes with swarm mode
Kubernetes
k8s comes with a dedicated dashboard, but if you are using k3s like myself, portainer is a good option to manage your cluster
The kubernetes dashboard can be found here https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
Cluster Setup
Docker
N/A
Swarm
To create a cluster is quite easy, we can use the command docker swarm
With the init we start the swarm and with docker swarm join-token we generate a command to join from the other nodes
Then you can check if the nodes are there
From here you can create a compose file and start doing your deploys
Kubernetes
Depending on which engine you are going to use installation varies, in my case i’m using k3s since my cluster is running in multiple Raspberry Pi’s so it comes handy, check https://docs.k3s.io/installation/requirements for details.
After the first installation we will have the cluster with the master node / control-plane, from there we can log in into another machines and do the same with the k3s_token.
To check the state of the cluster and nodes, you can do the following
Scalability
Docker
So we need that our application handles more traffic, if we can to scale we have to of course either expand the resources of the machine, add more replicas (manually) or deploy multiple containers if different machines which a load balancer up front, so docker standalone won’t help us scale, is up to us how we handle it
Swarm
Replicas? easy-peasy, containers across multiple computers? easy-peasy, internal load balancer? easy-peasy, outside automatic horizontal scaling swarm makes really easy this work, it is PERFECT when we have a b2b app or something that we are not going to handle 5000 users and the next day 20 million
Kubernetes
We can’t say much about kubernetes in terms of scalability since it is top-notch, at the cost of the learning curve, and hardware we have all the possible benefits that any company or product needs, I’ll say if you are in a business/product that can grow from day to night, you HAVE to use it. Imagine that you are hosting your website and one day MrBeast decides to promote your site, you know everything is going to blow up that day, kubernetes can save you that day (let’s not talk about $$)
Horizontal auto-scaling
Docker
N/A
Swarm
N/A
Kubernetes
Yes! One of the most useful features, the one that can save you (if you have enough resources to begin) if the example the created above becomes real! :) I’ll leave something to read about it https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Monitoring
Docker and Swarm
Third-party tools are your friends
https://docs.docker.com/engine/daemon/prometheus/
https://github.com/louislam/uptime-kuma
Kubernetes
Resources can be found here https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
Load balancer
Docker
N/A
Swarm
Suppressively it comes by default
Here I have a few services running, if I point my reverse proxy to any of that external ports it will internally load balance across the replicas.
Kubernetes
The easiest way is to create a Service alongside a Deployment which the type LoadBalancer. An example can be something like this
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
io.kompose.service: infobae-api
name: infobae-api-worker
spec:
replicas: 3
selector:
matchLabels:
io.kompose.service: infobae-api
template:
metadata:
labels:
io.kompose.service: infobae-api
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
io.kompose.service: infobae-api
topologyKey: "kubernetes.io/hostname"
containers:
- image: dyallo/infobae_api
name: infobae-api
ports:
- containerPort: 3000
protocol: TCP
restartPolicy: Always
apiVersion: v1
kind: Service
metadata:
labels:
io.kompose.service: infobae-api
name: infobae-api
spec:
ports:
- name: 'tcp'
port: 3000
targetPort: 3000
selector:
io.kompose.service: infobae-api
type: LoadBalancer
Then we can do run them with kubectl apply -f . if both files are in the same folder.
More information can be found here https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
Security
All
We can say that all of them have a lot of security options if we dig enough, good resource are
https://book.hacktricks.xyz/linux-hardening/privilege-escalation/docker-security
https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html
which I suggest learning in docker at least is capabilities, security_opts, apparmor, seccomp and Dockerfiles with the least privilege or distroless if applies.
I’ll leave an example of a docker-compose with some of the practices mentioned
services:
waifuland:
container_name: waifuland
security_opt:
- seccomp:seccomp.json
- apparmor:non-root-profile
- no-new-privileges:true
build:
context: ../
dockerfile: docker/Dockerfile
cap_add: ["DAC_OVERRIDE"]
cap_drop: ["ALL"]
restart: always
env_file:
- ../.env
ports:
- ${EXTERNAL_PORT}:${PORT}
networks:
- waifuland
networks:
waifuland:
driver: bridge
CLI
Docker
Docker engine itself comes with the docker command which is their own cli to manage both docker standalone and docker swarm
Swarm
Same as docker, with different commands
Kubernetes
The components of Kubernetes itself don’t include a CLI, you have to install it yourself.
Use case
Docker
Let’s say you are trying a new tool, building a feature or anything for development, docker standalone works perfect since the container is going to use the image that the server runs in the end.
Proof of concepts are also welcome since you can run services locally without the need of deployments.
Small companies who have environments which can vary in dependencies and want to regulate their deployments can decide to start using docker, which a really low cost.
All sort of companies can benefit from docker standalone, I’ll not recommend docker standalone for critical applications or core.
Swarm
So swarm is a big step already, one big candidate for swarm for me is B2B models where traffic or users vary only if we got new clients, our application can maybe live with the amount of users that is there and not ground since is a niche market or when is going to grow we now the exact number of users/traffic that we are going to receive so we can scale up in advance, the tools that swarm offers are not automatic but are easy to set up and effective.
Another use case for swarm is the learning process, k8s comes with a big learning curve, if you want to have a more healthy transition going from docker, to swarm, to k8s is going to make your life easier.
Small to medium companies can benefit from docker swarm, I’ll not recommend docker swarm for the core of your business.
Kubernetes
So let’s go with the example from above, MrBeast decides to promote our site, and we know the traffic is going to zero to hero, that’s cool but is not cool if our infrastructure can’t handle that traffic and users receive a painful experience or worst the page is down. With features like horizontal autoscale (as long as we have resources speaking of self-hosted) we can survive this incident, in core or critical applications we NEED things like that where HA is necessary.
So paying the cost of hosting or using/learning is big, but benefit can be also huge if we take the example from above.
K8S is the standard nowadays for enterprise application, my only recommendation is to evaluate each of the needs for the companies, projects and more, you can sometime get away with docker standalone or swarm.
Medium to large companies can benefit from Kubernetes.
