Overview

Today we are going to talk about the differences about each of some of the popular options out there (and the ones that I tested myself) article is inspired by https://phoenixnap.com/blog/kubernetes-vs-docker-swarm

In today computation we are going to evaluate the following points and my personal thoughts about why should we consider all the options (and more, ej podman)

Installation

Docker

Straight forward in all cases and distros, you probably won’t need any sort of instruction outside the documentation to land using docker.

Swarm

Since swarm comes with docker in most distros, you are already there!

Kubernetes

So here things get a little tricky, because kubernetes it’s a suite of components, when we talk about kubernetes we often refer to the cluster itself already working, but to get there we need things like kubeadm or kubectl, from there we can create our cluster with more things like minikube, microk8s, k3s. These provide functionality for the cluster, while the other tools mentioned provide a way to operate with them. You can read more about each one here https://www.linkedin.com/pulse/understanding-kubeadm-kubelet-kubectl-aloysius-pious-fqwsc/

And about the components here components are listed here https://kubernetes.io/docs/reference/command-line-tools-reference/

Learning Curve

Docker

For personal usage I’ll say is quite easy, especially if you are only going to use things that other built, then to build larger things or even try to use it in production (don’t, at least use swarm) it becomes quite a challenge, not because is hard but yes because you start to find the limitations of running docker standalone.

Swarm

After using docker itself, I’ll say transitioning into swarm is quite easy, you start seeing a few commands more and that’s all, what is not the same is that we start digging into container orchestrators. But compared to k8s trust me that is really easy!

Kubernetes

If you come from docker standalone, the concept of container orchestrators itself is going to be a challenge for you, coming from swarm I’ll say is hard but doable and healthy, looking into the additional components of kubernetes is up to you how hard it will become, you can start by trying things like minikube and say of wow this is not that hard, but once you start to think about doing a production ready cluster which is the point of kubernetes (if you are not aiming for that is clear that swarm is enough) challenges for your knowledge start to appear

GUI

Docker

Nothing pre-installed, we can use things docker desktop for personal or team usage, portainer, etc

Swarm

Same case as Docker standalone, dozzle is quite nice since it comes with swarm mode

Kubernetes

k8s comes with a dedicated dashboard, but if you are using k3s like myself, portainer is a good option to manage your cluster

The kubernetes dashboard can be found here https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

Cluster Setup

Docker

N/A

Swarm

To create a cluster is quite easy, we can use the command docker swarm

With the init we start the swarm and with docker swarm join-token we generate a command to join from the other nodes

Then you can check if the nodes are there

From here you can create a compose file and start doing your deploys

Kubernetes

Depending on which engine you are going to use installation varies, in my case i’m using k3s since my cluster is running in multiple Raspberry Pi’s so it comes handy, check https://docs.k3s.io/installation/requirements for details.

After the first installation we will have the cluster with the master node / control-plane, from there we can log in into another machines and do the same with the k3s_token.

To check the state of the cluster and nodes, you can do the following

Scalability

Docker

So we need that our application handles more traffic, if we can to scale we have to of course either expand the resources of the machine, add more replicas (manually) or deploy multiple containers if different machines which a load balancer up front, so docker standalone won’t help us scale, is up to us how we handle it

Swarm

Replicas? easy-peasy, containers across multiple computers? easy-peasy, internal load balancer? easy-peasy, outside automatic horizontal scaling swarm makes really easy this work, it is PERFECT when we have a b2b app or something that we are not going to handle 5000 users and the next day 20 million

Kubernetes

We can’t say much about kubernetes in terms of scalability since it is top-notch, at the cost of the learning curve, and hardware we have all the possible benefits that any company or product needs, I’ll say if you are in a business/product that can grow from day to night, you HAVE to use it. Imagine that you are hosting your website and one day MrBeast decides to promote your site, you know everything is going to blow up that day, kubernetes can save you that day (let’s not talk about $$)

Horizontal auto-scaling

Docker

N/A

Swarm

N/A

Kubernetes

Yes! One of the most useful features, the one that can save you (if you have enough resources to begin) if the example the created above becomes real! :) I’ll leave something to read about it https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Monitoring

Docker and Swarm

Third-party tools are your friends

https://docs.docker.com/engine/daemon/prometheus/

https://github.com/louislam/uptime-kuma

Kubernetes

Resources can be found here https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/

Load balancer

Docker

N/A

Swarm

Suppressively it comes by default

Here I have a few services running, if I point my reverse proxy to any of that external ports it will internally load balance across the replicas.

Kubernetes

The easiest way is to create a Service alongside a Deployment which the type LoadBalancer. An example can be something like this

## infobae-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    io.kompose.service: infobae-api
  name: infobae-api-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      io.kompose.service: infobae-api
  template:
    metadata:
      labels:
        io.kompose.service: infobae-api
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  io.kompose.service: infobae-api
              topologyKey: "kubernetes.io/hostname"
      containers:
        - image: dyallo/infobae_api
          name: infobae-api
          ports:
            - containerPort: 3000
              protocol: TCP
      restartPolicy: Always

## infobae-service.yml
apiVersion: v1
kind: Service
metadata:
  labels:
    io.kompose.service: infobae-api
  name: infobae-api
spec:
  ports:
    - name: 'tcp'
      port: 3000
      targetPort: 3000
  selector:
    io.kompose.service: infobae-api
  type: LoadBalancer

Then we can do run them with kubectl apply -f . if both files are in the same folder.

More information can be found here https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/

Security

All

We can say that all of them have a lot of security options if we dig enough, good resource are

https://book.hacktricks.xyz/linux-hardening/privilege-escalation/docker-security

https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html

which I suggest learning in docker at least is capabilities, security_opts, apparmor, seccomp and Dockerfiles with the least privilege or distroless if applies.

I’ll leave an example of a docker-compose with some of the practices mentioned

services:
  waifuland:
    container_name: waifuland
    security_opt:
      - seccomp:seccomp.json
      - apparmor:non-root-profile
      - no-new-privileges:true
    build:
        context: ../
        dockerfile: docker/Dockerfile
    cap_add: ["DAC_OVERRIDE"]
    cap_drop: ["ALL"]
    restart: always
    env_file:
      - ../.env
    ports:
      - ${EXTERNAL_PORT}:${PORT}
    networks:
      - waifuland

networks:
    waifuland:
        driver: bridge

CLI

Docker

Docker engine itself comes with the docker command which is their own cli to manage both docker standalone and docker swarm

Swarm

Same as docker, with different commands

Kubernetes

The components of Kubernetes itself don’t include a CLI, you have to install it yourself.

Use case

Docker

Let’s say you are trying a new tool, building a feature or anything for development, docker standalone works perfect since the container is going to use the image that the server runs in the end.

Proof of concepts are also welcome since you can run services locally without the need of deployments.

Small companies who have environments which can vary in dependencies and want to regulate their deployments can decide to start using docker, which a really low cost.

All sort of companies can benefit from docker standalone, I’ll not recommend docker standalone for critical applications or core.

Swarm

So swarm is a big step already, one big candidate for swarm for me is B2B models where traffic or users vary only if we got new clients, our application can maybe live with the amount of users that is there and not ground since is a niche market or when is going to grow we now the exact number of users/traffic that we are going to receive so we can scale up in advance, the tools that swarm offers are not automatic but are easy to set up and effective.

Another use case for swarm is the learning process, k8s comes with a big learning curve, if you want to have a more healthy transition going from docker, to swarm, to k8s is going to make your life easier.

Small to medium companies can benefit from docker swarm, I’ll not recommend docker swarm for the core of your business.

Kubernetes

So let’s go with the example from above, MrBeast decides to promote our site, and we know the traffic is going to zero to hero, that’s cool but is not cool if our infrastructure can’t handle that traffic and users receive a painful experience or worst the page is down. With features like horizontal autoscale (as long as we have resources speaking of self-hosted) we can survive this incident, in core or critical applications we NEED things like that where HA is necessary.

So paying the cost of hosting or using/learning is big, but benefit can be also huge if we take the example from above.

K8S is the standard nowadays for enterprise application, my only recommendation is to evaluate each of the needs for the companies, projects and more, you can sometime get away with docker standalone or swarm.

Medium to large companies can benefit from Kubernetes.