… und als dann der Loadbalancer wieder lief, da hat #kured die Mastodon-Datenbank auf einen anderen Node geschoben. Manchmal kommt eins zum anderen.

#k8s #kubernetes https://pastebin.com/raw/rvCipCaz

@samir Every single day a team of 25 people is kept busy running a #K8s cluster with 1200 nodes, that actually could be replaced by less than ten 1U machines using a system design that actually solves the 10K problem, instead of one that struggles to handle even 10 req/s.
This is the vicious cycle of technical debt.

This week's problem: cluster-autoscaler has a bug that causes machines that start up to get stuck in a zombie state without successfully registering with the control plane. This causes all kinds of cluster scale up issues, especially with multi-AZ workloads.

Every week is a new bug, a new edge case, a new issue with dependencies (K8s, helm, Rancher, Istio, etcd, ...) a new issue with AWS, it just goes on and on.

I yearn for the days of simplicity of just running servers in racks and you're like "oh, had another hard drive failure in rack 04, have to go swap out a HDD cartridge and rebuild the RAID".

And why did I choose Talos Linux instead of k3s, minikube, or so many other ways to deploy Kubernetes? Very simple answer: immutable deployment + GitOps. I have a number of hosts that need to run apt/dnf update on a regular basis. As much as this can be automated, it is still tiresome to manage. I don't have to worry as much about an immutable host running a Kubernetes cluster, mostly because the bulk of the attack surface is in the pods, which can be easily upgraded by Renovate/GitOps (which is also something I miss on the hosts running Docker Compose).

Now the research starts. I know Kubernetes, but I don't know Talos Linux, so there's a lot to read because each Kubernetes deployment has it's own nitpicks. Besides, I need to figure out how to fit this new player in my current environment (CA, DNS, storage, backups, etc).

Will my experience become a series of blog posts? Honestly: most likely not. In a previous poll the majority of people who read my blog posts expressed that they're more interested in Docker/Podman. Besides, the Fediverse is already full of brilliant people talking extensively talking about Kubernetes, so I will not be " yet another one".

You will, however, hear me ranting. A lot.

3/3

The main reason for replacing my Proxmox for a Kubernetes deployment, is because most of what I have deployed on it are LXC containers running Docker containers. This is very cumbersome, sounds really silly, and is not even recommended by the Proxmox developers.

The biggest feature I would miss with that move would be the possibility of running VMs. However, so far I've only needed a single one for a very specific test, that lasted exactly one hour, so it's not a hard requirement. But that problem can be easily solved by running Kubevirt. I've done that before, at work, and have tested it in my home lab, so I know it is feasible. Is it going to be horrible to manage VMs that way? Probably. But like I said, they're an exception. Worst case scenario I can run them on my personal laptop with kvm/libvirt.

2/3