#ranchervm

[dummdida]

That thing that makes KubeVirt a little different - or the Kubernetes virtualization API.

An unsual but persistent part in the KubeVirt development is the following question:

How does KubeVirt compare to virtlet, runv/hyper, oci-cc-runtime/ClearContainers, RancherVM, and alike?

It's a good question. And it actually took a while to get a clear view on this.

To get this clear view, I'll focus on two details:

KubeVirt is not a container runtime, and thus not using the CRI. The VM runtime (libvirt) is provided in a pod instead.

KubeVirt provides a dedicated API for VMs

Especially the first point is raised pretty often. The reason for this is - eventually - that it looks so obvious to just launch VMs instead of containers on the kubelet side - virtlet, runv, and oci-cc-runtime show that it really is an option (without going to much into the details here). Very often this is directly coupled to the question of why we ship our runtime in a pod.

The answer is simply that so far this setup "works":

we can spawn VMs

our runtime is Kubernetes native

because everything lives in pods, we can be delivered to "any" Kubernetes cluster, without messing with the host or kubelet.

We can be a native Kubernetes add-on.

Btw - A nice side effect is that VMs become just a workload - and this is a good separation of concerns and responsibility. Kubernetes owns the platform, and we provide our application to run specialized (virtual machine) processes on this platform.

But even with these benefits: There are some details (i.e. how to get VMs into ressource groups of other pods) which are a little nasty to solve for us, which might be easier to solve in the CRI based approach.

And we see - we are a few lines into the post, and the discussion is going down the winding road of seeing which runtime approach is the best.

That's how it often goes, but that's not where I want to right now.

KubeVirt is not only a runtime. KubeVirt also has an explicit API for virtualization. Initially we used TPRs, then as Kubernetes extension capabilities evolved we went over to CRDs, and are now working on our own User API Server to be used with API Server aggregation. Regardless of this history - The important bit is that we have an explicit API dedicated to virtualization.

And this - in retrospective (and forward facing) - is the significant difference.

The explicit virtualization related API - which consists of new top level types for different things in the virtualization area (like VMS) - gives us the freedom to express all the functionality we need to provide a robust, solid, and stable virtualization platform.

The other approaches (except for RancherVM) have a different view here: They re-use (intentionally and with different motivations) the existing pod specification, to derive VMs from them. But this approach comes to it's limits as soon as virtualization specific functionality (for example live migration) is needed. For those features pod's don't have an API, and annotations or other workarounds need to be used to "logically extend" the pod concept, to enable this missing (virtualization) functionality.

I'm not saying that the one way or the other is technically wrong. It's just that both approaches might just be addressig different goals. I.e. oci-cc-runtime has clearly the goal of using VMs for isolation purpose only - So far it does not look as if they want to expose them as "fully fledged VMs".

And to close the loop: The API is pretty independent of the runtime, and this tells me that in the end it is not that important how the VMs are run - with a runtime in a pod or via a CRI based runtime. The important bit is, that with KubeVirt the user is given the full controll over his VMs.

Side note - Instead of a dedicated API, there is also the option to extend the pod specification using annotations to cover the virtualization specific properties. The issue is, that this does not scale, and will bloat the pod specification.

Thus, the next time you are encountering this topic, you might want to think of KubeVirt as the Kubernetes virtualization API.

[dummdida]

Virtual Machines in Kubernetes? How and what makes sense?

Happy new year.

I stopped by saying that Kubernetes can run containers on a cluster. This implies that it can perform some cluster operations (i.e. scheduling). And the question is if the cluster logic plus some virtualization logic can actually provide us virtualization functionality as we know it from oVirt.

Can it?

Maybe. At least there are a few approaches which already tried to run VMs within or on-top of Kubernetes.

Note. I'm happy to get input on clarifications for the following implementations.

Hyper created a fork to launch the container runtimes inside a VM:

docker-proxy | v [VM | docker-runtime] | + container + container :

runV is also from hyper. It is a OCI compatible container runtime. But instead of launching a container, this runtime will really launch a VM (libvirtd, qemu, …) with a given kernel, initrd and a given docker (or OCI) image.

This is pretty straight forward, thanks to the OCI standard.

frakti is actually a component implementing Kubernetes CRI (container runtime interface), and it can be used to run VM-isolated-containers in Kubernetes by using Hyper above.

rkt is actually a container runtime, but it supports to be run inside of KVM. To me this looks similar to runv, as a a VM is used for isolation purpose around a pod (not a single container).

host OS └─ rkt └─ hypervisor └─ kernel └─ systemd └─ chroot └─ user-app1

ClearContainers seem to be also much like runv and the alternative stage1 for rkt.

RancherVM is using a different approach - The VM is run inside the contianer, instead of wrapping it (like the approaches above). This means the container contains the VM runtime (qemu, libirtd, …). The VM can actually be directly adressed, because it's an explicit component.

host OS └─ docker └─ container └─ VM

This brings me to the wrap-up. Most of the solutions above use VMs as an isolation mechanism to containers. This happens transparently - as far as I can tell the VM is not directly exposed to higher levels, an dcan thus not be directly adressed in the sense of configured (i.e. adding a second display).

Except for the RancherVM solution where the VM is running inside a container. Her ethe VM is layered on-top, and is basically not hidden in the stack. By default the VM is inheriting stuff form the pod (i.e. networking, which is pretty incely solved), but it would also allow to do more with the VM.

So what is the take away? So, so, I would say. Looks like there is at least interest to somehow get VMs working for the one or the other use-case in the Kubernetes context. In most cases the Vm was hidden in the stack - this currently prevents to directly access and modify the VM, and it actually could imply that the VM is handled like a pod. Which actually means that the assumptions you have on a container will also apply to the VM. I.e. it's stateless, it can be killed, and reinstantiated. (This statement is pretty rough and hides a lot of details).

VM The issue is that we do care about VMs in oVirt, and that we love modifying them - like adding a second display, migrating them, tuning boot order and other fancy stuff. RancherVM looks to be going into a direction where we could tnue, but the others don't seem to help here.

Cluster Another question is: All the implementations above cared about running a VM, but oVirt is also caring about more, it's caring about cluster tasks - i.e. live migration, host fencing. And if the cluster tasks are on Kubernetes shoulders, then the question is: Does Kubernetes care about them as much as oVirt does? Maybe.

Conceptually Where do VMs belong? Above implementations hide the VM details (except RancherVM) - one reaosn is that Kubernetes does not care about this. Kubernetes does not have a concept for VMs- not for isolation and not as an explicit entity. And the questoin is: Should Kubernetes care? Kubernetes is great on Containers - and VMs (in the oVirt sense) are so much more. Is it worth to push all the needed knowledge into Kubernetes? And would this actually see acceptance from Kubernetes itself?

I tend to say No. The strength of Kubernetes is that it does one thing, and it does it well. Why should it get so bloated to expose all VM details?

But maybe it can learn to run VMs, and knows enough about them, to provifde a mechanism to pass through additional configuration to fine tune a VM.

Many open questions. But also a little more knowledge - and a post that got a little long.