July 08, 2021 - by Michael Zeller

Share:

For the last few years Triton has offered two types of HVM (hardware virtual machine) instances. We wanted to take the time to (re)introduce Triton users to what Bhyve is, why we ported it to SmartOS, some recent enhancements, and some future directions for the underlying technology. While Joyent was the first to introduce double-hulled-virtualization with the introduction of KVM on SmartOS, Bhyve is now the primary production hypervisor within Triton. Joyent also added HMA (hypervisor management API) to the platform which allows multiple hypervisors to exist on the same physical hardware. That means Triton supports KVM and Bhyve VMs running side by side on the same Compute Node.

Like KVM, Bhyve is a type-2 hypervisor that allows users to run self contained Operating Systems of their choice as a guest of the host OS. All HVM instances on Triton run inside of zones (containers) that allow Triton to enforce resource controls, and security constraints. This is what we mean when we say double-hulled-virtualization. If there is a bug in the underlying hypervisor like there has been in the past with KVM, the bad actor would find themselves within a zone in the event of a guest escape with very limited privileges. This makes Bhyve a great solution for multi-tenant container orchestration infrastructure like Docker or Kubernetes.

There are a number of reasons Joyent chose to bring Bhyve to the product. KVM on Triton is effectively forked since 2012 and has not received much attention outside of the occasional security fix. Licensing for KVM/Qemu also means that the code cannot be kept in upstream illumos-gate. This means the code lives in separate repositories and is included at build time in the Triton platform. Lastly, KVM on Triton has only ever been officially supported on Intel systems with EPT.

Bhyve originated from a team within NetApp, and was eventually contributed to FreeBSD in 2011. Bhyve features a much more focused scope that more closely aligns with what we want out of a hypervisor. It was intended to be a modern hypervisor with a focus on supporting virtual I/O (virtio) devices rather than emulating older legacy hardware that existing hypervisors like Qemu/KVM, VirtualBox, and VMware support. Bhyve was designed with hardware assisted virtualization in mind from the start (VT-x / AMD-V + Extended Page Tables). Since Bhyve is BSD licensed, the code is capable of living within illumos-gate itself and in fact has already been upstreamed.

While KVM was effectively forked for Triton, Bhyve has seen a lot of growth from the community and internally. Joyent attends bi-weekly Bhyve developer calls where other companies and communities gather to discuss design of the hypervisor as well as elicit code reviews for incoming changes. We continue to do upstream syncs of changes directly into illumos-gate which make their way into the Triton releases.

Bhyve currently supports AMD and Intel CPUs, unlike our legacy KVM port. There continues to be additional features added such as early support for PCI passthrough, flexible disk space for Bhyve VMs, enhanced privilege reduction, VNC improvements, and many more behind the scenes improvements. There are exciting improvements coming from the community such as 9pfs, and additional PCI passthrough support. The future for Bhyve looks bright with other companies such as Oxide working on an additional Rust userspace for illumos called Propolis. They are also working on live migration and improvements to how Bhyve allocates memory from the system.

Hopefully this provides some insight into why Bhyve was the better choice for Triton and why Joyent chose it to be the primary hypervisor for the platform over KVM. With contributions from Joyent, Oxide, OmniOSce, FreeBSD, and universities there’s a lot to get excited about as the community works together to drive the roadmap forward. We hope you decide to take Bhyve for a spin if you are a Triton user or even just a SmartOS user.