be536eaaf7
* fix(kubevip): reject out-of-range routing protocol values Netlink carries the address and route protocol in a single byte, so a configured value above 255 was silently truncated on the wire and never matched again on readback. Reject it during config validation instead. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * feat(wireguard): qualify service tunnel IDs by protocol Sanitisation maps '-' onto the '_' separator, so "a-b/c" and "a/b-c" shared one nftables chain, and TCP and UDP on the same port collided. ServicePortIDs appends the protocol and, when sanitisation changed the name or the ID grew too long, a hash of the raw name. It also returns the previous port-only ID so existing chains can be migrated. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * fix(arp): guard manager state behind a single mutex Instances were kept in a sync.Map with a per-instance mutex for the refcount, so lookup and refcount update were not atomic: concurrent Insert and Remove could resurrect a deleted instance or drop a live one. Hold one manager mutex across both, and buffer link subscriptions so a netlink sender is never parked on an unread channel during shutdown. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * fix(instance): roll back partially created instances Instance creation added addresses, VLAN or macvlan links and DHCP clients incrementally, so a failure part way through left the node holding state nobody owned. Initialization now unwinds what it created, and link cleanup only deletes attachments this instance created that no remaining instance still uses. Namespace-dependent tests now skip unless KUBE_VIP_REQUIRE_NETNS is set, which CI sets on the privileged job so lost capabilities turn it red instead of silently skipping. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * test(e2e): give docker kill more time under parallel load The ARP suite runs four kind clusters against one Docker daemon, so acknowledging a leader kill regularly exceeded the 5s budget and failed the IPv6 failover specs before any assertion ran. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * fix: regression on preserveOnLeadershipLoss Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * fix: use the introduced wireguard service_id Signed-off-by: Marcel Fest <marcel.fest@telekom.de> * fix(services): reuse link attachment ownership on service delete deleteService removed VLAN and macvlan links unconditionally, which tore down interfaces kube-vip had only adopted and interfaces another Service still used. Route the delete path through CleanupLinkAttachments and pass the remaining instances so ownership is handed over instead. Signed-off-by: Marcel Fest <marcel.fest@telekom.de> --------- Signed-off-by: Marcel Fest <marcel.fest@telekom.de>
kube-vip
High Availability and Load-Balancing
&message=212&color=0094FF&logo=linuxfoundation&logoColor=white&style=flat)
Overview
Kubernetes Virtual IP and Load-Balancer for both control plane and Kubernetes services
The idea behind kube-vip is a small self-contained Highly-Available option for all environments, especially:
- Bare-Metal
- Edge (arm / Raspberry PI)
- Virtualisation
- Pretty much anywhere else :)
NOTE: All documentation of both usage and architecture are now available at https://kube-vip.io.
For upgrading an existing install in place (static Pod or DaemonSet), see the upgrade guide.
Features
Kube-Vip was originally created to provide a HA solution for the Kubernetes control plane, over time it has evolved to incorporate that same functionality into Kubernetes service type load-balancers.
- VIP addresses can be both IPv4 or IPv6
- Control Plane with ARP (Layer 2) or BGP (Layer 3)
- Control Plane using either leader election or raft
- Control Plane HA with kubeadm (static Pods)
- Control Plane HA with K3s/and others (daemonsets)
- Service LoadBalancer using leader election for ARP (Layer 2)
- Service LoadBalancer using multiple nodes with BGP
- Service LoadBalancer address pools per namespace or global
- Service LoadBalancer address via (existing network DHCP)
- Service LoadBalancer address exposure to gateway via UPNP
- Egress! Kube-vip will utilise a service loadbalancer as both the ingress and egress for a pod.
- ... manifest generation, vendor API integrations and many more...
Why?
The purpose of kube-vip is to simplify the building of HA Kubernetes clusters, which at this time can involve a few components and configurations that all need to be managed. This was blogged about in detail by thebsdbox here -> https://thebsdbox.co.uk/2020/01/02/Designing-Building-HA-bare-metal-Kubernetes-cluster/#Networking-load-balancing.
Alternative HA Options
kube-vip provides both a floating or virtual IP address for your kubernetes cluster as well as load-balancing the incoming traffic to various control-plane replicas. At the current time to replicate this functionality a minimum of two pieces of tooling would be required:
VIP:
- Keepalived
- UCARP
- Hardware Load-balancer (functionality differs per vendor)
LoadBalancing:
All of these would require a separate level of configuration and in some infrastructures multiple teams in order to implement. Also when considering the software components, they may require packaging into containers or if they’re pre-packaged then security and transparency may be an issue. Finally, in edge environments we may have limited room for hardware (no HW load-balancer) or packages solutions in the correct architectures might not exist (e.g. ARM). Luckily with kube-vip being written in GO, it’s small(ish) and easy to build for multiple architectures, with the added security benefit of being the only thing needed in the container.
Troubleshooting and Feedback
SELinux and IPVS kernel modules
When using IPVS load balancing on nodes with SELinux enforcing, kube-vip may be
blocked from requesting kernel modules from inside the container. Symptoms can
include the kube-vip pod entering Error or CrashLoopBackOff, logs that show
ensure IPVS kernel modules are loaded, or audit denials for module_request
from container_t.
Load the required IPVS modules on every node that can run kube-vip before deploying it:
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
To persist this across reboots, add the modules to a file such as
/etc/modules-load.d/kube-vip-ipvs.conf:
ip_vs
ip_vs_rr
Preloading only the required modules is preferred to enabling the SELinux
domain_kernel_load_modules boolean for containers.
Gateway API LoadBalancer services with no endpoints
Some Gateway API controllers create LoadBalancer services that intentionally have no Endpoints/EndpointSlices backends.
If you want kube-vip to reconcile such a service, opt in with:
metadata:
annotations:
kube-vip.io/allow-reconcile-without-endpoints: "true"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
Scope:
- Works only with
externalTrafficPolicy: Cluster - No effect for
Local - Default endpoint-gated behavior remains unchanged for services without this annotation
Please raise issues on the GitHub repository and as mentioned check the documentation at https://kube-vip.io.
Community Tools
- KubeStellar Console — Guided kube-vip Install — A step-by-step guided installation experience for kube-vip with pre-flight checks, validation, troubleshooting, and rollback support.
Contributing
Thanks for taking the time to join our community and start contributing! We welcome pull requests. Feel free to dig through the issues and jump in.
⚠️ This project has issue compiling on MacOS, please compile it on linux distribution
Additionally it is now relatively easy and quick to develop with skaffold, and the skaffold.yaml exists within the root folder of the gir repository.
Set up a kind development environment
kind create cluster --config ./testing/kind.yamlkubectl apply -f https://kube-vip.io/manifests/rbac.yaml- Create a load balancer range
configMapfrom the kind cluster - Apply the CCM manifest
- Start skaffold with
skaffold dev - Start developing and see changes applied in real-time.
Star History
License
