fanhuadesenlinnn/kube-vip
1
0
Fork 0
mirror of https://hubproxy.babadafafafafa.cn/https://github.com/kube-vip/kube-vip.git synced 2026-09-20 08:03:47 +08:00
Code Issues Packages Projects Releases Wiki Activity

Extensive linting, corrections, expansions

Browse Source 
Signed-off-by: Chip Zoller <chipzoller@gmail.com>
This commit is contained in:
Chip Zoller
2021-11-21 19:23:00 -05:00
parent 88d6df058b
commit f5a2bd8e59
7 changed files with 271 additions and 305 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/architecture/index.md
View File

@@ -1,13 +1,13 @@
# **kube-vip** architecture
This section covers two parts of the architecture:
This section covers two parts of the architecture:
1. The technical capabilities of `kube-vip`
2. The components to build a load-balancing service within [Kubernetes](https://kubernetes.io)
The `kube-vip` project is designed to provide both a highly available networking endpoint and load-balancing functionality for underlying networking services. The project was originally designed for the purpose of providing a resilient control-plane for Kubernetes, it has since expanded to provide the same functionality for applications within a Kubernetes cluster.
Additionally `kube-vip` is designed to be lightweight and **multi-architecture**, all of the components are built for Linux but are also built for both `x86` and `armv7`,`armhvf`,`ppc64le`. This means that `kube-vip` will run fine in **bare-metal**, **virtual** and **edge** (raspberry pi or small arm SoC devices).
Additionally `kube-vip` is designed to be lightweight and **multi-architecture**, all of the components are built for Linux but are also built for both `x86` and `armv7`,`armhvf`,`ppc64le`. This means that `kube-vip` will run fine in **bare-metal**, **virtual** and **edge** (raspberry pi or small arm SoC devices).
## Technologies
@@ -29,7 +29,7 @@ When the **vip** moves from one host to another any host that has been using the
(Optional) The `kube-vip` can be configured to broadcast a [gratuitous arp](https://wiki.wireshark.org/Gratuitous_ARP) that will typically immediately notify all local hosts that the `vip <-> MAC` has changed.
**Below** we can see that the failover is typically done within a few seconds as the ARP broadcast is recieved.
**Below** we can see that the failover is typically done within a few seconds as the ARP broadcast is received.
```
64 bytes from 192.168.0.75: icmp_seq=146 ttl=64 time=0.258 ms
@@ -54,7 +54,7 @@ Request timeout for icmp_seq 150
### Load Balancing
Kube-Vip has the capability to provide a HA address for both the Kubernetes control plane and for a Kubernetes service, it recently implemented support for "actual" load-balancing for the control plane to distribute API requests across control-plane nodes.
Kube-Vip has the capability to provide a HA address for both the Kubernetes control plane and for a Kubernetes service, it recently implemented support for "actual" load-balancing for the control plane to distribute API requests across control-plane nodes.
#### Kubernetes Service Load-Balancing
@@ -86,7 +86,8 @@ To enable control-plane load balancing, the following is required in the kube-vi
- name : lb_enable
value: "true"
```
The load balancing is provided through IPVS (IP Virtual Server) and provides a layer-4 (TCP Port) based round-robin across all of the control plane nodes. By default the load balancer will listen on the default 6443 port as the Kubernetes API server.
The load balancing is provided through IPVS (IP Virtual Server) and provides a layer-4 (TCP Port) based round-robin across all of the control plane nodes. By default the load balancer will listen on the default 6443 port as the Kubernetes API server.
**Note:** The IPVS virtual server lives in kernel space and doesn't create an "actual" service that listens on port 6443, this allows the kernel to parse packets before they're sent to an actual TCP port. This is important to know because it means we don't have any port conflicts having the IPVS load-balancer listening on the same port as the API server on the same host.
The load balancer port can be customised with the following snippet in the yaml.
@@ -100,14 +101,14 @@ The load balancer port can be customised with the following snippet in the yaml.
Once the `lb_enable` is set to true kube-vip will do the following:
- In Layer 2 it will create an IPVS service on the leader
- In Layer 3 all nodes will create an IPVS service
- It will start a Kubernetes node watcher for nodes with the control plane label
- It will add/delete them as they're added and removed from the cluster
- In Layer 2 it will create an IPVS service on the leader
- In Layer 3 all nodes will create an IPVS service
- It will start a Kubernetes node watcher for nodes with the control plane label
- It will add/delete them as they're added and removed from the cluster
#### Debugging control plane load-balancing
In order to inspect what is happening we will need to install the `ipvsadm` tool.
In order to inspect what is happening we will need to install the `ipvsadm` tool.
##### View the configuration
@@ -155,4 +156,3 @@ The `kube-vip` kubernetes load-balancer requires a number of components in order
- The Kube-Vip Cloud Provider -> [https://github.com/kube-vip/kube-vip-cloud-provider](https://github.com/kube-vip/kube-vip-cloud-provider)
- The Kube-Vip Deployment -> [https://github.com/kube-vip/kube-vip](https://github.com/kube-vip/kube-vip)

170
docs/flags/index.md
View File

@@ -2,95 +2,95 @@
## Flags
These flags are typically used in manifest generation.
These flags are typically used in the `kube-vip` manifest generation process.
| Category | Flag | Usage | Notes |
|--------------|------|-------|-------|
|**Troubleshooting** ||||
| |`--log`|default 4|Set to `5` for debugging logs|
|**Mode** ||||
| |`--controlPlane`|Enables `kube-vip` control-plane functionality||
| |`--services`|Enables `kube-vip` to watch services of type:LoadBalancer||
|**Vip Config** ||||
| |`--arp`|Enables ARP brodcasts from Leader||
| |`--bgp`|Enables BGP peering from `kube-vip`||
| |`--vip`|`<IP Address>`|(deprecated)|
| |`--address`|`<IP Address>` or `<DNS name>`||
| |`--interface`|`<linux interface>`||
| |`--leaderElection`|Enables Kubernetes LeaderElection|Used by ARP, as only the leader can broadcast|
| |`--enableLoadBalancer`|Enables IPVS load balancer||
| |`--lbPort`|6443|The port that the api server will load-balanced on|
|**Services**||||
| |`--cidr`|Defaults "32"|Used when advertising BGP addresses (typically as `x.x.x.x/32`)|
|**Kubernetes**||||
| |`--inCluster`|Defaults to looking inside the Pod for the token||
| |`--taint`|Enables a taint, stopping control plane daemonset being on workers||
|**LeaderElection**||||
| |`--leaseDuration`|default 5|Seconds a lease is held for|
| |`--leaseRenewDuration`|default 3|Seconds a leader can attempt to renew the lease|
| |`--leaseRetry`|default 1|Number of times the leader will hold the lease for|
| |`--namespace`|"kube-vip"|The namespace where the lease will reside|
|**BGP**||||
| |`--bgpRouterID`|`<IP Address>`|Typically the address of the local node|
| |`--localAS`|default 65000|The AS we peer from|
| |`--bgppeers`|`<address:AS:password:mutlihop>`|Comma seperate list of BGP peers|
| |`--peerAddress`|`<IP Address>`|Address of a single BGP Peer|
| |`--peerAS`|default 65000|AS of a single BGP Peer|
| |`--peerPass`|""| Password to work with a single BGP Peer|
| |`--multiHop`|Enables eBGP MultiHop| Enable multiHop with a single BGP Peer|
| |`--sourceif`|Source Interface| Determines which interface BGP should peer _from_|
| |`--sourceip`|Source Address| Determines which IP address BGP should peer _from_|
| |`--annotaions`|`<provider string>`|Startup will be paused until the node annotaions contain the BGP configuration|
|**Equinix Metal**|||(May be deprecated)|
| |`--metal`|Enables Equinix Metal API calls||
| |`--metalKey`|Equinix Metal API token||
| |`--metalProject`|Equinix Metal Project (Name)||
| |`--metalProjectID`|Equinix Metal Project (UUID)||
| |`--provider-config`|Path to the Equinix Metal provider configuration|Requires the Equinix Metal CCM|
| Category | Flag | Usage | Notes |
| ------------------- | ---------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| **Troubleshooting** | | | |
| | `--log` | default 4 | Set to `5` for debugging logs |
| **Mode** | | | |
| | `--controlplane` | Enables `kube-vip` control plane functionality | |
| | `--services` | Enables `kube-vip` to watch services of type `LoadBalancer` | |
| **VIP Config** | | | |
| | `--arp` | Enables ARP broadcasts from Leader | |
| | `--bgp` | Enables BGP peering from `kube-vip` | |
| | `--vip` | `<IP Address>` | (deprecated) |
| | `--address` | `<IP Address>` or `<DNS name>` | |
| | `--interface` | `<linux interface>` | |
| | `--leaderElection` | Enables Kubernetes LeaderElection | Used by ARP, as only the leader can broadcast |
| | `--enableLoadBalancer` | Enables IPVS load balancer | |
| | `--lbPort` | 6443 | The port that the api server will load-balanced on |
| **Services** | | | |
| | `--cidr` | Defaults "32" | Used when advertising BGP addresses (typically as `x.x.x.x/32`) |
| **Kubernetes** | | | |
| | `--inCluster` | Defaults to looking inside the Pod for the token | |
| | `--taint` | Enables a taint, stopping control plane DaemonSet being on workers | |
| **LeaderElection** | | | |
| | `--leaseDuration` | default 5 | Seconds a lease is held for |
| | `--leaseRenewDuration` | default 3 | Seconds a leader can attempt to renew the lease |
| | `--leaseRetry` | default 1 | Number of times the leader will hold the lease for |
| | `--namespace` | "kube-vip" | The namespace where the lease will reside |
| **BGP** | | | |
| | `--bgpRouterID` | `<IP Address>` | Typically the address of the local node |
| | `--localAS` | default 65000 | The AS we peer from |
| | `--bgppeers` | `<address:AS:password:multihop>` | Comma separated list of BGP peers |
| | `--peerAddress` | `<IP Address>` | Address of a single BGP Peer |
| | `--peerAS` | default 65000 | AS of a single BGP Peer |
| | `--peerPass` | "" | Password to work with a single BGP Peer |
| | `--multiHop` | Enables eBGP MultiHop | Enable multiHop with a single BGP Peer |
| | `--sourceif` | Source Interface | Determines which interface BGP should peer _from_ |
| | `--sourceip` | Source Address | Determines which IP address BGP should peer _from_ |
| | `--annotations` | `<provider string>` | Startup will be paused until the node annotations contain the BGP configuration |
| **Equinix Metal** | | | (May be deprecated) |
| | `--metal` | Enables Equinix Metal API calls | |
| | `--metalKey` | Equinix Metal API token | |
| | `--metalProject` | Equinix Metal Project (Name) | |
| | `--metalProjectID` | Equinix Metal Project (UUID) | |
| | `--provider-config` | Path to the Equinix Metal provider configuration | Requires the Equinix Metal CCM |
## Environment Variables
These environment variables are usually part of a kube-vip manifest.
These environment variables are usually part of a `kube-vip` manifest and used when running the `kube-vip` Pod.
More environment variables can be read through the `pkg/kubevip/config_envvar.go` file.
| Category | Environment Variable | Usage | Notes |
|--------------|------|-------|-------|
|**Troubleshooting** ||||
| |`vip_loglevel`|default 4|Set to `5` for debugging logs|
|**Mode** ||||
| |`cp_enable`|Enables `kube-vip` control-plane functionality||
| |`svc_enable`|Enables `kube-vip` to watch services of `type:LoadBalancer`||
|**Vip Config** ||||
| |`vip_arp`|Enables ARP brodcasts from Leader||
| |`bgp_enable`|Enables BGP peering from `kube-vip`||
| |`vip_address`|`<IP Address>`|(deprecated)|
| |`address`|`<IP Address>` or `<DNS name>`||
| |`vip_interface`|`<linux interface>`||
| |`vip_leaderelection`|Enables Kubernetes LeaderElection|Used by ARP, as only the leader can broadcast|
| |`lb_enable`|Enables IPVS LoadBalancer|Will watch Kubernetes nodes and add them to the IPVS load-balancer|
| |`lb_port`|6443|The IPVS port that will be used to load-balance control plane requests|
|**Services**||||
| |`vip_cidr`|Defaults "32"|Used when advertising BGP addresses (typically as `x.x.x.x/32`)|
|**LeaderElection**||||
| |`vip_leaseduration`|default 5|Seconds a lease is held for|
| |`vip_renewdeadline`|default 3|Seconds a leader can attempt to renew the lease|
| |`vip_retryperiod`|default 1|Number of times the leader will hold the lease for|
| |`cp_namespace`|"kube-vip"|The namespace where the lease will reside|
|**BGP**||||
| |`bgp_routerid`|`<IP Address>`|Typically the address of the local node|
| |`bgp_as`|default 65000|The AS we peer from|
| |`bgp_peers`|`<address:AS:password:mutlihop>`|Comma seperate list of BGP peers|
| |`bgp_peeraddress`|`<IP Address>`|Address of a single BGP Peer|
| |`bgp_peeras`|default 65000|AS of a single BGP Peer|
| |`bgp_peerpass`|""| Password to work with a single BGP Peer|
| |`bgp_multihop`|Enables eBGP MultiHop| Enable multiHop with a single BGP Peer|
| |`bgp_sourceif`|Source Interface| Determines which interface BGP should peer _from_|
| |`bgp_sourceip`|Source Address| Determines which IP address BGP should peer _from_|
| |`annotaions`|`<provider string>`|Startup will be paused until the node annotaions contain the BGP configuration|
|**Equinix Metal**|||(May be deprecated)|
| |`vip_packet`|Enables Equinix Metal API calls||
| |`PACKET_AUTH_TOKEN`|Equinix Metal API token||
| |`vip_packetproject`|Equinix Metal Project (Name)||
| |`vip_packetprojectid`|Equinix Metal Project (UUID)||
| |`provider_config`|Path to the Equinix Metal provider configuration|Requires the Equinix Metal CCM|
| Category | Environment Variable | Usage | Notes |
| ------------------- | --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Troubleshooting** | | | |
| | `vip_loglevel` | default 4 | Set to `5` for debugging logs |
| **Mode** | | | |
| | `cp_enable` | Enables `kube-vip` control plane functionality | |
| | `svc_enable` | Enables `kube-vip` to watch Services of type `LoadBalancer` | |
| **VIP Config** | | | |
| | `vip_arp` | Enables ARP broadcasts from Leader | |
| | `bgp_enable` | Enables BGP peering from `kube-vip` | |
| | `vip_address` | `<IP Address>` | (deprecated) |
| | `address` | `<IP Address>` or `<DNS name>` | |
| | `vip_interface` | `<linux interface>` | |
| | `vip_leaderelection` | Enables Kubernetes LeaderElection | Used by ARP, as only the leader can broadcast |
| | `lb_enable` | Enables IPVS LoadBalancer | Will watch Kubernetes nodes and add them to the IPVS load-balancer |
| | `lb_port` | 6443 | The IPVS port that will be used to load-balance control plane requests |
| **Services** | | | |
| | `vip_cidr` | Defaults "32" | Used when advertising BGP addresses (typically as `x.x.x.x/32`) |
| **LeaderElection** | | | |
| | `vip_leaseduration` | default 5 | Seconds a lease is held for |
| | `vip_renewdeadline` | default 3 | Seconds a leader can attempt to renew the lease |
| | `vip_retryperiod` | default 1 | Number of times the leader will hold the lease for |
| | `cp_namespace` | "kube-vip" | The namespace where the lease will reside |
| **BGP** | | | |
| | `bgp_routerid` | `<IP Address>` | Typically the address of the local node |
| | `bgp_as` | default 65000 | The AS we peer from |
| | `bgp_peers` | `<address:AS:password:multihop>` | Comma separated list of BGP peers |
| | `bgp_peeraddress` | `<IP Address>` | Address of a single BGP Peer |
| | `bgp_peeras` | default 65000 | AS of a single BGP Peer |
| | `bgp_peerpass` | "" | Password to work with a single BGP Peer |
| | `bgp_multihop` | Enables eBGP MultiHop | Enable multiHop with a single BGP Peer |
| | `bgp_sourceif` | Source Interface | Determines which interface BGP should peer _from_ |
| | `bgp_sourceip` | Source Address | Determines which IP address BGP should peer _from_ |
| | `annotations` | `<provider string>` | Startup will be paused until the node annotations contain the BGP configuration |
| **Equinix Metal** | | | (May be deprecated) |
| | `vip_packet` | Enables Equinix Metal API calls | |
| | `PACKET_AUTH_TOKEN` | Equinix Metal API token | |
| | `vip_packetproject` | Equinix Metal Project (Name) | |
| | `vip_packetprojectid` | Equinix Metal Project (UUID) | |
| | `provider_config` | Path to the Equinix Metal provider configuration | Requires the Equinix Metal CCM |

99
docs/hybrid/index.md
View File

@@ -1,45 +1,46 @@
# Using `kube-vip` in Hybrid Mode
# Using kube-vip in Hybrid Mode
We can deploy kube-vip in two different methods, which completely depends on your use-case and method for installing Kubernetes:
- Static Pods (hybrid)
- Daemonset (hybrid, requires taint)
## **Prerequisites**
## Prerequisites
In order for `kube-vip` to be able to speak with the Kubernetes API server, we need to be able to resolve the hostname within the pod. In order to ensure this will work as expected the `/etc/hosts` file should have the `hostname` of the server within it. The `/etc/hosts` file is passed into the running container and will ensure that the pod isn't "confused" by any Kubernetes networking.
## Kubernetes Services (`type:LoadBalancer`)
To learn more about how `kube-vip` in hybrid works with the LoadBalancer services within a kubernetes cluster the documentation is [here](./services/). To get `kube-vip` deployed read on !
To learn more about how `kube-vip` in hybrid works with the LoadBalancer services within a kubernetes cluster the documentation is [here](./services/). To get `kube-vip` deployed read on!
## Static Pods
Static pods are a Kubernetes pod that is ran by the `kubelet` on a single node, and is **not** managed by the Kubernetes cluster itself. This means that whilst the pod can appear within Kubernetes it can't make use of a variety of kubernetes functionality (such as the kubernetes token or `configMaps`). The static pod approach is primarily required for [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/), this is due to the sequence of actions performed by `kubeadm`. Ideally we want `kube-vip` to be part of the kubernetes cluster, for various bits of functionality we also need `kube-vip` to provide a HA virtual IP as part of the installation.
Static pods are a Kubernetes pod that is ran by the `kubelet` on a single node, and is **not** managed by the Kubernetes cluster itself. This means that whilst the pod can appear within Kubernetes it can't make use of a variety of kubernetes functionality (such as the kubernetes token or `configMaps`). The static pod approach is primarily required for [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/), this is due to the sequence of actions performed by `kubeadm`. Ideally we want `kube-vip` to be part of the kubernetes cluster, for various bits of functionality we also need `kube-vip` to provide a HA virtual IP as part of the installation.
The sequence of events for this to work follows:
1. Generate a `kube-vip` manifest in the static pods manifest folder
2. Run `kubeadm init`, this generates the manifests for the control plane and wait to connect to the VIP
3. The `kubelet` will parse and execute all manifest, including the `kube-vip` manifest
4. `kube-vip` starts and advertises our VIP
5. The `kubeadm init` finishes succesfully.
5. The `kubeadm init` finishes successfully.
## Daemonset
Other Kubernetes distributions can bring up a Kubernetes cluster, without depending on a VIP (BUT they are configured to support one). A prime example of this would be k3s, that can be configured to start and also sign the certificates to allow incoming traffic to a virtual IP. Given we don't need the VIP to exist **before** the cluster, we can bring up the k3s node(s) and then add `kube-vip` as a daemonset for all control plane nodes.
# Deploying `kube-vip`
## Deploying `kube-vip`
The simplest method for generating the Kubernetes manifests is with `kube-vip` itself.. The subcommand `manifest pod|daemonset` can be used to generate specific types of Kubernetes manifests for use in a cluster. These subcommands can be configured with additional flags to enable/disable BGP/ARP/LeaderElection and a host of other options.
Both Examples will use the same Architecture:
#### Infrastructure architecture
## Infrastructure architecture
The infrastructure for our example HA Kubernetes cluster is as follows:
| Node | Address |
|----------------|------------|
| Node | Address |
| -------------- | --------- |
| VIP | 10.0.0.40 |
| controlPlane01 | 10.0.0.41 |
| controlPlane02 | 10.0.0.42 |
@@ -56,45 +57,45 @@ The details for creating a static pod are available [here](./static/)
When using `kube-vip` as a daemonset the details are available [here](./daemonset/)
# Kube-Vip flag reference
## Kube-Vip flag reference
| Category | Flag | Usage | Notes |
|--------------|------|-------|-------|
|**Mode** ||||
| |`--controlPlane`|Enables `kube-vip` control-plane functionality||
| |`--services`|Enables `kube-vip` to watch services of type:LoadBalancer||
|**Vip Config** ||||
| |`--arp`|Enables ARP brodcasts from Leader||
| |`--bgp`|Enables BGP peering from `kube-vip`||
| |`--vip`|`<IP Address>`|(deprecated)|
| |`--address`|`<IP Address>` or `<DNS name>`||
| |`--interface`|`<linux interface>`||
| |`--leaderElection`|Enables Kubernetes LeaderElection|Used by ARP, as only the leader can broadcast|
|**Services**||||
| |`--cidr`|Defaults "32"|Used when advertising BGP addresses (typically as `x.x.x.x/32`)|
|**Kubernetes**||||
| |`--inCluster`|Defaults to looking inside the Pod for the token||
| |`--taint`|Enables a taint, stopping control plane daemonset being on workers||
|**LeaderElection**||||
| |`--leaseDuration`|default 5|Seconds a lease is held for|
| |`--leaseRenewDuration`|default 3|Seconds a leader can attempt to renew the lease|
| |`--leaseRetry`|default 1|Number of times the leader will hold the lease for|
| |`--namespace`|"kube-vip"|The namespace where the lease will reside|
|**BGP**||||
| |`--bgpRouterID`|`<IP Address>`|Typically the address of the local node|
| |`--localAS`|default 65000|The AS we peer from|
| |`--bgppeers`|`<address:AS:password:mutlihop>`|Comma seperate list of BGP peers|
| |`--peerAddress`|`<IP Address>`|Address of a single BGP Peer|
| |`--peerAS`|default 65000|AS of a single BGP Peer|
| |`--peerPass`|""| Password to work with a single BGP Peer|
| |`--multiHop`|Enables eBGP MultiHop| Enable multiHop with a single BGP Peer|
| |`--annotaions`|`<provider string>`|Startup will be paused until the node annotaions contain the BGP configuration|
|**Equinix Metal**|||(May be deprecated)|
| |`--metal`|Enables Equinix Metal API calls||
| |`--metalKey`|Equinix Metal API token||
| |`--metalProject`|Equinix Metal Project (Name)||
| |`--metalProjectID`|Equinix Metal Project (UUID)||
| |`--provider-config`|Path to the Equinix Metal provider configuration|Requires the Equinix Metal CCM|
| Category | Flag | Usage | Notes |
| ------------------ | ---------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| **Mode** | | | |
| | `--controlPlane` | Enables `kube-vip` control-plane functionality | |
| | `--services` | Enables `kube-vip` to watch services of type:LoadBalancer | |
| **Vip Config** | | | |
| | `--arp` | Enables ARP broadcasts from Leader | |
| | `--bgp` | Enables BGP peering from `kube-vip` | |
| | `--vip` | `<IP Address>` | (deprecated) |
| | `--address` | `<IP Address>` or `<DNS name>` | |
| | `--interface` | `<linux interface>` | |
| | `--leaderElection` | Enables Kubernetes LeaderElection | Used by ARP, as only the leader can broadcast |
| **Services** | | | |
| | `--cidr` | Defaults "32" | Used when advertising BGP addresses (typically as `x.x.x.x/32`) |
| **Kubernetes** | | | |
| | `--inCluster` | Defaults to looking inside the Pod for the token | |
| | `--taint` | Enables a taint, stopping control plane daemonset being on workers | |
| **LeaderElection** | | | |
| | `--leaseDuration` | default 5 | Seconds a lease is held for |
| | `--leaseRenewDuration` | default 3 | Seconds a leader can attempt to renew the lease |
| | `--leaseRetry` | default 1 | Number of times the leader will hold the lease for |
| | `--namespace` | "kube-vip" | The namespace where the lease will reside |
| **BGP** | | | |
| | `--bgpRouterID` | `<IP Address>` | Typically the address of the local node |
| | `--localAS` | default 65000 | The AS we peer from |
| | `--bgppeers` | `<address:AS:password:multihop>` | Comma separated list of BGP peers |
| | `--peerAddress` | `<IP Address>` | Address of a single BGP Peer |
| | `--peerAS` | default 65000 | AS of a single BGP Peer |
| | `--peerPass` | "" | Password to work with a single BGP Peer |
| | `--multiHop` | Enables eBGP MultiHop | Enable multiHop with a single BGP Peer |
| | `--annotations` | `<provider string>` | Startup will be paused until the node annotations contain the BGP configuration |
| **Equinix Metal** | | | (May be deprecated) |
| | `--metal` | Enables Equinix Metal API calls | |
| | `--metalKey` | Equinix Metal API token | |
| | `--metalProject` | Equinix Metal Project (Name) | |
| | `--metalProjectID` | Equinix Metal Project (UUID) | |
| | `--provider-config` | Path to the Equinix Metal provider configuration | Requires the Equinix Metal CCM |
## Changelog
@@ -112,7 +113,7 @@ Once DHCP returns an IP for the FQDN, the same `dnsUpdater` runs to periodically
## BGP Support (added in 0.1.8)
In version `0.1.8` `kube-vip` was updated to support [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol) as a VIP failover mechanism. When a node is elected as a leader then it will update it's peers so that they are aware to route traffic to that node in order to access the VIP.
In version `0.1.8` `kube-vip` was updated to support [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol) as a VIP failover mechanism. When a node is elected as a leader then it will update it's peers so that they are aware to route traffic to that node in order to access the VIP.
The following new flags are used:
@@ -128,7 +129,7 @@ If the `--bgp` flag is passed along with the Equinix Metal flags `metal, metalKe
## Equinix Metal Control Plane Support (added in 0.1.8)
Recently in version `0.1.7` of `kube-vip` we added the functionality to use a Equinix Metal Elastic IP as the virtual IP fronting the Kubernetes Control plane cluster. In order to first get out virtual IP we will need to use our Equinix Metal account and create a EIP (either public or private). We will only need a single address so a `/32` will suffice, once this is created as part of a Equinix Metal project we can now apply this address to the servers that live in the same project.
Recently in version `0.1.7` of `kube-vip` we added the functionality to use a Equinix Metal Elastic IP as the virtual IP fronting the Kubernetes Control plane cluster. In order to first get out virtual IP we will need to use our Equinix Metal account and create a EIP (either public or private). We will only need a single address so a `/32` will suffice, once this is created as part of a Equinix Metal project we can now apply this address to the servers that live in the same project.
In this example we've logged into the UI can created a new EIP of `147.75.1.2`, and we've deployed three small server instances with Ubuntu.

37
docs/index.md
View File

@@ -1,45 +1,45 @@
![kube-vip.png](kube-vip.png)
## Overview
Kubernetes Virtual IP and Load-Balancer for both control plane and Kubernetes services
The idea behind `kube-vip` is a small self-contained Highly-Available option for all environments, especially:
Kubernetes Virtual IP and Load-Balancer for both control plane and Kubernetes services.
The idea behind `kube-vip` is a small, self-contained, highly-available option for all environments, especially:
- Bare-Metal
- On-Prem
- Edge (ARM / Raspberry PI)
- Edge (ARM / Raspberry Pi)
- Virtualisation
- Pretty much anywhere else :)
## Features
Kube-Vip was originally created to provide a HA solution for the Kubernetes control plane, over time it has evolved to incorporate that same functionality into Kubernetes service type [load-balancers](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer).
Kube-Vip was originally created to provide a HA solution for the Kubernetes control plane, but over time it has evolved to incorporate that same functionality for Kubernetes Services of type [LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer).
- VIP addresses can be both IPv4 or IPv6
- Control Plane with ARP (Layer 2) or BGP (Layer 3)
- Control Plane using either [leader election](https://godoc.org/k8s.io/client-go/tools/leaderelection) or [raft](https://en.wikipedia.org/wiki/Raft_(computer_science))
- Control Plane HA with kubeadm (static Pods)
- Control Plane HA with K3s/and others (daemonsets)
- Control Plane LoadBalancing with IPVS (kube-vip > 0.4)
- Control Plane HA with K3s/and others (DaemonSets)
- Control Plane LoadBalancing with IPVS (kube-vip ≥ 0.4)
- Service LoadBalancer using [leader election](https://godoc.org/k8s.io/client-go/tools/leaderelection) for ARP (Layer 2)
- Service LoadBalancer using multiple nodes with BGP
- Service LoadBalancer address pools per namespace or global
- Service LoadBalancer address via (existing network DHCP)
- Service LoadBalancer address exposure to gateway via UPNP
- ... manifest generation, vendor API integrations and many nore...
- Service LoadBalancer address exposure to gateway via UPnP
- ... manifest generation, vendor API integrations and many more...
## Why?
The "original" purpose of `kube-vip` was to simplify the building of HA Kubernetes clusters, which at this time can involve a few components and configurations that all need to be managed. This was blogged about in detail by [thebsdbox](https://twitter.com/thebsdbox/) here -> [https://thebsdbox.co.uk/2020/01/02/Designing-Building-HA-bare-metal-Kubernetes-cluster/#Networking-load-balancing](https://thebsdbox.co.uk/2020/01/02/Designing-Building-HA-bare-metal-Kubernetes-cluster/#Networking-load-balancing). As the project evolved it now can use those same technologies to provide load-balancing capabilities within a Kubernetes Cluster.
The "original" purpose of `kube-vip` was to simplify the building of HA Kubernetes clusters, which at the time involved a few components and configurations that all needed to be managed. This was blogged about in detail by [thebsdbox](https://twitter.com/thebsdbox/) [here](https://thebsdbox.co.uk/2020/01/02/Designing-Building-HA-bare-metal-Kubernetes-cluster/#Networking-load-balancing). Since the project has evolved, it can now use those same technologies to provide load balancing capabilities within a Kubernetes Cluster.
## Architecture
The architecture for `kube-vip` (and associated kubernetes components) is covered in detail [here](/architecture/)
The architecture for `kube-vip` (and associated Kubernetes components) is covered in detail [here](/architecture/).
## Installation
There are two main routes for deploying `kube-vip`, either through a [static pod](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) when bringing up a Kubernetes cluster with [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) or as a [daemon set](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) (typically with distributions like [k3s](https://k3s.io)).
There are two main routes for deploying `kube-vip`: either through a [static pod](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) when bringing up a Kubernetes cluster with [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) or as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) (typically with distributions like [K3s](https://k3s.io)).
The infrastructure for our example HA Kubernetes cluster is as follows:
@@ -51,10 +51,10 @@ The infrastructure for our example HA Kubernetes cluster is as follows:
| controlPlane03 | 10.0.0.43 |
| worker01 | 10.0.0.44 |
All nodes are running Ubuntu 20.04, Docker CE and will use Kubernetes 1.21.0, we only have one worker as we're going to use our controlPlanes in "hybrid" mode.
All nodes are running Ubuntu 20.04, Docker CE and will use Kubernetes 1.21.0. We only have one worker as we're going to use our control plane in "hybrid" mode.
- [Static Pod](/install_static)
- [Daemon Set](/install_daemonset)
- [DaemonSet](/install_daemonset)
## Usage
@@ -69,12 +69,13 @@ All nodes are running Ubuntu 20.04, Docker CE and will use Kubernetes 1.21.0, we
## Links
- The Kube-Vip Cloud Provider Repository -> [https://github.com/kube-vip/kube-vip-cloud-provider](https://github.com/kube-vip/kube-vip-cloud-provider)
- The Kube-Vip Repository -> [https://github.com/kube-vip/kube-vip](https://github.com/kube-vip/kube-vip)
- The Kube-Vip RBAC (required for the daemonset) -> [https://kube-vip.io/manifests/rbac.yaml](https://kube-vip.io/manifests/rbac.yaml)
- [Kube-Vip Cloud Provider Repository](https://github.com/kube-vip/kube-vip-cloud-provider)
- [Kube-Vip Repository](https://github.com/kube-vip/kube-vip)
- [Kube-Vip RBAC manifest (required for the DaemonSet)](https://kube-vip.io/manifests/rbac.yaml)
## Copyright
© 2021 [The Linux Foundation](https://www.linuxfoundation.org/). All right reserved
© 2021 [The Linux Foundation](https://www.linuxfoundation.org/). All rights reserved.
The Linux Foundation has registered trademarks and uses trademarks.

20
docs/install_daemonset/index.md
View File

@@ -1,18 +1,18 @@
# Kube-Vip as a daemonset
# Kube-Vip as a DaemonSet
## Daemonset
Other Kubernetes distributions can bring up a Kubernetes cluster, without depending on a VIP (BUT they are configured to support one). A prime example of this would be k3s, that can be configured to start and also sign the certificates to allow incoming traffic to a virtual IP. Given we don't need the VIP to exist **before** the cluster, we can bring up the k3s node(s) and then add `kube-vip` as a daemonset for all control plane nodes.
Other Kubernetes distributions can bring up a Kubernetes cluster, without depending on a VIP (BUT they are configured to support one). A prime example of this would be k3s, that can be configured to start and also sign the certificates to allow incoming traffic to a virtual IP. Given we don't need the VIP to exist **before** the cluster, we can bring up the k3s node(s) and then add `kube-vip` as a DaemonSet for all control plane nodes.
If the Kubernetes installer allows for adding a Virtual IP as an additional [SAN](https://en.wikipedia.org/wiki/Subject_Alternative_Name) to the API server certificate then we can apply `kube-vip` to the cluster once the first node has been brought up.
If the Kubernetes installer allows for adding a Virtual IP as an additional [SAN](https://en.wikipedia.org/wiki/Subject_Alternative_Name) to the API server certificate then we can apply `kube-vip` to the cluster once the first node has been brought up.
## Kube-Vip as **HA**, **Load-Balancer** or both ` ¯\_(ツ)_/¯`
## Kube-Vip as HA, Load Balancer, or both
When generating a manifest for `kube-vip` we will pass in the flags `--controlplane` / `--services` these will enable the various types of functionality within `kube-vip`.
When generating a manifest for `kube-vip` we will pass in the flags `--controlplane` / `--services` these will enable the various types of functionality within `kube-vip`.
With both enabled `kube-vip` will manage a virtual IP address that is passed through it's configuration for a Highly Available Kubernetes cluster, it will also "watch" services of `type:LoadBalancer` and once their `spec.LoadBalancerIP` is updated (typically by a cloud controller) it will advertise this address using BGP/ARP.
**Note about Daemonsets**
**Note about DaemonSets**
Unlike generating the static manifest there are a few more things that may need configuring, this page will cover most scenarios.
@@ -28,7 +28,7 @@ kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
This section only covers generating a simple *BGP* configuration, as the main focus is will be on additional changes to the manifest. For more examples we can look at [here](/hybrid/static/).
**Note:** Pay attention if using the "static" examples, as the `manifest` subcommand should use `daemonset` and NOT `pod`.
**Note:** Pay attention if using the "static" examples, as the `manifest` subcommand should use `DaemonSet` and NOT `pod`.
### Set configuration details
@@ -38,7 +38,7 @@ This section only covers generating a simple *BGP* configuration, as the main fo
### Configure to use a container runtime
#### Get latest version
#### Get latest version
We can parse the GitHub API to find the latest version (or we can set this manually)
@@ -58,9 +58,9 @@ The easiest method to generate a manifest is using the container itself, below w
### BGP Example
This configuration will create a manifest that will start `kube-vip` providing **controlplane** and **services** management. **Unlike** ARP, all nodes in the BGP configuration will advertise virtual IP addresses.
This configuration will create a manifest that will start `kube-vip` providing **controlplane** and **services** management. **Unlike** ARP, all nodes in the BGP configuration will advertise virtual IP addresses.
**Note** we bind the address to `lo` as we don't want multiple devices that have the same address on public interfaces. We can specify all the peers in a comma seperate list in the format of `address:AS:password:multihop`.
**Note** we bind the address to `lo` as we don't want multiple devices that have the same address on public interfaces. We can specify all the peers in a comma separate list in the format of `address:AS:password:multihop`.
**Note 2** we pass the `--inCluster` flag as this is running as a daemonSet within the Kubernetes cluster and therefore will have access to the token inside the running pod.

76
docs/install_static/index.md
View File

@@ -1,57 +1,67 @@
# Kube-vip as a Static Pod
# Kube-Vip as a Static Pod
## Static Pods
Static pods are a Kubernetes pod that is ran by the `kubelet` on a single node, and is **not** managed by the Kubernetes cluster itself. This means that whilst the pod can appear within Kubernetes it can't make use of a variety of kubernetes functionality (such as the kubernetes token or `configMaps`). The static pod approach is primarily required for [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/), this is due to the sequence of actions performed by `kubeadm`. Ideally we want `kube-vip` to be part of the kubernetes cluster, for various bits of functionality we also need `kube-vip` to provide a HA virtual IP as part of the installation.
[Static Pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) are Kubernetes Pods that are run by the `kubelet` on a single node and are not managed by the Kubernetes cluster itself. This means that whilst the Pod can appear within Kubernetes, it can't make use of a variety of Kubernetes functionality (such as the Kubernetes token or ConfigMap resources). The static Pod approach is primarily required for [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) as this is due to the sequence of actions performed by `kubeadm`. Ideally, we want `kube-vip` to be part of the Kubernetes cluster, but for various bits of functionality we also need `kube-vip` to provide a HA virtual IP as part of the installation.
The sequence of events for this to work follows:
1. Generate a `kube-vip` manifest in the static pods manifest folder
2. Run `kubeadm init`, this generates the manifests for the control plane and wait to connect to the VIP
3. The `kubelet` will parse and execute all manifest, including the `kube-vip` manifest
4. `kube-vip` starts and advertises our VIP
5. The `kubeadm init` finishes succesfully.
The sequence of events for building a highly available Kubernetes cluster with `kubeadm` and `kube-vip` are as follows:
## Kube-Vip as **HA**, **Load-Balancer** or both ` ¯\_(ツ)_/¯`
1. Generate a `kube-vip` manifest in the static Pods manifest directory (see the [generating a manifest](#generating-a-manifest) section below).
2. Run `kubeadm init` with the `--control-plane-endpoint` flag using the VIP address provided when generating the static Pod manifest.
3. The `kubelet` will parse and execute all manifests, including the `kube-vip` manifest generated in step one and the other control plane components including `kube-apiserver`.
4. `kube-vip` starts and advertises the VIP address.
5. The `kubelet` on this first control plane will connect to the VIP advertised in the previous step.
6. `kubeadm init` finishes successfully on the first control plane.
7. Using the output from the `kubeadm init` command on the first control plane, run the `kubeadm join` command on the remainder of the control planes.
8. Copy the generated `kube-vip` manifest to the remainder of the control planes and place in their static Pods manifest directory (default of `/etc/kubernetes/manifests/`).
When generating a manifest for `kube-vip` we will pass in the flags `--controlplane` / `--services` these will enable the various types of functionality within `kube-vip`.
## Kube-Vip as HA, Load Balancer, or both
With both enabled `kube-vip` will manage a virtual IP address that is passed through it's configuration for a Highly Available Kubernetes cluster, it will also "watch" services of `type:LoadBalancer` and once their `spec.LoadBalancerIP` is updated (typically by a cloud controller) it will advertise this address using BGP/ARP.
The functionality of `kube-vip` depends on the flags used to create the static Pod manifest. By passing in `--controlplane` we instruct `kube-vip` to provide and advertise a virtual IP to be used by the control plane. By passing in `--services` we tell `kube-vip` to provide load balancing for Kubernetes Service resources created inside the cluster. With both enabled, `kube-vip` will manage a virtual IP address that is passed through its configuration for a highly available Kubernetes cluster. It will also watch Services of type `LoadBalancer` and once their `spec.LoadBalancerIP` is updated (typically by a cloud controller, including (optionally) the one provided by kube-vip in [on-prem](/usage/on-prem) scenarios) it will advertise this address using BGP/ARP. In this example, we will use both when generating the manifest.
## Generating a Manifest
This section details creating a number of manifests for various use cases
In order to create an easier experience of consuming the various functionality within `kube-vip`, we can use the `kube-vip` container itself to generate our static Pod manifest. We do this by running the `kube-vip` image as a container and passing in the various [flags](/flags/) for the capabilities we want to enable.
### Set configuration details
We use environment variables to predefine the values of the inputs to supply to `kube-vip`.
Set the `VIP` address to be used for the control plane:
`export VIP=192.168.0.40`
`export INTERFACE=<interface>`
Set the `INTERFACE` name to the name of the interface on the control plane(s) which will announce the VIP. In many Linux distributions this can be found with the `ip a` command.
## Configure to use a container runtime
`export INTERFACE=ens160`
### Get latest version
We can parse the GitHub API to find the latest version (or we can set this manually)
Get the latest version of the `kube-vip` release by parsing the GitHub API. This step requires that `jq` and `curl` are installed.
`KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")`
or manually:
To set manually instead, find the desired [release tag](https://github.com/kube-vip/kube-vip/releases):
`export KVVERSION=vx.x.x`
`export KVVERSION=v0.4.0`
The easiest method to generate a manifest is using the container itself, below will create an alias for different container runtimes.
### Creating the manifest
With the input values now set, we can pull and run the `kube-vip` image supplying it the desired flags and values. Once the static Pod manifest is generated for your desired method (ARP or BGP), if running multiple control plane nodes, ensure it is placed in each control plane's static manifest directory (by default, `/etc/kubernetes/manifests`).
Depending on the container runtime, use one of the two aliased commands to create a `kube-vip` command which runs the `kube-vip` image as a container.
For containerd, run the below command:
### containerd
`alias kube-vip="ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"`
### Docker
For Docker, run the below command:
`alias kube-vip="docker run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"`
## ARP
### ARP
This configuration will create a manifest that starts `kube-vip` providing **controlplane** and **services** management, using **leaderElection**. When this instance is elected as the leader it will bind the `vip` to the specified `interface`, this is also the same for services of `type:LoadBalancer`.
With the inputs and alias command set, we can run the `kube-vip` container to generate a static Pod manifest which will be directed to a file at `/etc/kubernetes/manifests/kube-vip.yaml`. As such, this is assumed to run on the first control plane node.
`export INTERFACE=eth0`
This configuration will create a manifest that starts `kube-vip` providing control plane VIP and Kubernetes Service management using the `leaderElection` method and ARP. When this instance is elected as the leader, it will bind the `vip` to the specified `interface`. This is the same behavior for Services of type `LoadBalancer`.
```
kube-vip manifest pod \
@@ -60,10 +70,10 @@ kube-vip manifest pod \
--controlplane \
--services \
--arp \
--leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml
--leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml
```
### Example manifest
#### Example ARP Manifest
```
apiVersion: v1
@@ -128,11 +138,11 @@ spec:
status: {}
```
## BGP
### BGP
This configuration will create a manifest that will start `kube-vip` providing **controlplane** and **services** management. **Unlike** ARP, all nodes in the BGP configuration will advertise virtual IP addresses.
This configuration will create a manifest that starts `kube-vip` providing control plane VIP and Kubernetes Service management. Unlike ARP, all nodes in the BGP configuration will advertise virtual IP addresses.
**Note** we bind the address to `lo` as we don't want multiple devices that have the same address on public interfaces. We can specify all the peers in a comma seperate list in the format of `address:AS:password:multihop`.
**Note** we bind the address to `lo` as we don't want multiple devices that have the same address on public interfaces. We can specify all the peers in a comma-separated list in the format of `address:AS:password:multihop`.
`export INTERFACE=lo`
@@ -145,10 +155,10 @@ kube-vip manifest pod \
--bgp \
--localAS 65000 \
--bgpRouterID 192.168.0.2 \
--bgppeers 192.168.0.10:65000::false,192.168.0.11:65000::false | tee /etc/kubernetes/manifests/kube-vip.yaml
--bgppeers 192.168.0.10:65000::false,192.168.0.11:65000::false | tee /etc/kubernetes/manifests/kube-vip.yaml
```
### Example Manifest
#### Example BGP Manifest
```
apiVersion: v1
@@ -213,4 +223,4 @@ spec:
path: /etc/kubernetes/admin.conf
name: kubeconfig
status: {}
```
```

152
docs/usage/on-prem/index.md
View File

@@ -1,123 +1,81 @@
# Kube-vip on-prem
# Kube-Vip On-Prem
We've designed `kube-vip` to be as de-coupled or agnostic from other components that may exist within a Kubernetes cluster as possible. This has lead to `kube-vip` having a very simplistic but robust approach to advertising Kubernetes services to the outside world and marking these services as ready to use.
We've designed `kube-vip` to be as decoupled or agnostic from other components that may exist within a Kubernetes cluster as possible. This has lead to `kube-vip` having a very simplistic but robust approach to advertising Kubernetes Services to the outside world and marking these Services as ready to use.
## CCM
## Cloud Controller Manager
We can see from the [flow](#Flow) above that `kube-vip` isn't coupled to anything other than the Kubernetes API, and will only act upon an existing Kubernetes primative (in this case the object of type `Service`). This makes it easy for existing CCMs to simply apply their logic to services of type LoadBalancer and leave `kube-vip` to take the next steps to advertise these load-balancers to the outside world.
`kube-vip` isn't coupled to anything other than the Kubernetes API and will only act upon an existing Kubernetes primitive (in this case the object of type `Service`). This makes it easy for existing [cloud controller managers (CCMs)](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) to simply apply their logic to services of type LoadBalancer and leave `kube-vip` to take the next steps to advertise these load balancers to the outside world.
## Using the Kube-Vip Cloud Provider
## Using the Kube-vip Cloud Provider
The `kube-vip` cloud provider can be used to populate an IP address for Services of type `LoadBalancer` similar to what public cloud providers allow through a Kubernetes CCM. The below instructions *should just work* on Kubernetes regardless of the architecture (a Linux OS being the only requirement) and will install the latest components.
The below instructions *should just work* on Kubernetes regardless of architecture (Linux Operating System is the only requirement) - you can quickly install the "latest" components:
## Install the Kube-Vip Cloud Provider
### Create the RBAC settings
As a daemonSet runs within the Kubernetes cluster it needs the correct access to be able to watch Kubernetes services and other objects. In order to do this we create a User, Role, and a binding.. we can apply this with the command:
The `kube-vip` cloud provider can be installed from the latest release in the `main` branch by using the following command:
```
kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml
```
### Install the `kube-vip-cloud-provider`
## Create a global CIDR or IP Range
In order for `kube-vip` to set an IP address for a Service of type `LoadBalancer`, it needs to have an availability of IP address to assign. This information is stored in a Kubernetes ConfigMap to which `kube-vip` has access. You control the scope of the IP allocations with the `key` within the ConfigMap. Either CIDR blocks or IP ranges may be specified and scoped either globally (cluster-side) or per-Namespace.
To allow a global (cluster-wide) CIDR block which `kube-vip` can use to allocate an IP to Services of type `LoadBalancer` in any Namespace, create a ConfigMap named `kubevip` with the key `cidr-global` and value equal to a CIDR block available in your environment. For example, the below command creates a global CIDR with value `192.168.0.220/29` from which `kube-vip` will allocate IP addresses.
```
$ kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml
kubectl create configmap -n kube-system kubevip --from-literal cidr-global=192.168.0.220/29
```
It uses a `statefulSet` and can always be viewed with the following command:
To use a global range instead, create the key `range-global` with the value set to a valid range of IP addresses. For example, the below command creates a global range using the pool `192.168.1.220-192.168.1.230`.
```
kubectl describe pods -n kube-system kube-vip-cloud-provider-0
kubectl create configmap -n kube-system kubevip --from-literal range-global=192.168.1.220-192.168.1.230
```
**Create a global CIDR or IP Range**
Creating services of type `LoadBalancer` in any Namespace will now take addresses from one of the global pools defined in the ConfigMap unless a Namespace-specific pool is created.
Any `service` in any `namespace` can use an address from the global CIDR `cidr-global` or range `range-global`
### The Kube-Vip Cloud Provider ConfigMap
```
kubectl create configmap --namespace kube-system kubevip --from-literal cidr-global=192.168.0.220/29
```
or
```
kubectl create configmap --namespace kube-system kubevip --from-literal range-global=192.168.1.220-192.168.1.230
```
To manage the IP address ranges for Services of type `LoadBalancer`, the `kube-vip-cloud-provider` uses a ConfigMap held in the `kube-system` Namespace. IP addresses can be configured using one or multiple formats:
Creating services of `type: LoadBalancer` in *any namespace* will now take addresses from the **global** cidr defined in the `configmap` unless a specific
## The Detailed guide
### Create the RBAC settings
As a daemonSet runs within the Kubernetes cluster it needs the correct access to be able to watch Kubernetes services and other objects. In order to do this we create a User, Role, and a binding.. we can apply this with the command:
```
kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
```
### Install the `kube-vip-cloud-provider`
```
$ kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml
```
The following output should appear when the manifest is applied:
```
serviceaccount/kube-vip-cloud-controller created
clusterrole.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-binding created
statefulset.apps/kube-vip-cloud-provider created
```
We can validate the cloud provider by examining the pods and following the logs:
```
kubectl describe pods -n kube-system kube-vip-cloud-provider-0
kubectl logs -n kube-system kube-vip-cloud-provider-0 -f
```
### The Kube-vip Cloud Provider `configmap`
To manage the IP address ranges for the load balancer instances the `kube-vip-cloud-provider` uses a `configmap` held in the `kube-system` namespace. IP address ranges can be configured using:
- IP address pools by CIDR
- CIDR blocks
- IP ranges [start address - end address]
- Multiple pools by CIDR per namespace
- Multiple IP ranges per namespace (handles overlapping ranges)
- Setting of static addresses through --load-balancer-ip=x.x.x.x
- Multiple pools by CIDR per Namespace
- Multiple IP ranges per Namespace (handles overlapping ranges)
- Setting of static addresses through --load-balancer-ip=x.x.x.x (`kubectl expose` command)
To control which IP address range is used for which service the following rules are applied:
- Global address pools (`cidr-global` or `range-global`) are available for use by *any* `service` in *any* `namespace`
- Namespace specific address pools (`cidr-<namespace>` or `range-<namespace>`) are *only* available for use by `service` in the *specific* `namespace`
- Static IP addresses can be applied to a load balancer `service` using the `loadbalancerIP` setting, even outside of the assigned ranges
To control which IP address range is used for which Service, the following rules are applied:
- Global address pools (`cidr-global` or `range-global`) are available for use by *any* Service in *any* Namespace
- Namespace specific address pools (`cidr-<namespace>` or `range-<namespace>`) are *only* available for use by a Service in the *specific* Namespace
- Static IP addresses can be applied to a Service of type `LoadBalancer` using the `spec.loadBalancerIP` field, even outside of the assigned ranges
Example Configmap:
```
$ kubectl get configmap -n kube-system kubevip -o yaml
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kubevip
namespace: kube-system
data:
cidr-default: 192.168.0.200/29 # CIDR-based IP range for use in the default namespace
range-development: 192.168.0.210-192.168.0.219 # Range-based IP range for use in the development namespace
cidr-finance: 192.168.0.220/29,192.168.0.230/29 # Multiple CIDR-based ranges for use in the finance namespace
cidr-global: 192.168.0.240/29 # CIDR-based range which can be used in any namespace
cidr-default: 192.168.0.200/29 # CIDR-based IP range for use in the default Namespace
range-development: 192.168.0.210-192.168.0.219 # Range-based IP range for use in the development Namespace
cidr-finance: 192.168.0.220/29,192.168.0.230/29 # Multiple CIDR-based ranges for use in the finance Namespace
cidr-global: 192.168.0.240/29 # CIDR-based range which can be used in any Namespace
```
### Expose a service
### Expose a Service
We can now expose a service and once the cloud provider has provided an address `kube-vip` will start to advertise that address to the outside world as shown below!
We can now expose a Service and once the cloud provider has provided an address, `kube-vip` will start to advertise that address to the outside world as shown below:
```
kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx
```
or via a `service` YAML definition
or via a Service YAML definition:
```
apiVersion: v1
@@ -134,14 +92,13 @@ spec:
type: LoadBalancer
```
We can also expose a specific address by specifying it on the command line:
We can also expose a specific address by specifying it imperatively on the command line:
```
kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx --load-balancer-ip=1.1.1.1
```
or including it in the `service` definition:
or including it in the Service definition:
```
apiVersion: v1
@@ -161,14 +118,12 @@ spec:
### Using DHCP for Load Balancers (experimental)
With the latest release of `kube-vip` > 0.2.1, it is possible to use the local network DHCP server to provide `kube-vip` with a load-balancer address that can be used to access a
Kubernetes service on the network.
With `kube-vip` > 0.2.1, it is possible to use the local network DHCP server to provide `kube-vip` with a load balancer address that can be used to access a Kubernetes service on the network.
In order to do this we need to signify to `kube-vip` and the cloud-provider that we don't need one of their managed addresses. We do this by explicitly exposing a service on the
address `0.0.0.0`. When `kube-vip` sees a service on this address it will create a `macvlan` interface on the host and request a DHCP address, once this address is provided it will assign it as the VIP and update the Kubernetes service!
In order to do this, we need to signify to `kube-vip` and the cloud provider that we don't need one of their managed addresses. We do this by explicitly exposing a Service on the address `0.0.0.0`. When `kube-vip` sees a Service on this address, it will create a `macvlan` interface on the host and request a DHCP address. Once this address is provided, it will assign it as the `LoadBalancer` IP and update the Kubernetes Service.
```
$ k expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx-dhcp --load-balancer-ip=0.0.0.0; k get svc
$ kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx-dhcp --load-balancer-ip=0.0.0.0; kubectl get svc
service/nginx-dhcp exposed
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17m
@@ -176,44 +131,43 @@ nginx-dhcp LoadBalancer 10.97.150.208 0.0.0.0 80:31184/TCP 0s
{ ... a second or so later ... }
$ k get svc
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17m
nginx-dhcp LoadBalancer 10.97.150.208 192.168.0.155 80:31184/TCP 3s
```
### Using UPNP to expose a service to the outside world
### Using UPnP to expose a Service to the outside world
With the latest release of `kube-vip` > 0.2.1, it is possible to expose a load-balancer on a specific port and using UPNP (on a supported gateway) expose this service to the inte
rnet.
With `kube-vip` > 0.2.1, it is possible to expose a Service of type `LoadBalancer` on a specific port to the Internet by using UPnP (on a supported gateway).
Most simple networks look something like the following:
`<----- <internal network 192.168.0.0/24> <Gateway / router> <external network address> ----> Internet`
Using UPNP we can create a matching port on the `<external network address>` allowing your service to be exposed to the internet.
Using UPnP we can create a matching port on the `<external network address>` allowing your Service to be exposed to the Internet.
#### Enable UPNP
#### Enable UPnP
Add the following to the `kube-vip` `env:` section, and the rest should be completely automated.
Add the following to the `kube-vip` `env:` section of either the static Pod or DaemonSet for `kube-vip`, and the rest should be completely automated.
**Note** some environments may require (Unifi) will require `Secure mode` being `disabled` (this allows a host with a different address to register a port)
**Note** some environments may require (Unifi) `Secure mode` being `disabled` (this allows a host with a different address to register a port).
```
- name: enableUPNP
value: "true"
```
#### Exposing a service
#### Exposing a Service
To expose a port successfully we'll need to change the command slightly:
To expose a port successfully, we'll need to change the command slightly:
`--target-port=80` the port of the application in the pods (HTT/NGINX)
`--port=32380` the port the service will be exposed on (and what you should connect to in order to receive traffic from the service)
`--port=32380` the port the Service will be exposed on (and what you should connect to in order to receive traffic from the Service)
`kubectl expose deployment plunder-nginx --port=32380 --target-port=80 --type=LoadBalancer --namespace plunder`
The above example should expose a port on your external (internet facing address), that can be tested externally with:
The above example should expose a port on your external (Internet facing) address that can be tested externally with:
```
$ curl externalIP:32380