KaaS – Deploy workload cluster

Workload cluster deployment

Prerequisite

• A virtual datacenter
• A routed network with internet access (SNAT)
• The LoadBalancing feature enabled (request to be made)
• Bootstrap cluster deployed
• Command-line tools KaaS – Tooling
• Internet DNS resolution
• For Internal IaaS only : Source NAT for API Access

All these steps should be made when connected to the management cluster.

NOTE : If you don’t have yet a management cluster in your environment, please follow the KaaS – Information Map page before continue.

To verify you can use this command :

# kubectl config current-context

and 

# kubectl get nodes

Deployment

Preparing the configuration file for the cluster to be created

The clusterctl tool relies on template files (provided by VMware in our case) and a configuration file with all the required parameters for creation.

The example configuration file is located at: ~/.cluster-api/clusterctl.yaml

So, we use it as a basis for our ‘wrk01‘ cluster.

cp ~/.cluster-api/clusterctl.yaml wrk01-config.yaml

Fill the ***-config.yml file previously copied

The ***-config.yml file is used to provide all the necessary information to Cluster API for deploying a cluster.

Les autres paramètres peuvent être ignorés

Generate the YAML creation file.

clusterctl generate cluster wrk01 --config wrk01-config.yaml -f v1.22.9-crs > wrk01.yaml

Apply the YAML file to initiate cluster creation.

Then, wait for the cluster to be created:

• Verify that the Cluster object is in the “Provisioned” phase using the command: `kubectl get cluster`
• Wait for all master nodes to be in the “READY” status using the command: `kubectl get kubeadmcontrolplane`

kubectl apply -f wrk01.yaml

# kubectl get kubeadmcontrolplane

NAME                   CLUSTER   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
wrk01-control-plane    wrk01     true          true                   1          1       1         0             24m   v1.22.9+vmware.1v1.22.9+vmware.1

Thanks to the ClusterResourceSet, CNI, CSI, and CPI PODs are deployed automatically. However, they are missing some configuration elements, which we will add now.

This is why these PODs are not yet running on the new cluster created.

To do this, you should be connected to the management cluster, not the new cluster.

Modify the first 6 provided commands to the right and apply them all.

– CLUSTERNAME: Name of the management cluster you are currently creating.

– MGT: Path to the kubeconfig of your management cluster (acting as your management cluster).

– VCDHOST: URL of the vCloud Director console.

– ORG: Name of your organization.

– OVCD: Name of the virtual datacenter where the cluster is located.

– ONETWORK: Name of the VCD network where the cluster is located.

After a few minutes, all PODs should be in the RUNNING phase on the ‘wrk01’ cluster.

CLUSTERNAME=mgt01
MGT=~/.kube/kube-bootstrap01
VCDHOST=https://*******.com/
ORG=***
OVDC=***
ONETWORK=****

WRK=~/.kube/kube-${CLUSTERNAME}
clusterctl get kubeconfig ${CLUSTERNAME} > ${WRK}
export CLUSTERID=$(kubectl --kubeconfig=${MGT} get vcdclusters ${CLUSTERNAME} -o jsonpath="{.status.infraId}")
kubectl --kubeconfig=${WRK} -n kube-system create secret generic vcloud-clusterid-secret --from-literal=clusterid=${CLUSTERID}

export REFRESH_TOKEN=$(kubectl --kubeconfig=${MGT} get secret capi-user-credentials -o jsonpath="{.data.refreshToken}" | base64 --decode)
kubectl --kubeconfig=${WRK} -n kube-system create secret generic vcloud-basic-auth --from-literal=refreshToken=${REFRESH_TOKEN} --from-literal=username="" --from-literal=password=""

Create the configmaps.

Use the two commands provided here to create the required configmaps.

You can customize them, including choosing a vipSubnet.

echo "apiVersion: v1
kind: ConfigMap
metadata:
    name: vcloud-csi-configmap
    namespace: kube-system
data:
    vcloud-csi-config.yaml: |+
        vcd:
          host: ${VCDHOST}
          org: ${ORG}
          vdc: ${OVDC}
          vAppName: ${CLUSTERNAME}
        clusterid: ${CLUSTERID}
immutable: true" | kubectl apply --kubeconfig=${WRK} -f -

echo "apiVersion: v1
kind: ConfigMap
metadata:
      name: vcloud-ccm-configmap
      namespace: kube-system
data:
      vcloud-ccm-config.yaml: |+
        vcd:
          host: ${VCDHOST}
          org: ${ORG}
          vdc: ${OVDC}
        loadbalancer:
          oneArm:
            startIP: \"192.168.8.2\"
            endIP: \"192.168.8.100\"
          ports:
            http: 80
            https: 443
          network: ${ONETWORK}
          vipSubnet: \"\"
          certAlias: ${CLUSTERID}-cert
          enableVirtualServiceSharedIP: false # supported for VCD >= 10.4
        clusterid: ${CLUSTERID}
        vAppName: ${CLUSTERNAME}
immutable: true" | kubectl apply --kubeconfig=${WRK} -f -

After a few moments, all PODs should be in the “Running” state on the cluster, and the machinedeployment components on the management cluster should be in the “READY” status and “Running” phase.

# kubectl get machinedeployment

NAME          CLUSTER   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
wrk01-md-0    wrk01     1          1       1         0             Running   28m   v1.22.9+vmware.1

Retrieve the kubeconfig file for the new cluster

clusterctl get kubeconfig CLUSTERNAME > ~/.kube/kube-CLUSTERNAME

Example :

clusterctl get kubeconfig wrk01 > ~/.kube/kube-wrk01

• Switch to the management cluster with kctx or any other way you want
• Check if you are correctly connected to the good cluster with kubectl get nodes command

kctx
kubectl get nodes

Delete workload cluster

Connected to the management cluster, run the kubectl delete cluster command :

kubectl delete cluster CLUSTER-NAME

Create a Storage Class

Using KaaS Standard you can consume storage in your Virtual Data center Automatically for the Persistent Volume for your PODS.

For that you have to create K8S Storage Class that is link to a VCD Storage Policy. In that way you can use storage with different performance.

Create a file mystorageclass.yaml and copy the content provided on the right and replace

by the storage policy of your choice.

The example provided create a default storage class, if you don’t want to set this Storage Class as default you can simply change the annotation storageclass.kubernetes.io/is-default-class to false.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: storage-class-name
provisioner: named-disk.csi.cloud-director.vmware.com
reclaimPolicy: Delete
parameters:
  storageProfile: "YOUR_STORAGE_POLICY"
  filesystem: "ext4"

Set the vipSubnet to use for external service

The vipSubnet correspond to the vCloud Director external network cidr to use. When a load balancer is created an ip will be taken inside this CIDR and will point to the LB interface using a NAT rule.

To choose or simply change the CIDR to use on a cluster, follow the below steps.

Extract the actual configmap in place

kubectl -n kube-system get cm/vcloud-ccm-configmap -o yaml > vcloud-ccm-configmap.yaml

Get tue actual CPI version in use

# kubectl get pod -n kube-system  -l app=vmware-cloud-director-ccm -o=jsonpath='{range .items[*]}{.status.containerStatuses[0].image}{"\r\n"}{end}'

projects.registry.vmware.com/vmware-cloud-director/cloud-provider-for-cloud-director:1.1.0.latest

Update the vipSubnet in the generated file

apiVersion: v1
data:
  vcloud-ccm-config.yaml: |
    vcd:
      host: "https://console1.cloudavenue.orange-business.com"
      org: "MYORG01"
      vdc: "MYVDC01"
      vAppName: mycluster01
      network: "myprodnetwork"
      vipSubnet: "10.102.122.102/26"
    loadbalancer:
      oneArm:
        startIP: "192.168.8.2"
        endIP: "192.168.8.100"
      ports:
        http: 80
        https: 443
immutable: true
kind: ConfigMap
metadata:
  name: vcloud-ccm-configmap
  namespace: kube-system

Delete the configmap & the ccm deployment

kubectl -n kube-system delete cm/vcloud-ccm-configmap

kubectl -n kube-system delete deploy/vmware-cloud-director-ccm

Recreate the ccm with the good cpi version.

Depending of the version installed on your cluster the cloud-director-ccm.yaml url can change so compare with the version you got in step 2 and adapt the url with your version.

kubectl apply -f vcloud-ccm-configmap.yaml
kubectl apply -f https://raw.githubusercontent.com/vmware/cloud-provider-for-cloud-director/1.1.0/manifests/cloud-director-ccm.yaml

Retrieve refreshToken

• Open the vCloud Director console with the account you want to use for cluster creation (kind of a service account)
• On the top right of the page clic on the 3 dots vertical button
• Clic on user preferences
• On the Access Tokens section clic New
• Give a client name to the token then clic Create
• Copy and keep safely the token generated