Troubleshooting

Cluster API Troubleshooting

Verification of the PODS

When a problem occur with cluster API the first thing to do is to check if all the PODs involved are running.

Check is these 4 pods have the STATUS Running

# kubectl get pods -A

NAMESPACE                           NAME                                                            READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-7dc44947-hrmvc        1/1     Running   0          36m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-cb9d954f5-r8w54   1/1     Running   0          36m
capi-system                         capi-controller-manager-7594c7bc57-jr75r                        1/1     Running   0          37m
capvcd-system                       capvcd-controller-manager-89758d745-kw4sm                       1/1     Running   0          13s

If one of these pods don’t have a Running status, two things can be check in order to get information about the issue.

1. Describe the pod that have error
2. Get the logs of the pod that have error

Example :

# kubectl describe pod capi-controller-manager-7594c7bc57-jr75r -n capi-system

and

# kubectl logs capi-controller-manager-7594c7bc57-jr75r -n capi-system

Check the provider VCD

If issue occurs with the actions made on vCloud Director, during cluster creation, upgrade or scaling operation the problem can come from the Cluster API Provider.

The corresponding pod is capvcd-controller-manager inside the capvcd namespace.

Error can be found in the logs :

# kubectl logs capvcd-controller-manager-7594c7bc57-jr75r -n capvcd-system

An option exists to display more logs regarding communication between the provider and vCloud Director.

For that apply this command :

kubectl set env -n capvcd-system deployment/capvcd-controller-manager GOVCD_LOG_ON_SCREEN=true -oyaml

These option could be verbose, so do not forget to disable it when your diagnose is ended, for that :

kubectl set env -n capvcd-system deployment/capvcd-controller-manager GOVCD_LOG_ON_SCREEN-

Check the Cluster API Objects

Cluster API use different types of objects to describe a Kubernetes cluster to manage.

The idea is to explore these objects step by step to find the object that have an error in his status or when we describe the object

Depending if the issue is about worker nodes, master nodes or the overall cluster, it is possible to choose the objects involved using the above diagram.

• Get the objects to find the exact name and to check the status

# kubectl get MachineDeployment -A

• Describe the object that have an issue

# kubectl describe MachineDeployment mycluster-workers-0 -n mynamespace

Repeat action 1 & 2 for all other objects till find the error.

Export logs script

VMware creates a script that export logs in a tar ball and some information about the cluster.

generate-k8s-log-bundle.sh

export KUBECONFIG=[PATH_TO_YOUR_KUBECONFIG]
curl -s https://raw.githubusercontent.com/vmware/cloud-provider-for-cloud-director/[YOUR_CAPVCD_VERSION]/scripts/generate-k8s-log-bundle.sh | bash -

Node deployment troubleshooting

Check kubelet status

systemctl status kubelet

Journalctl

journalctl -xeu containerd

journalctl -xeu kubelet

Multiple files could be parsed to troubleshoot an issue during deployment :

Cloud-init

/var/log/cloud-init-output.log

/var/log/capvcd/customization/status.log

/var/log/capvcd/customization/error.log

containerd

/var/log/containers/*