How to troubleshoot Canonical Kubernetes

Identifying issues in a Kubernetes cluster can be difficult, especially to new users. With Canonical Kubernetes we aim to make deploying and managing your cluster as easy as possible. This how-to guide will walk you through the steps to troubleshoot your Canonical Kubernetes cluster.

Common issues

Maybe your issue has already been solved? Check out the troubleshooting reference page to see a list of common issues and their solutions. Otherwise continue with this guide to help troubleshoot your Canonical Kubernetes cluster.

Check the cluster status

Verify that the cluster status is ready by running:

juju status

You should see a command output similar to the following:

Model        Controller           Cloud/Region         Version  SLA          Timestamp
k8s-testing  localhost-localhost  localhost/localhost  3.6.1    unsupported  09:06:50Z

App         Version  Status  Scale  Charm       Channel    Rev  Exposed  Message
k8s         1.32.0   active      1  k8s         1.32/beta  179  no       Ready
k8s-worker  1.32.0   active      1  k8s-worker  1.32/beta  180  no       Ready

Unit           Workload  Agent  Machine  Public address  Ports     Message
k8s-worker/0*  active    idle   1        10.94.106.154             Ready
k8s/0*         active    idle   0        10.94.106.136   6443/tcp  Ready

Machine  State    Address        Inst id        Base          AZ  Message
0        started  10.94.106.136  juju-380ff2-0  [email protected]      Running
1        started  10.94.106.154  juju-380ff2-1  [email protected]      Running

Interpreting the Output:

  • The Workload column shows the status of a given service.

  • The Message section details the health of a given service in the cluster.

  • The Agent column reflects any activity of the Juju agent.

During deployment and maintenance the workload status will reflect the node’s activity. An example workload may display maintenance along with the message details: Ensuring snap installation.

During normal cluster operation the Workload column reads active, the Agent column shows idle, and the messages will either read Ready or another descriptive term.

Test the API server health

Fetch the kubeconfig file for a control-plane node in the cluster by running:

juju run k8s/leader get-kubeconfig | yq .kubeconfig > cluster-kubeconfig.yaml

Warning

When running juju run k8s/leader get-kubeconfig you retrieve the kubeconfig file that uses one of the unit’s public IP addresses in the kubernetes endpoint. This endpoint ip can be overriden by providing a server argument if the api is exposed through a load-balancer.

Verify that the API server is healthy and reachable by running:

kubectl --kubeconfig cluster-kubeconfig.yaml get all

This command lists resources that exist under the default namespace. If the API server is healthy you should see a command output similar to the following:

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   29m

A typical error message may look like this if the API server can not be reached:

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Check the status of the API server service:

juju exec --unit k8s/0 -- systemctl status snap.k8s.kube-apiserver

Access the logs of the API server service by running:

juju exec --unit k8s/0 -- journalctl -u snap.k8s.kube-apiserver

A failure can mean that:

  • The API server is not reachable due to network issues or firewall limitations

  • The API server on the particular node is unhealthy

  • The control-plane node that’s being reached is down

Try reaching the API server on a different unit by retrieving the kubeconfig file with juju run <k8s/unit#> get-kubeconfig. Please replace # with the desired unit’s number.

Check the cluster nodes’ health

Confirm that the nodes in the cluster are healthy by looking for the Ready status:

kubectl --kubeconfig cluster-kubeconfig.yaml get nodes

You should see a command output similar to the following:

NAME            STATUS   ROLES                  AGE     VERSION
juju-380ff2-0   Ready    control-plane,worker   9m30s   v1.32.0
juju-380ff2-1   Ready    worker                 77s     v1.32.0

Troubleshoot an unhealthy node

Every healthy Canonical Kubernetes node has certain services up and running. The required services depend on the type of node.

Services running on both the control plane and worker nodes:

  • k8sd

  • kubelet

  • containerd

  • kube-proxy

Services running only on the control-plane nodes:

  • kube-apiserver

  • kube-controller-manager

  • kube-scheduler

  • k8s-dqlite

Services running only on the worker nodes:

  • k8s-apiserver-proxy

SSH into the unhealthy node by running:

juju ssh <k8s/unit#>

Check the status of the services on the failing node by running:

sudo systemctl status snap.k8s.<service>

Check the logs of a failing service by executing:

sudo journalctl -xe -u snap.k8s.<service>

If the issue indicates a problem with the configuration of the services on the node, examine the arguments used to run these services.

The arguments of a service on the failing node can be examined by reading the file located at /var/snap/k8s/common/args/<service>.

Investigate system pods’ health

Check whether all of the cluster’s pods are Running and Ready:

kubectl --kubeconfig cluster-kubeconfig.yaml get pods -n kube-system

The pods in the kube-system namespace belong to Canonical Kubernetes’ features such as network. Unhealthy pods could be related to configuration issues or nodes not meeting certain requirements.

Troubleshoot a failing pod

Look at the events on a failing pod by running:

kubectl --kubeconfig cluster-kubeconfig.yaml describe pod <pod-name> -n <namespace>

Check the logs on a failing pod by executing:

kubectl --kubeconfig cluster-kubeconfig.yaml logs <pod-name> -n <namespace>

You can check out the upstream debug pods documentation for more information.

Use the built-in inspection script

Canonical Kubernetes ships with a script to compile a complete report on Canonical Kubernetes and its underlying system. This is an essential tool for bug reports and for investigating whether a system is (or isn’t) working.

Inspection script can be executed on a specific unit by running the following commands:

juju exec --unit <k8s/unit#> -- sudo /snap/k8s/current/k8s/scripts/inspect.sh /home/ubuntu/inspection-report.tar.gz
juju scp <k8s/unit#>:/home/ubuntu/inspection-report.tar.gz ./

The command output is similar to the following:

Collecting service information
Running inspection on a control-plane node
 INFO:  Service k8s.containerd is running
 INFO:  Service k8s.kube-proxy is running
 INFO:  Service k8s.k8s-dqlite is running
 INFO:  Service k8s.k8sd is running
 INFO:  Service k8s.kube-apiserver is running
 INFO:  Service k8s.kube-controller-manager is running
 INFO:  Service k8s.kube-scheduler is running
 INFO:  Service k8s.kubelet is running
Collecting registry mirror logs
Collecting service arguments
 INFO:  Copy service args to the final report tarball
Collecting k8s cluster-info
 INFO:  Copy k8s cluster-info dump to the final report tarball
Collecting SBOM
 INFO:  Copy SBOM to the final report tarball
Collecting system information
 INFO:  Copy uname to the final report tarball
 INFO:  Copy snap diagnostics to the final report tarball
 INFO:  Copy k8s diagnostics to the final report tarball
Collecting networking information
 INFO:  Copy network diagnostics to the final report tarball
Building the report tarball
 SUCCESS:  Report tarball is at /home/ubuntu/inspection-report.tar.gz

Use the report to ensure that all necessary services are running and dive into every aspect of the system.

Collect debug information

To collect comprehensive debug output from your Canonical Kubernetes cluster, install and run juju-crashdump on a computer that has the Juju client installed. Please ensure that the current controller and model are pointing at your Canonical Kubernetes deployment.

sudo snap install juju-crashdump --classic --channel edge
juju-crashdump -a debug-layer -a config

Running the juju-crashdump script will generate a tarball of debug information that includes systemd unit status and logs, Juju logs, charm unit data, and Kubernetes cluster information. Please include the generated tarball when filing a bug.

Report a bug

If you cannot solve your issue and believe that the fault may lie in Canonical Kubernetes, please file an issue on the project repository.

Help us deal effectively with issues by including the report obtained from the inspect script, the tarball obtained from juju-crashdump, as well as any additional logs, and a summary of the issue.

You can check out the upstream debug documentation for more details on troubleshooting a Kubernetes cluster.