How to use Prometheus with Canonical Kubernetes

Observability is an essential component in any system for understanding, managing, and improving its performance and reliability. The main pillars of observability are metrics, logs and traces.

One of these pillars is covered by Prometheus, an open-source systems monitoring and alerting toolkit designed to collect, process, and query time-series metrics.

This guide walks you through installing Prometheus in a Canonical Kubernetes environment.

Prerequisites

This guide assumes the following:

Install Prometheus

Prometheus and its operator can be installed with a Helm chart. Start by adding the community Helm chart repository to your system:

sudo helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
sudo helm repo update

Before deploying the Helm chart, you can customize it with a values.yaml file. You can generate it by running:

sudo helm show values prometheus-community/kube-prometheus-stack > values.yaml

In order to ensure High Availability for the Prometheus services, make sure to configure the volumeClaimTemplate sections appropriately for Alertmanager and Prometheus (and ThanosRuler, if enabled), and the persistence section for Grafana. If the storageClassName field is not set, the cluster’s default StorageClass will be used instead. Make sure you have a StorageClass created for your persistent storage solution of choice. You can list them by running the following:

sudo k8s kubectl get storageclass

After the Prometheus deployment has been customized with the values.yaml file, run the following command:

sudo helm install prometheus prometheus-community/kube-prometheus-stack \
  --create-namespace --namespace observability -f values.yaml

Note that this Helm chart installs a few dependent charts:

  • prometheus-community/kube-state-metrics

  • prometheus-community/prometheus-node-exporter

  • grafana/grafana

Verify that Prometheus is running

It is recommended to ensure that Prometheus initializes properly and is running without issues. Check that the Prometheus pods are running:

sudo k8s kubectl get pods -n observability -l "app.kubernetes.io/name=prometheus"

Next, connect to the Prometheus dashboard through its Kubernetes Service:

SVC_NAME="prometheus-kube-prometheus-prometheus"
sudo k8s kubectl get -n observability svc/$SVC_NAME
CLUSTER_IP="$(sudo k8s kubectl get -n observability svc/$SVC_NAME -o jsonpath='{.spec.clusterIP}')"
CLUSTER_IP_PORT="$(sudo k8s kubectl get -n observability svc/$SVC_NAME -o jsonpath='{.spec.ports[0].port}')"
echo "Prometheus dashboard URL (ClusterIP): http://${CLUSTER_IP}:${CLUSTER_IP_PORT}/graph"

If you do not have access to the cluster network, or if the Prometheus Kubernetes service is not exposed externally, you can instead create a temporary local port-forward to the Prometheus dashboard:

export POD_NAME=$(sudo k8s kubectl get pods --namespace observability -l "app.kubernetes.io/name=prometheus" -o jsonpath="{.items[0].metadata.name}")
sudo k8s kubectl --namespace observability port-forward $POD_NAME 9090

You can check the metrics that have been scraped so far by running:

curl -s http://${CLUSTER_IP}:${CLUSTER_IP_PORT}/metrics

Accessing Grafana

Grafana is an open-source analytics and visualization web application that enables you query, visualize, alert on, and explore metrics, logs, and traces.

If you’ve deployed Prometheus with the Helm chart above, you should already have Grafana deployed in your cluster:

sudo k8s kubectl get pods -n observability -l "app.kubernetes.io/name=grafana"

Next, connect to the Grafana dashboard through its Kubernetes service:

SVC_NAME="prometheus-grafana"
sudo k8s kubectl get -n observability svc/$SVC_NAME
CLUSTER_IP="$(sudo k8s kubectl get -n observability svc/$SVC_NAME -o jsonpath='{.spec.clusterIP}')"
CLUSTER_IP_PORT="$(sudo k8s kubectl get -n observability svc/$SVC_NAME -o jsonpath='{.spec.ports[0].port}')"
echo "Grafana dashboard URL (ClusterIP): http://${CLUSTER_IP}:${CLUSTER_IP_PORT}/"

If you do not have access to the cluster network, or if the Grafana Kubernetes service is not exposed externally, you can instead create a temporary local port-forward to the Grafana dashboard:

export POD_NAME=$(sudo k8s kubectl get pods --namespace observability -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
sudo k8s kubectl --namespace observability port-forward $POD_NAME 3000

The default username/password for Grafana are: admin/prom-operator

Removing Prometheus

Prometheus and its related components (including Grafana) can be removed by running:

sudo helm delete prometheus -n observability

NOTE:: The Persistent Volumes created for Prometheus and its related services may not deleted when removing Prometheus. You can check them by running:

sudo k8s kubectl get -n observability pvc
sudo k8s kubectl get pv