Integrate with Canonical Observability Stack

This how-to guide provides instructions for connecting Charmed HPC to the Canonical Observability Stack (COS). This integration enables you to monitor your deployed Charmed HPC cluster by forwarding collected logs and metrics from your cluster’s services to COS for interactive analysis.

New to COS?

If you’re unfamiliar with operating COS, see the COS tutorials for a high-level introduction to the Canonical Observability Stack.

Prerequisites

To integrate Charmed HPC with COS, you will need:

Ingress enabled

Your COS deployment must have ingress enabled. If your COS deployment does not have ingress enabled, Charmed HPC will be unable to forward collected logs and metrics as COS will be unreachable over the network.

Before you begin

The instructions below assume that Charmed HPC and COS have their own, individual controllers, and that they are connected together with cross-model integration endpoints.

The instructions below also assume that the name of the COS controller is cos-controller, and that the model holding your COS deployment is named cos. If the name of your COS controller is not cos-controller, or the name of your model is not cos, substitute cos-controller and cos with the names of your COS controller and model in the commands below.

Connect to COS

Follow the instructions below for how to integrate COS with Charmed HPC.

Deploy Grafana Agent

First, within your charmed-hpc cloud, use juju deploy to deploy Grafana Agent on your cluster:

juju deploy grafana-agent --model charmed-hpc-controller:slurm \
  --base "ubuntu@24.04"

Connect Charmed HPC to Grafana Agent

Still within your charmed-hpc cloud, use juju integrate to connect Grafana Agent to your cluster’s applications:

juju integrate --model charmed-hpc-controller:slurm grafana-agent slurmctld

After integrating Grafana Agent with your cluster’s applications, Grafana Agent will install itself on each unit of the application to collect logs and metrics.

Get COS URLs

Before you connect Grafana Agent to COS, you’ll want to ensure that your charmed-hpc cloud can successfully contact your COS deployment over the network. To perform this connectivity check, first grab the URLs of the COS services from Catalogue by running the following command:

juju show-unit --model cos-controller:cos catalogue/0 --format json | \
  jq '.[]."relation-info".[]."application-data".url | select (. != null)'

The piped output of the juju show-unit command should be similar to the following:

user@host:~$
juju show-unit --model cos-controller:cos catalogue/0 --format json | jq '.[]."relation-info".[]."application-data".url | select (. != null)' 
"http://10.190.89.230/cos-grafana"
"http://10.190.89.230/cos-prometheus-0"
"http://10.190.89.230/cos-alertmanager"

Save these URLs as they will be useful in the Access monitoring resources section.

Check connectivity

To verify that your charmed-hpc cloud can connect to your COS deployment, try to access Prometheus by running the following curl command with juju exec:

juju exec --unit grafana-agent/0 \
  "curl -s http://10.190.89.230/cos-prometheus-0/api/v1/status/runtimeinfo"

If the output of juju exec looks similar to the success message below, this means that your charmed-hpc cloud can successfully contact your COS deployment:

user@host:~$
juju exec --unit grafana-agent/0 -- curl -s http://10.190.89.230/cos-prometheus-0/api/v1/status/runtimeinfo 
{"status":"success","data":{"startTime":"2025-02-06T19:09:05.141616388Z","CWD":"/","reloadConfigSuccess":true,"lastConfigTime":"2025-02-06T19:10:36Z","corruptionCount":0,"goroutineCount":56,"GOMAXPROCS":8,"GOMEMLIMIT":9223372036854775807,"GOGC":"","GODEBUG":"","storageRetention":"15d or 819MiB204KiB819B"}}

Make offers from COS

Now that you have verified that your charmed-hpc cloud can connect to your COS deployment, create offers in your COS deployment using juju offer. You can use juju switch to switch to your cos model:

juju switch cos-controller:cos
juju offer cos.grafana:grafana-dashboard grafana-dashboards
juju offer cos.loki:logging loki-logging
juju offer cos.prometheus:receive-remote-write prometheus-receive-remote-write

Consume offers in Charmed HPC

After making the offers in your cos model, use juju consume to consume the offers in your slurm model on your charmed-hpc cloud. You can use juju switch to switch to your slurm model:

juju switch charmed-hpc-controller:slurm
juju consume cos-controller:cos.prometheus-receive-remote-write
juju consume cos-controller:cos.grafana-dashboards
juju consume cos-controller:cos.loki-logging

Connect Grafana Agent to COS endpoints

Now use juju integrate to connect Grafana Agent to the COS endpoints:

juju switch charmed-hpc-controller:slurm
juju integrate grafana-agent prometheus-receive-remote-write
juju integrate grafana-agent loki-logging
juju integrate grafana-agent grafana-dashboards

With Grafana Agent connected to COS, you can now use the URLs from the Get COS URLs section to access monitoring resources such as metrics, logs, and alerts collected from your Charmed HPC cluster. See the Access monitoring resources section below for how to access these monitoring resources through your browser.

Access monitoring resources

Follow the instructions below for how to view alerts, logs, and metrics collected from your Charmed HPC cluster after you have integrated your cluster with COS.

Access Grafana dashboards

First, in your terminal, retrieve the Grafana admin password with the get-admin-password action:

juju run grafana/leader --model cos-controller:cos \
  --wait 1m \
  get-admin-password

About the admin password

The get-admin-password action returns the initial admin password that is generated when COS is first deployed. The action will return a notice if the initial admin password has been changed by the COS administrator. If this is the case, you will need to get either a Grafana account or the admin password from your COS administrator.

Next, open your browser and navigate to the Grafana dashboard URL you received after running the piped juju show-unit command in the Get COS URLs section.

Log in as the user admin using the password returned by the get-admin-password action. You can see the available dashboards by opening the sidebar menu and clicking on Dashboards.

Next steps

You can now use COS to monitor your Charmed HPC cluster. Explore the Grafana web interface to see what dashboards you can create using the metrics collected from your Charmed HPC cluster.