Loki logs

The following table lists all the charms used as part of Charmed HPC that expose logs to Loki, and the corresponding query to see the exported logs in Grafana. Follow the Visualize log data tutorial from the Grafana documentation for instructions on where and how to query for Loki logs.

charm	query
slurmctld	{juju_charm="opentelemetry-collector"} | filename =~ ".*/var/log/slurm/slurmctld.*"
slurmd	{juju_charm="opentelemetry-collector"} | filename =~ ".*/var/log/slurm/slurmd.*"
slurmdbd	{juju_charm="opentelemetry-collector"} | filename =~ ".*/var/log/slurm/slurmdbd.*"
mysql	{juju_charm="opentelemetry-collector"} | filename =~ ".*/var/log/mysql/.*"
postgresql-k8s	{juju_charm="postgresql-k8s"}
glauth-k8s	{charm="glauth-k8s"}

Ignoring log files

By default, every instance of the opentelemtry-collector charm will log all the files in the /var/log directory. This is sometimes not ideal, since it increases the amount of logs stored by Loki which are unrelated to the running application. A solution for this is to set the path_exclude configuration for opentelemetry-collector, which will allow it to ignore such log files:

juju config opentelemetry-collector path_exclude="/var/log/{unattended-upgrades,apt,}/*.log"