How to manage logs¶

See also: Log

Tip

For an integrated solution consider the Loki charm.

Manage the logs¶

Stream the logs¶

To stream the logs in the current model (including agent logs and logs from charm code that uses the Python logging module), run the debug-log command:

juju debug-log

In a machine deployment, this will output logs for all the Juju machine and unit agents, starting with the last 10 lines, in the following format:

<entity (machine or unit)> <timestamp> <log level> <Juju module> <message>

The command has various options that allow you to control the length, appearance, amount and type of detail.

See more: juju debug-log

Configure the logging level¶

Juju saves or discards logs according to the value of the model config key logging-config. Therefore, logging-config needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug).

Set values.

To change the logging configuration for machine and unit agents:
Run the model-config command with the logging-config key set to a "-enclosed, semi-colon-separated list of <filter>=<verbosity level> pairs.

See more: Configure a model, logging-config

Caution

To avoid filling up the database unnecessarily:
When verbose logging is no longer needed, return logging to normal levels!

To change the logging configuration on a per-unit-agent basis:

SSH into the unit’s machine. E.g., for mysql/0:

juju ssh mysql/0

Open the unit’s agent configuration file. For our example, it will be /var/lib/juju/agents/unit-mysql-0/agent.conf/ Then, find the values section, and add a line with the field LOGGING_OVERRIDE set to juju=<desired log level>, below TRACE. The bottom of the file should now look as below:

loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"

Restart the affected agent:

sudo systemctl restart jujud-unit-mysql-0.service

Get values. To verify the current logging configuration for machine and unit agents, run model-config followed by the logging-config key:

juju model-config logging-config

Sample output:

<root>=WARNING;unit=DEBUG;#http=TRACE

which means that the machine agent (<root>) log level is set to WARNING, the unit agent (unit) log level is set at DEBUG, and the http label is set to TRACE.

See more: Configure a model, logging-config

Forward logs to an external logsink¶

You can optionally forward log messages to a remote syslog server over a secure TLS connection, on a per-model basis, as below:

See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.

Configure the controller for remote logging by configuring it during controller creation as below:

juju bootstrap <cloud> --config mylogconfig.yaml

where the YAML file is as below:

syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
 <cert-contents>
-----END PRIVATE KEY-----

See more: Configure a controller

Enable log forwarding for a model by configuring it as below:

juju model-config -m <model> logforward-enabled=True

An initial 100 (maximum) existing log lines will be forwarded.

See more: Configure a model

Tip

You can configure remote logging and enable log forwarding on all the controller’s models in one step by running

juju bootstrap <cloud> --config mylogconfig.yaml --config logforward-enabled=True

Manage the log files¶

Caution

Only applicable for machines – for Kubernetes logs are written directly to stdout of the container and can be retrieved with native Kubernetes methods, e.g., kubectl logs -c <container-name> <pod-name> -n <model-name> .

View the log files¶

To view the Juju log files in a Juju machine:

Open a shell into the machine:

(1a) If Juju can connect to the machine (i.e., the output contains Connected to <IP address>) and the machine is fully provisioned, use juju ssh. For example, to connect to machine 0:

juju ssh 0

(1b) If Juju can connect to the machine (i.e., the output contains Connected to <IP address>) but the machine is not fully provisioned (e.g., command hangs at Running machine configuration script...), use the ssh command followed by the address of the machine and the path to the place where Juju stores your SSH keys (including the ones it generates automatically for you):

ssh ubuntu@<ip-address> -i <juju-data-dir>/ssh/juju_id_rsa

Here, <juju-data-dir> defaults to ~/.local/share/juju, but if you’ve set the JUJU_DATA environment variable, it will be equal to that instead.

(1c) If Juju cannot connect to the machine (i.e., the command never reaches Connected to <IP address>), use cloud-specific tools. For example, for the LXD cloud:

lxc exec <container name> bash

or, for the MicroK8s cloud:

microk8s kubectl exec controller-0 -itc api-server -n [namespace] -- bash

See more: Access a machine via SSH

Examine the log files under /var/log with commands such as cat, less, or tail -f, for example:

cat /var/log/juju

Important

Which log to look at depends on the type of failure, but generally speaking, syslog, cloud-init.log, cloud-init-output.log, and /var/log/juju are good ones to look at.

Control the log file rotation¶

See also: List of controller configuration keys

Juju has settings to control the rotation of the various log files it produces.

There are two settings for each log file type: maximum size and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file.

Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file:

$ juju bootstrap localhost --config agent-logfile-max-size=1MB
$ lxc exec juju-6bf629-0 -- ls -l /var/log/juju
...
-rw-r----- 1 syslog adm   3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz
-rw-r----- 1 syslog adm   3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz
-rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log

The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the --config option (see Configure a controller).

The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs:

agent-logfile-max-backups
agent-logfile-max-size

The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings):

audit-log-max-backups
audit-log-max-size

The following config settings configure the model log files:

model-logfile-max-backups
model-logfile-max-size