How to send logs to Loki

LXD publishes information about its activity in the form of events. The lxc monitor command allows you to view this information in your shell. There are two categories of LXD events: logs and life cycle. The lxc monitor --type=logging --pretty command will filter and display log type events like activity of the raft cluster, for instance, while lxc monitor --type=lifecycle --pretty will only display life cycle events like instances starting or stopping.

In a production environment, you might want to keep a log of these events in a dedicated system. Loki is one such system, and LXD provides a configuration option to forward its event stream to Loki.

Configure LXD to send logs

See the Loki documentation for instructions on installing it:

Once you have a Loki server up and running, you can instruct LXD to send logs to your Loki server by setting the following option:

lxc config set loki.api.url=http://<loki_server_IP>:3100

Note

If Loki logs are to be viewed in the Grafana dashboard, ensure the loki.instance configuration key matches the name of the Prometheus job. See Set up a Grafana dashboard.

Query Loki logs

Loki logs are typically viewed/queried using Grafana but Loki provides a command line utility called LogCLI allowing to query logs from your Loki server without the need for Grafana.

See the LogCLI documentation for instructions on installing it:

With your LogCLI utility up and running, first configure it to query the server you have installed before by setting the appropriate environment variable:

export LOKI_ADDR=http://<loki_server_IP>:3100

You can then query the Loki server to validate that your LXD events are getting through. LXD events all have the app key set to lxd so you can use the following logcli command to see LXD logs in Loki.

user@host:~$ logcli query -t '{app="lxd"}'
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Updating instance types2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Expiring log files2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Pruning resolved warnings2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Updating images2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done pruning resolved warnings2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done expiring log files2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done updating images...

Add labels

LXD pushes log entries with a set of predefined labels like app, project, instance and name. To see all existing labels, you can use logcli labels. Some log entries might contain information in their message that you would like to access as if they were keys. In the example below, you might want to have requester-username as a key to query.

2024-02-15T22:52:25Z {app="lxd", instance="node3", location="node3", name="c1", project="default", type="lifecycle"} requester-username="ubuntu" action="instance-started" source="/1.0/instances/c1" requester-address="@" requester-protocol="unix" instance-started
...

Use the following command to instruct LXD to move all occurrences of requester-username="<user>" into the label section:

lxc config set loki.labels="requester-username"

This will transform the above log entry into:

2024-02-09T21:26:32Z {app="lxd", instance="node3", location="node3", name="c2", project="default", requester_username="ubuntu", type="lifecycle"} action="instance-started" source="/1.0/instances/c2" requester-address="@" requester-protocol="unix" instance-started
...

Note the replacement of - by _, as - cannot be used in keys. As requested_username is now a key, you can query Loki using it like this:

logcli query -t '{requester_username="ubuntu"}'