How to send logs to Loki¶
LXD publishes information about its activity in the form of events. The lxc monitor
command allows you to view this information in your shell. There are two categories of LXD events: logs and life cycle. The lxc monitor --type=logging --pretty
command will filter and display log type events like activity of the raft cluster, for instance, while lxc monitor --type=lifecycle --pretty
will only display life cycle events like instances starting or stopping.
In a production environment, you might want to keep a log of these events in a dedicated system. Loki is one such system, and LXD provides a configuration option to forward its event stream to Loki.
Configure LXD to send logs¶
See the Loki documentation for instructions on installing it:
Once you have a Loki server up and running, you can instruct LXD to send logs to your Loki server by setting the following option:
lxc config set loki.api.url=http://<loki_server_IP>:3100
Note
If Loki logs are to be viewed in the Grafana dashboard, ensure the loki.instance
configuration key matches the name of the Prometheus job. See Set up a Grafana dashboard.
Query Loki logs¶
Loki logs are typically viewed/queried using Grafana but Loki provides a command line utility called LogCLI allowing to query logs from your Loki server without the need for Grafana.
See the LogCLI documentation for instructions on installing it:
With your LogCLI utility up and running, first configure it to query the server you have installed before by setting the appropriate environment variable:
export LOKI_ADDR=http://<loki_server_IP>:3100
You can then query the Loki server to validate that your LXD events are getting through. LXD events all have the app
key set to lxd
so you can use the following logcli
command to see LXD logs in Loki.
user@host:~$
logcli query -t '{app="lxd"}'
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Updating instance types
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Expiring log files
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Pruning resolved warnings
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Updating images
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done pruning resolved warnings
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done expiring log files
2024-02-14T21:31:20Z {app="lxd", instance="node3", type="logging"} level="info" Done updating images
...
Add labels¶
LXD pushes log entries with a set of predefined labels like app
, project
, instance
and name
. To see all existing labels, you can use logcli labels
. Some log entries might contain information in their message that you would like to access as if they were keys. In the example below, you might want to have requester-username
as a key to query.
2024-02-15T22:52:25Z {app="lxd", instance="node3", location="node3", name="c1", project="default", type="lifecycle"} requester-username="ubuntu" action="instance-started" source="/1.0/instances/c1" requester-address="@" requester-protocol="unix" instance-started
...
Use the following command to instruct LXD to move all occurrences of requester-username="<user>"
into the label section:
lxc config set loki.labels="requester-username"
This will transform the above log entry into:
2024-02-09T21:26:32Z {app="lxd", instance="node3", location="node3", name="c2", project="default", requester_username="ubuntu", type="lifecycle"} action="instance-started" source="/1.0/instances/c2" requester-address="@" requester-protocol="unix" instance-started
...
Note the replacement of -
by _
, as -
cannot be used in keys. As requested_username
is now a key, you can query Loki using it like this:
logcli query -t '{requester_username="ubuntu"}'