How to debug LXD¶
For information on debugging instance issues, see How to troubleshoot failing instances.
Here are different ways to help troubleshooting
--debug flag to any client command will give extra information
about internals. If there is no useful info, it can be added with the
logger.Debugf("Hello: %s", "Debug")
This command will monitor messages as they appear on remote server.
REST API through local socket¶
On server side the most easy way is to communicate with LXD through
local socket. This command accesses
GET /1.0 and formats JSON into
human readable form using jq
curl --unix-socket /var/lib/lxd/unix.socket lxd/1.0 | jq .
or for snap users:
curl --unix-socket /var/snap/lxd/common/lxd/unix.socket lxd/1.0 | jq .
See the RESTful API for available API.
REST API through HTTPS¶
openssl can be used to examine the certificate (
~/snap/lxd/common/config/client.crt for snap users):
openssl x509 -text -noout -in client.crt
Among the lines you should see:
SSL client : Yes
With command line tools¶
wget --no-check-certificate --certificate=$HOME/.config/lxc/client.crt --private-key=$HOME/.config/lxc/client.key -qO - https://127.0.0.1:8443/1.0
# or for snap users
wget --no-check-certificate --certificate=$HOME/snap/lxd/common/config/client.crt --private-key=$HOME/snap/lxd/common/config/client.key -qO - https://127.0.0.1:8443/1.0
Some browser plugins provide convenient interface to create, modify
and replay web requests. To authenticate against LXD server, convert
lxc client certificate into importable format and import it into
For example this produces
client.pfx in Windows-compatible format:
openssl pkcs12 -clcerts -inkey client.key -in client.crt -export -out client.pfx
After that, opening
https://127.0.0.1:8443/1.0 should work as expected.
Debug the LXD database¶
The files of the global database are stored under the
sub-directory of your LXD data directory (e.g.
/var/snap/lxd/common/lxd/database/global for snap users).
Since each member of the cluster also needs to keep some data which is specific
to that member, LXD also uses a plain SQLite database (the “local” database),
which you can find in
Backups of the global database directory and of the local database file are made
before upgrades, and are tagged with the
.bak suffix. You can use those if
you need to revert the state as it was before the upgrade.
Dumping the database content or schema¶
If you want to get a SQL text dump of the content or the schema of the databases,
lxd sql <local|global> [.dump|.schema] command, which produces the
equivalent output of the
.schema directives of the
command line tool.
Running custom queries from the console¶
If you need to perform SQL queries (e.g.
against the local or global database, you can use the
lxd sql command (run
lxd sql --help for details).
Running custom queries at LXD daemon startup¶
In case the LXD daemon fails to start after an upgrade because of SQL data
migration bugs or similar problems, it’s possible to recover the situation by
.sql files containing queries that repair the broken update.
To perform repairs against the local database, write a
./database/patch.local.sql file containing the relevant queries, and
./database/patch.global.sql for global database repairs.
Those files will be loaded very early in the daemon startup sequence and deleted if the queries were successful (if they fail, no state will change as they are run in a SQL transaction).
As above, please consult the LXD team first.
Syncing the cluster database to disk¶
If you want to flush the content of the cluster database to disk, use the
lxd sql global .sync command, that will write a plain SQLite database file into
./database/global/db.bin, which you can then inspect with the
command line tool.