Network configuration#

LXD supports the following network types:

  • bridge: Creates an L2 bridge for connecting instances to (can provide local DHCP and DNS).

The configuration keys are namespaced with the following namespaces currently supported for all network types:

  • maas (MAAS network identification)

  • user (free form key/value for user metadata)

network: bridge#

As one of the possible network configuration types under LXD, LXD supports creating and managing network bridges. LXD bridges can leverage underlying native Linux bridges and Open vSwitch.

Creation and management of LXD bridges is performed via the lxc network command. A bridge created by LXD is by default “managed” which means that LXD also will additionally set up a local dnsmasq DHCP server and if desired also perform NAT for the bridge (this is the default.)

When a bridge is managed by LXD, configuration values under the bridge namespace can be used to configure it.

Additionally, LXD can utilize a pre-existing Linux bridge. In this case, the bridge does not need to be created via lxc network and can simply be referenced in an instance or profile device configuration as follows:

devices:
  eth0:
     name: eth0
     nictype: bridged
     parent: br0
     type: nic

Network configuration properties:

A complete list of configuration settings for LXD networks can be found below.

The following configuration key namespaces are currently supported for bridge networks:

  • bridge (L2 interface configuration)

  • fan (configuration specific to the Ubuntu FAN overlay)

  • tunnel (cross-host tunneling configuration)

  • ipv4 (L3 IPv4 configuration)

  • ipv6 (L3 IPv6 configuration)

  • dns (DNS server and resolution configuration)

  • raw (raw configuration file content)

It is expected that IP addresses and subnets are given using CIDR notation (1.1.1.1/24 or fd80:1234::1/64).

The exception being tunnel local and remote addresses which are just plain addresses (1.1.1.1 or fd80:1234::1).

Key

Type

Condition

Default

Description

bridge.driver

string

-

native

Bridge driver (“native” or “openvswitch”)

bridge.external_interfaces

string

-

-

Comma separate list of unconfigured network interfaces to include in the bridge

bridge.hwaddr

string

-

-

MAC address for the bridge

bridge.mode

string

-

standard

Bridge operation mode (“standard” or “fan”)

bridge.mtu

integer

-

1500

Bridge MTU (default varies if tunnel or fan setup)

dns.domain

string

-

lxd

Domain to advertise to DHCP clients and use for DNS resolution

dns.mode

string

-

managed

DNS registration mode (“none” for no DNS record, “managed” for LXD generated static records or “dynamic” for client generated records)

fan.overlay_subnet

string

fan mode

240.0.0.0/8

Subnet to use as the overlay for the FAN (CIDR notation)

fan.type

string

fan mode

vxlan

The tunneling type for the FAN (“vxlan” or “ipip”)

fan.underlay_subnet

string

fan mode

auto (on create only)

Subnet to use as the underlay for the FAN (CIDR notation). Use “auto” to use default gateway subnet

ipv4.address

string

standard mode

auto (on create only)

IPv4 address for the bridge (CIDR notation). Use “none” to turn off IPv4 or “auto” to generate a new random unused subnet

ipv4.dhcp

boolean

ipv4 address

true

Whether to allocate addresses using DHCP

ipv4.dhcp.expiry

string

ipv4 dhcp

1h

When to expire DHCP leases

ipv4.dhcp.gateway

string

ipv4 dhcp

ipv4.address

Address of the gateway for the subnet

ipv4.dhcp.ranges

string

ipv4 dhcp

all addresses

Comma separated list of IP ranges to use for DHCP (FIRST-LAST format)

ipv4.firewall

boolean

ipv4 address

true

Whether to generate filtering firewall rules for this network

ipv4.nat

boolean

ipv4 address

false

Whether to NAT (defaults to true for regular bridges where ipv4.address is generated and always defaults to true for fan bridges)

ipv4.nat.order

string

ipv4 address

before

Whether to add the required NAT rules before or after any pre-existing rules

ipv4.nat.address

string

ipv4 address

-

The source address used for outbound traffic from the bridge

ipv4.routes

string

ipv4 address

-

Comma separated list of additional IPv4 CIDR subnets to route to the bridge

ipv4.routing

boolean

ipv4 address

true

Whether to route traffic in and out of the bridge

ipv6.address

string

standard mode

auto (on create only)

IPv6 address for the bridge (CIDR notation). Use “none” to turn off IPv6 or “auto” to generate a new random unused subnet

ipv6.dhcp

boolean

ipv6 address

true

Whether to provide additional network configuration over DHCP

ipv6.dhcp.expiry

string

ipv6 dhcp

1h

When to expire DHCP leases

ipv6.dhcp.ranges

string

ipv6 stateful dhcp

all addresses

Comma separated list of IPv6 ranges to use for DHCP (FIRST-LAST format)

ipv6.dhcp.stateful

boolean

ipv6 dhcp

false

Whether to allocate addresses using DHCP

ipv6.firewall

boolean

ipv6 address

true

Whether to generate filtering firewall rules for this network

ipv6.nat

boolean

ipv6 address

false

Whether to NAT (will default to true if unset and a random ipv6.address is generated)

ipv6.nat.order

string

ipv6 address

before

Whether to add the required NAT rules before or after any pre-existing rules

ipv6.nat.address

string

ipv6 address

-

The source address used for outbound traffic from the bridge

ipv6.routes

string

ipv6 address

-

Comma separated list of additional IPv6 CIDR subnets to route to the bridge

ipv6.routing

boolean

ipv6 address

true

Whether to route traffic in and out of the bridge

maas.subnet.ipv4

string

ipv4 address

-

MAAS IPv4 subnet to register instances in (when using network property on nic)

maas.subnet.ipv6

string

ipv6 address

-

MAAS IPv6 subnet to register instances in (when using network property on nic)

raw.dnsmasq

string

-

-

Additional dnsmasq configuration to append to the configuration file

tunnel.NAME.group

string

vxlan

239.0.0.1

Multicast address for vxlan (used if local and remote aren’t set)

tunnel.NAME.id

integer

vxlan

0

Specific tunnel ID to use for the vxlan tunnel

tunnel.NAME.interface

string

vxlan

-

Specific host interface to use for the tunnel

tunnel.NAME.local

string

gre or vxlan

-

Local address for the tunnel (not necessary for multicast vxlan)

tunnel.NAME.port

integer

vxlan

0

Specific port to use for the vxlan tunnel

tunnel.NAME.protocol

string

standard mode

-

Tunneling protocol (“vxlan” or “gre”)

tunnel.NAME.remote

string

gre or vxlan

-

Remote address for the tunnel (not necessary for multicast vxlan)

tunnel.NAME.ttl

integer

vxlan

1

Specific TTL to use for multicast routing topologies

Those keys can be set using the lxc tool with:

lxc network set <network> <key> <value>

Integration with systemd-resolved#

If the system running LXD uses systemd-resolved to perform DNS lookups, it’s possible to notify resolved of the domain(s) that LXD is able to resolve. This requires telling resolved the specific bridge(s), nameserver address(es), and dns domain(s).

For example, if LXD is using the lxdbr0 interface, get the ipv4 address with lxc network get lxdbr0 ipv4.address command (the ipv6 can be used instead or in addition), and the domain with lxc network get lxdbr0 dns.domain (if unset, the domain is lxd as shown in the table above). Then notify resolved:

systemd-resolve --interface lxdbr0 --set-domain '~lxd' --set-dns n.n.n.n

Replace lxdbr0 with the actual bridge name, and n.n.n.n with the actual address of the nameserver (without the subnet netmask).

Also replace lxd with the domain name. Note the ~ before the domain name is important; it tells resolved to use this nameserver to look up only this domain; no matter what your actual domain name is, you should prefix it with ~. Also, since the shell may expand the ~ character, you may need to include it in quotes.

In newer releases of systemd, the systemd-resolve command has been deprecated, however it is still provided for backwards compatibility (as of this writing). The newer method to notify resolved is using the resolvectl command, which would be done in two steps:

resolvectl dns lxdbr0 n.n.n.n
resolvectl domain lxdbr0 '~lxd'

This resolved configuration will persist as long as the bridge exists, so you must repeat this command each reboot and after LXD is restarted (see below on how to automate this).

Also note this only works if the bridge dns.mode is not none.

Note that depending on the dns.domain used, you may need to disable DNSSEC in resolved to allow for DNS resolution. This can be done through the DNSSEC option in resolved.conf.

To automate the systemd-resolved DNS configuration when LXD creates the lxdbr0 interface so that it is applied on system start you need to create a systemd unit file /etc/systemd/system/lxd-dns-lxdbr0.service containing:

[Unit]
Description=LXD per-link DNS configuration for lxdbr0
BindsTo=sys-subsystem-net-devices-lxdbr0.device
After=sys-subsystem-net-devices-lxdbr0.device

[Service]
Type=oneshot
ExecStart=/usr/bin/resolvectl dns lxdbr0 n.n.n.n
ExecStart=/usr/bin/resolvectl domain lxdbr0 '~lxd'

[Install]
WantedBy=sys-subsystem-net-devices-lxdbr0.device

Be sure to replace n.n.n.n in that file with the IP of the lxdbr0 bridge.

Then enable and start it using:

sudo systemctl daemon-reload
sudo systemctl enable --now lxd-dns-lxdbr0

If the lxdbr0 interface already exists (i.e LXD is running), then you can check that the new service has started:

sudo systemctl status lxd-dns-lxdbr0.service
● lxd-dns-lxdbr0.service - LXD per-link DNS configuration for lxdbr0
     Loaded: loaded (/etc/systemd/system/lxd-dns-lxdbr0.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Mon 2021-06-14 17:03:12 BST; 1min 2s ago
    Process: 9433 ExecStart=/usr/bin/resolvectl dns lxdbr0 n.n.n.n (code=exited, status=0/SUCCESS)
    Process: 9434 ExecStart=/usr/bin/resolvectl domain lxdbr0 ~lxd (code=exited, status=0/SUCCESS)
   Main PID: 9434 (code=exited, status=0/SUCCESS)

You can then check it has applied the settings using:

sudo resolvectl status lxdbr0
Link 6 (lxdbr0)
      Current Scopes: DNS
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: n.n.n.n
         DNS Servers: n.n.n.n
          DNS Domain: ~lxd

IPv6 prefix size#

For optimal operation, a prefix size of 64 is preferred. Larger subnets (prefix smaller than 64) should work properly too but aren’t typically that useful for SLAAC.

Smaller subnets while in theory possible when using stateful DHCPv6 for IPv6 allocation aren’t properly supported by dnsmasq and may be the source of issue. If you must use one of those, static allocation or another standalone RA daemon be used.

Allow DHCP, DNS with Firewalld#

In order to allow instances to access the DHCP and DNS server that LXD runs on the host when using firewalld you need to add the host’s bridge interface to the trusted zone in firewalld.

To do this permanently (so that it persists after a reboot) run the following command:

firewall-cmd --zone=trusted --change-interface=<LXD network name> --permanent

E.g. for a bridged network called lxdbr0 run the command:

firewall-cmd --zone=trusted --change-interface=lxdbr0 --permanent

This will then allow LXD’s own firewall rules to take effect.

How to let Firewalld control the LXD’s iptables rules#

When using firewalld and LXD together, iptables rules can overlaps. For example, firewalld could erase LXD iptables rules if it is started after LXD daemon, then LXD container will not be able to do any oubound internet access. One way to fix it is to delegate to firewalld the LXD’s iptables rules and to disable the LXD ones.

First step is to allow DNS and DHCP.

Then to tell to LXD totally stop to set iptables rules (because firewalld will do it):

lxc network set lxdbr0 ipv4.nat false
lxc network set lxdbr0 ipv6.nat false
lxc network set lxdbr0 ipv6.firewall false
lxc network set lxdbr0 ipv4.firewall false

Finally, to enable iptables firewalld’s rules for LXD usecase (in this example, we suppose the bridge interface is lxdbr0 and the associated IP range is 10.0.0.0/24:

firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -i lxdbr0 -s 10.0.0.0/24 -m comment --comment "generated by firewalld for LXD" -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -o lxdbr0 -d 10.0.0.0/24 -m comment --comment "generated by firewalld for LXD" -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i lxdbr0 -s 10.0.0.0/24 -m comment --comment "generated by firewalld for LXD" -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.0.0.0/24 ! -d 10.0.0.0/24 -m comment --comment "generated by firewalld for LXD" -j MASQUERADE
firewall-cmd --reload

To check the rules are taken into account by firewalld:

firewall-cmd --direct --get-all-rules

Warning: what is exposed above is not a fool-proof approach and may end up inadvertently introducing a security risk.