Server settings for a LXD production setup

To allow your LXD server to run a large number of instances, configure the following settings to avoid hitting server limits.

The Value column contains the suggested value for each parameter.

/etc/security/limits.conf

Note

For users of the snap, those limits are automatically raised.

Domain

Type

Item

Value

Default

Description

*

soft

nofile

1048576

unset

Maximum number of open files

*

hard

nofile

1048576

unset

Maximum number of open files

root

soft

nofile

1048576

unset

Maximum number of open files

root

hard

nofile

1048576

unset

Maximum number of open files

*

soft

memlock

unlimited

unset

Maximum locked-in-memory address space (KB)

*

hard

memlock

unlimited

unset

Maximum locked-in-memory address space (KB)

root

soft

memlock

unlimited

unset

Maximum locked-in-memory address space (KB), only need with bpf syscall supervision

root

hard

memlock

unlimited

unset

Maximum locked-in-memory address space (KB), only need with bpf syscall supervision

/etc/sysctl.conf

Note

Reboot the server after changing any of these parameters.

fs.aio-max-nr

Maximum number of concurrent asynchronous I/O operations

Key: fs.aio-max-nr
Type:

integer
Default:

65536

Suggested value: 524288

You might need to increase this limit further if you have a lot of workloads that use the AIO subsystem (for example, MySQL).

fs.inotify.max_queued_events

Upper limit on the number of events that can be queued

Key: fs.inotify.max_queued_events
Type:

integer
Default:

16384

Suggested value: 1048576

This option specifies the maximum number of events that can be queued to the corresponding inotify instance (see inotify for more information).

fs.inotify.max_user_instances

Upper limit on the number of inotify instances

Key: fs.inotify.max_user_instances
Type:

integer
Default:

128

Suggested value: 1048576

This option specifies the maximum number of inotify instances that can be created per real user ID (see inotify for more information).

fs.inotify.max_user_watches

Upper limit on the number of watches

Key: fs.inotify.max_user_watches
Type:

integer
Default:

8192

Suggested value: 1048576

This option specifies the maximum number of watches that can be created per real user ID (see inotify for more information).

kernel.dmesg_restrict

Whether to deny access to the messages in the kernel ring buffer

Key: kernel.dmesg_restrict
Type:

integer
Default:

0

Suggested value: 1

Set this option to 1 to deny container access to the messages in the kernel ring buffer. Note that setting this value to 1 will also deny access to non-root users on the host system.

kernel.keys.maxbytes

Maximum size of the key ring that non-root users can use

Key: kernel.keys.maxbytes
Type:

integer
Default:

20000

Suggested value: 2000000

kernel.keys.maxkeys

Maximum number of keys that a non-root user can use

Key: kernel.keys.maxkeys
Type:

integer
Default:

200

Suggested value: 2000

Set this option to a value that is higher than the number of instances.

net.core.bpf_jit_limit

Limit on the size of eBPF JIT allocations

Key: net.core.bpf_jit_limit
Type:

integer
Default:

varies

Suggested value: 1000000000

On kernels < 5.15 that are compiled with CONFIG_BPF_JIT_ALWAYS_ON=y, this value might limit the amount of instances that can be created.

net.ipv4.neigh.default.gc_thresh3

Maximum number of entries in the IPv4 ARP table

Key: net.ipv4.neigh.default.gc_thresh3
Type:

integer
Default:

1024

Suggested value: 8192

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration. See ip-sysctl for more information.

net.ipv6.neigh.default.gc_thresh3

Maximum number of entries in IPv6 ARP table

Key: net.ipv6.neigh.default.gc_thresh3
Type:

integer
Default:

1024

Suggested value: 8192

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration. See ip-sysctl for more information.

vm.max_map_count

Maximum number of memory map areas a process may have

Key: vm.max_map_count
Type:

integer
Default:

65530

Suggested value: 262144

Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.