How to tune IRQ affinity¶
IRQ (Interrupt Request) affinity refers to assigning interrupt service routines (ISRs) to specific processor cores in a computer system. Having IRQ affinity rightly set up becomes crucial for improved performance and responsiveness in a real-time system. By tuning IRQ affinity, you dedicate specific interrupts to particular CPU cores, preventing interruptions from being handled by multiple cores simultaneously. The tuning enhances efficiency, reduces latency, and ensures more predictable and consistent processing times for time-sensitive tasks in real-time systems. Proper IRQ affinity configuration helps to decrease contention, manage resource usage, and ultimately improve the overall responsiveness and reliability of the system, especially in scenarios where time precision is critical.
The most common kernel parameters to handle IRQ interrupts are:
kthread_cpus which defines the CPUs that kernel threads are allowed to run on
isolcpus used to specify CPUs to be isolated from the general SMP balancing and scheduler algorithms.
The IRQ affinity on the other hand is a way to tune the system interruptions without modifying the kernel boot parameters. This is the focus of this document. With IRQ affinity, you can set a default set of CPUs which are permitted to handle incoming IRQs.
First check the interruption sources in the system, all the IRQs are listed in
the /proc/interrupts file.
cat /proc/interrupts
It’s useful to monitor the IRQs while tuning the affinity. You can use the
watch command to monitor the IRQs in real-time:
watch -n 1 cat /proc/interrupts
You can list the isolated CPUs:
cat /sys/devices/system/cpu/isolated
An empty output means that no core is isolated.
Conversely, you can list all the available CPUs:
$ cat /sys/devices/system/cpu/present
0-19
In this case, the system has 20 CPUs available.
The best way to tune a system is to isolate one or more CPUs to be used to run the real-time application and the others to handle the IRQs and kthreads.
To check which CPUs are set for handling the IRQ, read the smp_affinity file
for each IRQ:
$ cat /proc/irq/<IRQ-NUMBER>/smp_affinity
ffffff
The smp_affinity file is a bitmask, where each bit represents a CPU, where
1 means that the CPU is allowed to handle the IRQ and 0 means that the CPU
is not allowed to handle the IRQ. An output of ffffff means that all CPUs are
allowed to handle the IRQ.
To list all the IRQs associated with a given CPU, you can use the
check_irqs.sh script:
$ ./check_irqs.sh 13
IRQ 0 is associated with Core 13. Affinity Mask: fffff
IRQ 1 is associated with Core 13. Affinity Mask: fffff
IRQ 10 is associated with Core 13. Affinity Mask: fffff
IRQ 11 is associated with Core 13. Affinity Mask: fffff
IRQ 12 is associated with Core 13. Affinity Mask: fffff
IRQ 120 is associated with Core 13. Affinity Mask: fffff
IRQ 121 is associated with Core 13. Affinity Mask: fffff
IRQ 13 is associated with Core 13. Affinity Mask: fffff
IRQ 14 is associated with Core 13. Affinity Mask: fffff
IRQ 141 is associated with Core 13. Affinity Mask: 02000
IRQ 15 is associated with Core 13. Affinity Mask: fffff
IRQ 150 is associated with Core 13. Affinity Mask: fffff
IRQ 16 is associated with Core 13. Affinity Mask: fffff
IRQ 164 is associated with Core 13. Affinity Mask: 02000
IRQ 167 is associated with Core 13. Affinity Mask: fffff
IRQ 17 is associated with Core 13. Affinity Mask: fffff
IRQ 2 is associated with Core 13. Affinity Mask: fffff
IRQ 3 is associated with Core 13. Affinity Mask: fffff
IRQ 4 is associated with Core 13. Affinity Mask: fffff
IRQ 5 is associated with Core 13. Affinity Mask: fffff
IRQ 6 is associated with Core 13. Affinity Mask: fffff
IRQ 7 is associated with Core 13. Affinity Mask: fffff
IRQ 8 is associated with Core 13. Affinity Mask: fffff
IRQ 9 is associated with Core 13. Affinity Mask: fffff
Then you can rewrite the smp_affinity file to set the IRQ to be handled by the
CPUs you want.
Tip
Since kernel 3.0 it’s possible to use the
/proc/irq/<IRQ-NUMBER>/smp_affinity_list. For example, to
set the IRQ 16 to be handled by the CPUs 0-12 and 14-19 (excluding the
CPU 13), run:
echo 0-12,14-19 > /proc/irq/16/smp_affinity_list
$ cat /proc/irq/0/smp_affinity_list
0-12,14-19
Do this for all the IRQs that are being handled by the CPUs that you want to isolate.
Note
The changes made on the /proc filesystem are not persistent, meaning that
the changes will be lost after a reboot. To make the changes persistent, you
can set the irqaffinity parameter as a persistent parameter as
described in How to modify kernel boot parameters.
For example, to isolate the CPU 13 in a system with 20 cpus and leave the IRQs
to be handled by the CPUs 0-12 and 14-19, you can add the following:
irqaffinity=0-12,14-19.
Note that unlike the changes made to /proc/irq, the setting passed as
kernel command line applies to all IRQs.
Warning
It’s not allowed to turn off all CPUs for a given IRQ, meaning that you
should ensure every IRQ is handled by at least one CPU. In other words, the
smp_affinity mask should never be 0.
Now you can run your real-time application in the isolated CPUs and check if the IRQs are being handled by the CPUs that you want.
taskset -c <CPU-NUM[s]> <COMMAND-TO-REAL-TIME-APP>
Or attaching to an already running process:
taskset -pc <CPU_NUM[s]> <PID>
Then, you can check if the application is correctly running on the designated CPU cores:
ps -eo psr,tid,pid,comm,%cpu,priority,nice -T | grep <PID>
It’s also important to disable the irqbalance service, which is responsible for
distributing IRQs across all available cores. To do so, you can run:
systemctl disable irqbalance
systemctl stop irqbalance
systemctl status irqbalance
Lastly, it’s useful to keep the systemd services separated from the real-time
application. You can do this by setting the CPUAffinity parameter in the
/etc/systemd/system.conf file to the cores you want to isolate. For example:
$ cat /etc/systemd/system.conf | grep CPUAffinity
CPUAffinity=0,1