Boot Settings (Grub)
Grub settings:
IOMMU (Input–Output Memory Management)
Some hardware, including the Mellanox/Nvidia 100G NICs, support IOMMU. While IOMMU is particularly important in a virtualized environment, we've seen parallel stream throughput go from 80 Gbps to 180 Gbps on non-virtualized hosts as well. We recommend enabling this for any host with a NIC 100 Gbps or higher.
iommu support must be configured in grub.
Add this line to the file: /etc/default/grub
GRUB_CMDLINE_LINUX="iommu=pt"
You might also want to try these settings:
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt"
or
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"
Depending on your host architecture.
Then do:
/usr/sbin/update-grub
and reboot.
To confirm the setting worked, do:
cat \/proc\/cmdline
and you should see something like:
root=/dev/mapper/vg0-root ro iommu=pt
Other Grub Settings: mitigations
Another option you might consider added to GRUB_CMDLINE_LINUX in /etc/defaults/grub is to disable security mitigations. e.g.:
GRUB_CMDLINE_LINUX="iommu=pt mitigations=off"
We've seen over 30% increased throughput with this setting! On ESnet test hosts, single stream iperf3 went from 43 Gbps to 57 Gbps just by changing this.
Check with your security team before doing this. Setting mitigations=off in GRUB disables all security mitigations for various CPU vulnerabilities, such as Spectre, Meltdown, L1TF, MDS, and others. While this can improve system performance, it exposes the system to several security risks. These vulnerabilities exploit speculative execution to access unauthorized memory, potentially leaking sensitive data such as passwords or encryption keys. Only consider this on hosts where you trust all users, and the system operates in a tightly controlled environment.
For more information, see:
