================================= Installation Bison Router DPDK 21 ================================= A 30-day trial build of **Bison Router** is available only by request. Please `contact us `_. The following installation steps apply to the BisonRouter DPDK 21 build for Ubuntu 22.04. .. important:: The Main DPDK 21 build for Ubuntu 22.04 is now the preferred **Bison Router** build. - It supports Intel Ethernet 800 Series (E810) and 700 Series (X710) NICs. - It is not compatible with some older CPUs. If **Bison Router** fails to start with an ``illegal instruction`` error, switch to a build from the *Old CPUs* list. Update the system ================= To update the system and install Linux kernel headers, run: .. code-block:: none apt update apt-get install linux-headers-$(uname -r) Configure Linux kernel ====================== Edit ``GRUB_CMDLINE_LINUX`` variable in the ``/etc/default/grub``. .. code-block:: none GRUB_CMDLINE_LINUX="intel_idle.max_cstate=1 isolcpus=1-12 rcu_nocbs=1-12 rcu_nocb_poll=1-12 default_hugepagesz=2M hugepagesz=2M hugepages=4000 intel_iommu=on iommu=pt" .. note:: This is an example of syntax, use core numbers and the amount of hugepages according to the hardware configuration of your machine. .. note:: isolcpus, rcu_nocbs, and rcu_nocb_poll parameters must have identical values. .. note:: The ``intel_iommu=on iommu=pt`` parameters are required for the ``vfio-pci`` driver used by the DPDK 21 based builds. Run. .. code-block:: none update-grub .. note:: You may want to isolate a different set of cores or reserve different amount of RAM for huge pages, depending on the hardware configuration of your server. To maximize the performance of your system, always isolate the cores used by **Bison Router**. Enable VT-d / IOMMU in BIOS --------------------------- For the ``vfio-pci`` driver to work correctly, Intel VT-d (Virtualization Technology for Directed I/O) or IOMMU must be enabled in the BIOS/UEFI. Typical steps: - Reboot the server and enter the BIOS/UEFI Setup. - Locate the setting for **Intel VT-d** or **IOMMU**. - Common menu locations: *Chipset Configuration*, *System Agent (SA) Configuration*, *North Bridge*, or *Processor Configuration*. - Ensure VT-d/IOMMU is **Enabled**. - Save changes and reboot. Configure hugepages ------------------- Reboot your machine and check that hugepages are available and free. .. code-block:: none grep -i huge /proc/meminfo The expected result will have a structure similar to this example: .. code-block:: none HugePages_Total: 4000 HugePages_Free: 4000 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Make a mount point for hugepages. .. code-block:: none mkdir /mnt/huge Create a mount point entry in the ``/etc/fstab``. .. code-block:: none huge /mnt/huge hugetlbfs pagesize=2M 0 0 Mount hugepages. .. code-block:: none mount huge Verify kernel boot parameters ----------------------------- After reboot, verify that IOMMU parameters are present: .. code-block:: none cat /proc/cmdline Ensure that both ``intel_iommu=on`` and ``iommu=pt`` are listed in the output. Bison Router ============ Download Bison Router --------------------- Please, contact us at info@bisonrouter.net Install Bison Router -------------------- To install **Bison Router** use the following command: .. code-block:: none apt install ./bison-router-xxx.deb Install DPDK ------------ Use the ``bisonrouter`` utility to download and install DPDK. .. code-block:: none bisonrouter install_dpdk Configure DPDK ports -------------------- 1. Determine what NIC devices are available in your system. The following command will output information about NIC devices and their PCI addresses: .. code-block:: none bisonrouter dev_status 2. Edit ``/etc/bisonrouter/bisonrouter.env`` and save the NIC PCI addresses you want to use in the ``br_pci_devs`` list. For example: .. code-block:: none br_pci_devs=( "0000:04:00.0" "0000:04:00.1" ) 3. Set the DPDK device driver used by Bison Router to ``vfio-pci``: .. code-block:: none br_dev_driver="vfio-pci" 4. Run: .. code-block:: none bisonrouter bind_devices bisonrouter dev_status Now your PCI devices use DPDK drivers and are ready for **Bison Router**. The response to ``bisonrouter dev_status`` has the following form: .. code-block:: none Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe Prepare a configuration file ---------------------------- For configuration examples and options, see Command reference. First of all, you need to create ``/etc/bisonrouter/brouter.conf``. .. code-block:: none nano /etc/bisonrouter/brouter.conf Configure RSS (DPDK 21) ----------------------- .. warning:: **Bison Router** DPDK 21 based builds require **manual RSS configuration** for the DPDK ports. Missing RSS configuration will lead to serious performance issues and may cause errors in packet processing. Configure RSS according to the **Flow Director / RSS** documentation: - https://docs.bisonrouter.com/runtime_commands.html#flow-director-rss Configure CPU cores ------------------- Edit ``/etc/bisonrouter/bisonrouter.env`` and update the variable ``br_lcores`` according to your hardware setup. Use only the cores from the same NUMA socket if your system has multiple CPUs. A command that outputs information about available cores: .. code-block:: none bisonrouter cpu_layout Example of ``bison router cpu_layout`` output: .. code-block:: none # bisonrouter cpu_layout ====================================================================== Core and Socket Information (as reported by '/sys/devices/system/cpu') ====================================================================== cores = [0, 1, 2, 3] sockets = [0] Socket 0 -------- Core 0 [0] Core 1 [1] Core 2 [2] Core 3 [3] .. note:: On servers with more than one CPU socket (NUMA systems), BisonRouter must use CPU cores from only one NUMA socket. Using cores from two or more sockets at the same time is not supported. The NUMA socket you choose must match where your network card (NIC) is connected. For example, if the NIC is in a PCI slot that belongs to NUMA socket 0, then you should use only CPU cores from NUMA socket 0 for BisonRouter. The command ``bisonrouter dev_numa_info`` displays the NUMA configuration for the NICs used by BisonRouter. It shows each NIC's PCI address and the NUMA node (CPU socket) it is attached to. .. code-block:: none Device NUMA 0000:01:00.0 -1 - Values ``0``, ``1``, etc. indicate the NUMA node. - A value of ``-1`` means the kernel does not report a node for this device. This is normal on many single-socket systems, but on multi-socket systems it may mean the mapping is not available. If the NUMA value is ``-1`` (unknown), you can try to obtain additional information using: .. code-block:: none numactl --hardware If this still does not provide a clear mapping, you may need to check the physical PCI slot on the motherboard and consult the motherboard manual to determine which CPU socket the slot is connected to. After you know which NUMA socket the NIC uses (for example, socket 0), use the ``cpu_layout`` output to choose the CPU cores for Bison Router. In the ``cpu_layout`` table: - The column titles "Socket 0", "Socket 1", etc. show the NUMA socket. - The numbers in square brackets ``[ ]`` are the CPU core numbers. For example, if your NIC is on NUMA socket 0, then you must select only the core numbers in the "Socket 0" column (the numbers in ``[ ]`` under "Socket 0") when you configure: - ``isolcpus`` in the kernel command line - ``br_lcores`` in ``/etc/bisonrouter/bisonrouter.env`` Do not mix cores from different sockets in ``br_lcores``. Set values according to your ``bisonrouter cpu_layout`` output: .. code-block:: none br_lcores=' lcores at cpu cores ' .. warning:: Bison Router must never use CPU cores from more than one NUMA socket at the same time. All cores listed in ``br_lcores`` must belong to a single NUMA socket, and this socket must be chosen based on the NIC connection (see ``bisonrouter dev_numa_info``). The format for ``br_lcores`` consists of a positive integer ```` followed by the literal text ``lcores at cpu cores`` and a comma-separated list of CPU core subranges. Each subrange must be specified in the form ``X-Y``, where ``X`` is the starting CPU core number and ``Y`` is the ending CPU core number (inclusive). All CPU core numbers must be unique and the total number of CPU cores specified across all subranges must equal ````. CPU core numbers must lie between 0 and 128. For example, .. code-block:: none br_lcores='16 lcores at cpu cores 1-5,7-8,20-28' assigns 16 lcores (numbered 0 to 15) to CPU cores 1 through 5, 7 through 8, and 20 through 28, while .. code-block:: none br_lcores='4 lcores at cpu cores 1-4' assigns 4 lcores (numbered 0 to 3) to CPU cores 1 through 4. Run **Bison Router** -------------------- Start **Bison Router**. .. code-block:: none bisonrouter start Check the ``syslog`` to ensure that **Bison Router** has started successfully. .. code-block:: none ROUTER: router configuration file '/etc/bisonrouter/brouter.conf' successfully loaded Use the ``rcli`` utility to configure and control **Bison Router**. .. code-block:: none # rcli sh uptime Uptime: 0 day(s), 1 hour(s), 38 minute(s), 14 sec(s) To stop/restart **Bison Router** also use the ``bisonrouter`` utility which supports the following options: .. code-block:: none # bisonrouter Usage: bisonrouter [OPTIONS] Options: start - start the Bison Router (BR) daemon stop - stop the BR daemon restart - restart the BR daemon status - show Bison Router daemon status bind_devices - load kernel modules and bind PCI devices unbind_devices - unload kernel modules and unbind devices dev_status - show device status cpu_layout - show core and socket information install_dpdk - fetch the DPDK source code and install reinstall_dpdk - remove the current DPDK and reinstall