################################################################## BisonRouter L2TP Performance Lab: Design, Methodology, and Results ################################################################## Overview ******** This document describes an L2TPv2 LAC/LNS performance test lab built around BisonRouter. The goals of the lab are: - Validate L2TP LAC switching performance with 40k PPPoE subscribers. - Validate L2TP LNS termination performance with 40k L2TP subscribers. - Measure bidirectional subscriber traffic throughput through a full PPPoE-to-L2TP-to-IP forwarding chain. - Compare a near-lossless run with a maximum-load run. The lab emulates an ISP-style access topology in which PPPoE subscribers are terminated by a LAC and forwarded through an L2TP tunnel to a LNS. Topology ******** Physical layout The testbed consists of two hosts connected with 40G links: ger1 Traffic generator running BNG Blaster DPDK edition. ger1beast Device under test. The host runs two independent BisonRouter instances: one L2TP LAC instance and one L2TP LNS instance. Each instance uses a dedicated Intel XL710 40G NIC. Topology diagram: .. code-block:: text BNG Blaster (PPPoE) <-> BisonRouter LAC <-> BisonRouter LNS <-> BNG Blaster (network) ger1 ger1beast ger1beast ger1 Traffic path ************ Access-to-network direction: .. code-block:: text BNG Blaster PPPoE access -> BR LAC -> L2TP tunnel -> BR LNS -> BNG Blaster network Network-to-access direction: .. code-block:: text BNG Blaster network -> BR LNS -> L2TP tunnel -> BR LAC -> BNG Blaster PPPoE access In this setup each end-to-end stream packet is processed by both BisonRouter instances. The LAC handles PPPoE-to-L2TP switching, while the LNS terminates the L2TP session and forwards subscriber IP traffic. Therefore the BNG Blaster end-to-end stream rate is only one view of the test. The total packet processing work on ``ger1beast`` is higher because the same traffic crosses a BisonRouter LAC instance and a BisonRouter LNS instance. Hardware ******** BisonRouter host (ger1beast) ============================ - CPU: AMD Ryzen 9 9950X3D 16-Core Processor. - LAC NIC: Intel XL710 40G. - LNS NIC: Intel XL710 40G. CPU allocation on ger1beast =========================== The two BisonRouter instances were intentionally split across the host CPU resources: - LAC instance: 7 cores assigned. - LNS instance: 7 cores assigned. - In each instance, 6 out of 7 cores were forwarding traffic. This is important when interpreting the result. The test does not show a single BisonRouter instance using all available CPU resources. It shows two BisonRouter instances running at the same time, each limited to roughly one half of the available host resources. Generator host (ger1) ===================== - CPU: AMD Ryzen 9 5900XT 16-Core Processor. - NIC: Intel XL710 40G. Software ******** - BisonRouter L2TP LAC instance on ger1beast. - BisonRouter L2TP LNS instance on ger1beast. - BNG Blaster DPDK edition on ger1. Test methodology **************** Each test run consists of two stages: - Session establishment. - Bidirectional stream traffic forwarding. BNG Blaster establishes 40,000 PPPoE subscribers. The LAC forwards each PPP subscriber through L2TP to the LNS, resulting in 40,000 LAC sessions and 40,000 LNS sessions on a single L2TP tunnel. After all sessions are established, BNG Blaster verifies 80,000 traffic flows and starts bidirectional subscriber traffic. Two load levels were tested: Test 1 4,000,000 total stream packets per second. This run represents the near-lossless operating point. Test 2 8,000,000 total stream packets per second. This run pushes the lab to the maximum observed throughput and accepts measurable packet loss. Result summary ************** .. list-table:: :header-rows: 1 :widths: 16 18 18 18 18 18 * - Test - Offered stream rate - Sessions - Streams - Access loss - Network loss * - Test 1 - 4.0 Mpps - 40,000 - 80,000 - 2,336 packets, 0.001% - 1,838 packets, 0.000% * - Test 2 - 8.0 Mpps - 40,000 - 80,000 - 13,365,361 packets, 0.745% - 2,249,658 packets, 0.125% Test 1 is the clean performance result. It keeps all 40,000 subscribers established and forwards approximately 4 Mpps of end-to-end stream traffic with almost zero packet loss. Test 2 is the overload result. It reaches approximately 8 Mpps of end-to-end stream traffic, but packet loss becomes visible and BNG Blaster reports 43 flapped sessions. Host workload summary ===================== Because both the LAC and LNS run on ``ger1beast``, the host processes the LAC datapath and the LNS datapath at the same time. A 4 Mpps end-to-end test means approximately 4 Mpps through the LAC and another 4 Mpps through the LNS. .. list-table:: :header-rows: 1 :widths: 16 24 28 28 * - Test - End-to-end stream rate - Approximate per-instance forwarding - Approximate total on ger1beast * - Test 1 - 4.0 Mpps - 4.0 Mpps on LAC and 4.0 Mpps on LNS - 8.0 Mpps ingress and 8.0 Mpps egress * - Test 2 - 8.0 Mpps - 8.0 Mpps on LAC and 8.0 Mpps on LNS - 16.0 Mpps ingress and 16.0 Mpps egress The Test 1 result is therefore stronger than a simple 4 Mpps subscriber traffic number. It shows that the machine can forward at least the combined LAC and LNS workload shown above while using only 6 forwarding cores per BisonRouter instance. Test 1 run ********** BNG Blaster summary =================== .. code-block:: none F1: Select View F7/F8: Start/Stop Traffic F9: Terminate Sessions May 22 16:23:05.444925 Total PPS of all streams: 4000000.00 F2: Network Interface Left/Right: Access Interface May 22 16:23:05.552461 Resolve network interfaces May 22 16:23:05.552493 All network interfaces resolved ____ __ ____ _ __ ,/ May 22 16:24:12.197948 ALL SESSIONS ESTABLISHED / __ \ / /_ / __ ) _____ (_)_____ / /__ ,'/ May 22 16:24:12.447146 ALL STREAM TRAFFIC FLOWS VERIFIED / /_/ // __// __ |/ ___// // ___// //_/ ,' / / _, _// /_ / /_/ // / / // /__ / ,< ,' /_____, /_/ |_| \__//_____//_/ /_/ \___//_/|_| .'____ ,' ____ _ _ ______ ____ _ _ / ,' / __ ) / | / // ____/ / __ ) / /____ _ _____ / /_ ___ ____ / ,' / __ |/ |/ // / __ / __ |/ // __ `// ___// __// _ \ / ___/ /,' / /_/ // /| // /_/ / / /_/ // // /_/ /(__ )/ /_ / __// / / /_____//_/ |_/ \____/ /_____//_/ \__,_//____/ \__/ \___//_/ Test Duration: 244s All Sessions Established: 177s Sessions 40000 (40000 PPPoE / 0 IPoE) Established 40000 [############################################################] Outstanding 0 [ ] Terminated 0 [ ] DHCPv6 0/40000 [ ] Setup Time 66646 ms Setup Rate 600.19 CPS (MIN: 260.87 AVG: 601.66 MAX: 663.51) Flapped 0 Traffic Flows Verified Stream 80000/80000 [############################################################] Network Interface ( 0000:06:00.1 ) TX Packets 421544766 |2000191 PPS 4320411 Kbps 4.320 Gbps RX Packets 421542783 |2000244 PPS 4320528 Kbps 4.321 Gbps TX Stream Packets 421544728 |2000190 PPS RX Stream Packets 421542769 |2000244 PPS 1838 Loss (0.000%) TX Multicast Packets 0 | 0 PPS Access Interface ( 0000:06:00.0 ) TX Packets 424501849 |2011212 PPS 4587396 Kbps 4.587 Gbps RX Packets 422790890 |2003305 PPS 4578035 Kbps 4.578 Gbps TX Stream Packets 421544797 |2000187 PPS RX Stream Packets 421542229 |2000220 PPS 2336 Loss (0.001%) RX Multicast Packets 0 | 0 PPS 0 Loss BisonRouter LAC port rate ========================= Representative samples from the LAC instance: .. code-block:: none Port 0: RX 2,010,898.0 pps, TX 2,002,889.9 pps, Missed 0.0/sec Port 1: RX 2,002,885.7 pps, TX 2,010,895.8 pps, Missed 13.3/sec Port 0: RX 2,010,584.3 pps, TX 2,002,575.3 pps, Missed 0.0/sec Port 1: RX 2,002,569.8 pps, TX 2,010,580.4 pps, Missed 5.7/sec BisonRouter LNS port rate ========================= Representative samples from the LNS instance: .. code-block:: none Port 0: RX 2,010,796.8 pps, TX 2,002,965.4 pps, Missed 0.0/sec Port 1: RX 2,000,005.5 pps, TX 2,000,023.0 pps, Missed 0.0/sec Port 0: RX 2,010,749.5 pps, TX 2,003,023.5 pps, Missed 15.2/sec Port 1: RX 1,999,994.4 pps, TX 1,999,980.2 pps, Missed 0.0/sec Tunnel and session validation ============================= LAC tunnel: .. code-block:: none root@alex-gerbeast:~# rcli sh l2tp tunnels id our addr:port peer addr:port sessions uptime 1 1.0.0.1:1701 1.0.0.2:1701 40000 0 day(s), 0 hour(s), 4 min(s), 45 sec(s) LNS tunnel: .. code-block:: none root@alex-gerbeast:~# rcli2 sh l2tp tunnels id our addr:port peer addr:port sessions uptime 1 1.0.0.2:1701 1.0.0.1:1701 40000 0 day(s), 0 hour(s), 4 min(s), 48 sec(s) Subscriber counters: .. code-block:: none root@alex-gerbeast:~# rcli sh l2tp lac | wc -l 40001 root@alex-gerbeast:~# rcli2 sh l2tp lns | wc -l 40001 The output includes the header line, so both the LAC and LNS have 40,000 online subscriber sessions. Test 1 observations =================== At 4 Mpps of bidirectional stream traffic, both BisonRouter instances forward traffic steadily. The BNG Blaster loss counters remain effectively zero, and the BisonRouter missed counters are either zero or only a few packets per second on isolated samples. This is the near-lossless operating point for this lab configuration. Test 2 run ********** BNG Blaster summary =================== .. code-block:: none F1: Select View F7/F8: Start/Stop Traffic F9: Terminate Sessions May 22 16:33:47.804927 Total PPS of all streams: 8000000.00 F2: Network Interface Left/Right: Access Interface May 22 16:33:47.912367 Resolve network interfaces May 22 16:33:47.912400 All network interfaces resolved ____ __ ____ _ __ ,/ May 22 16:35:09.472641 ALL SESSIONS ESTABLISHED / __ \ / /_ / __ ) _____ (_)_____ / /__ ,'/ May 22 16:35:09.808527 ALL STREAM TRAFFIC FLOWS VERIFIED / /_/ // __// __ |/ ___// // ___// //_/ ,' / / _, _// /_ / /_/ // / / // /__ / ,< ,' /_____, /_/ |_| \__//_____//_/ /_/ \___//_/|_| .'____ ,' ____ _ _ ______ ____ _ _ / ,' / __ ) / | / // ____/ / __ ) / /____ _ _____ / /_ ___ ____ / ,' / __ |/ |/ // / __ / __ |/ // __ `// ___// __// _ \ / ___/ /,' / /_/ // /| // /_/ / / /_/ // // /_/ /(__ )/ /_ / __// / / /_____//_/ |_/ \____/ /_____//_/ \__,_//____/ \__/ \___//_/ Test Duration: 482s All Sessions Established: 400s Sessions 40000 (40000 PPPoE / 0 IPoE) Established 40000 [############################################################] Outstanding 0 [ ] Terminated 0 [ ] DHCPv6 0/40000 [ ] Setup Time 81561 ms Setup Rate 490.43 CPS (MIN: 490.43 AVG: 605.06 MAX: 1935.48) Flapped 43 Traffic Flows Verified Stream 80000/80000 [############################################################] Network Interface ( 0000:06:00.1 ) TX Packets 1793201556 |4000355 PPS 8640767 Kbps 8.641 Gbps RX Packets 1790951789 |3994777 PPS 8628718 Kbps 8.629 Gbps TX Stream Packets 1793201486 |4000355 PPS RX Stream Packets 1790951756 |3994777 PPS 2249658 Loss (0.125%) TX Multicast Packets 0 | 0 PPS Access Interface ( 0000:06:00.0 ) TX Packets 1798694099 |4011127 PPS 9163643 Kbps 9.164 Gbps RX Packets 1781730865 |3975553 PPS 9090853 Kbps 9.091 Gbps TX Stream Packets 1793201508 |4000324 PPS RX Stream Packets 1779828087 |3972645 PPS 13365361 Loss (0.745%) RX Multicast Packets 0 | 0 PPS 0 Loss BisonRouter LNS port rate ========================= Representative samples from the LNS instance: .. code-block:: none Port 0: RX 4,005,859.5 pps, TX 4,002,665.9 pps, Missed 4,204.6/sec Port 1: RX 3,999,978.5 pps, TX 3,995,345.5 pps, Missed 77.8/sec Port 0: RX 4,006,908.6 pps, TX 4,002,928.6 pps, Missed 3,536.7/sec Port 1: RX 3,999,906.4 pps, TX 3,995,989.2 pps, Missed 57.4/sec BisonRouter LAC port rate ========================= Representative samples from the LAC instance: .. code-block:: none Port 0: RX 4,010,657.6 pps, TX 3,999,073.0 pps, Missed 288.2/sec Port 1: RX 3,999,175.7 pps, TX 4,010,664.4 pps, Missed 3,771.8/sec Port 0: RX 4,010,094.7 pps, TX 3,997,657.7 pps, Missed 814.5/sec Port 1: RX 3,997,555.3 pps, TX 4,010,097.5 pps, Missed 5,377.3/sec Tunnel validation ================= After the maximum-load run, the LNS still has 40,000 sessions on the L2TP tunnel: .. code-block:: none root@alex-gerbeast:~# rcli2 sh l2tp tunnels id our addr:port peer addr:port sessions uptime 1 1.0.0.2:1701 1.0.0.1:1701 40000 0 day(s), 0 hour(s), 19 min(s), 20 sec(s) Test 2 observations =================== At 8 Mpps of bidirectional stream traffic, the lab reaches its maximum observed throughput. Both BisonRouter instances continue forwarding at approximately 4 Mpps per active port, but missed counters increase to thousands of packets per second on the busiest ingress ports. BNG Blaster reports measurable packet loss: - 0.125% on the network-side stream direction. - 0.745% on the access-side stream direction. BNG Blaster also reports 43 flapped sessions. This run should therefore be treated as an overload test, not as a lossless performance point. Conclusion ********** With 40,000 PPPoE subscribers forwarded through L2TP, this lab reached a stable near-lossless point at 4 Mpps total bidirectional stream traffic. At that rate, both the LAC and LNS maintained 40,000 sessions on one L2TP tunnel and packet loss stayed effectively zero. Since the LAC and LNS were both running on ``ger1beast``, this 4 Mpps end-to-end result corresponds to approximately 8 Mpps of ingress packet processing and 8 Mpps of egress packet processing across the host. Each BisonRouter instance used only 6 forwarding cores, so the result should be read as a lower-bound capacity result for this split-instance lab, not as a full-machine ceiling. The maximum-load run at 8 Mpps demonstrated that the same topology can forward substantially more traffic, but it crosses into overload. Packet loss becomes visible, missed counters rise on both BisonRouter instances, and session flaps appear. In terms of host workload, this run corresponds to approximately 16 Mpps ingress and 16 Mpps egress across the two BisonRouter instances. For this hardware and configuration, the practical clean result is the 4 Mpps run. The 8 Mpps run is useful as a stress boundary and shows where the lab starts to drop packets.