Benchmarking Network Throughput



apt install iperf3

slackware, with SCTP which gets enabled at compile time without any specific flag, as long as you got the headers available,

sbopkg -i lksctp-tools
sbopkg -i iperf3

from source – same thing, about the SCTP headers,

pkill iperf3
ps aux | grep iperf3
removepkg iperf3

tar xzf 3.6.tar.gz
cd iperf-3.6/
grep -i sctp config.log
make -j16
make -j16 install


start the server,

iperf3 -s

or in daemon mode,

echo -n iperf3...
/usr/bin/iperf3 -s -D -I /var/run/ && echo done

run the client test,

iperf3 --udp -c SERVER_ADDRESS
iperf3 --sctp -c SERVER_ADDRESS

iperf3 -P 3 -c SERVER_ADDRESS #1.88 vs 2.68 Gbit/s with balance-rr
iperf3 --udp -P 3 -c SERVER_ADDRESS #3.15 Git/s with balance-rr
iperf3 --sctp -P 3 -c SERVER_ADDRESS

use all cores

on the server,

iperf3 -s -p 5101 -D; iperf3 -s -p 5102 -D; iperf3 -s -p 5103 -D

on the client,

ksh -c "iperf3 -c $server -T s1 -p 5101 &; iperf3 -c $server -T s2 -p 5102 &; iperf3 -c $server -T s3 -p 5103" | egrep 'receiver$|sender$'
unset server

balance-rr TCP re-ordering

#net.ipv4.tcp_reordering = 3 #default
net.ipv4.tcp_reordering = 127

adjusting TCP window size

check the default values first,

sysctl net.core.wmem_max
sysctl net.core.rmem_max
sysctl net.ipv4.tcp_rmem
sysctl net.ipv4.tcp_wmem

then change those,

cat >> /etc/sysctl.conf <<-EOF
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304
sysctl -p

iperf3 -P3 -c $server -w 8M

jumbo frames

check the current values first,

ifconfig bond0
ifconfig xenbr0
ifconfig eth0
ifconfig eth2
ifconfig eth3

then edit your init scripts,

echo MTU 9000 on TRUNK
ip link set bond0 mtu 9000 txqueuelen 10000
ip link set xenbr0 mtu 9000 txqueuelen 10000
ip link set eth0 mtu 9000 txqueuelen 10000
ip link set eth2 mtu 9000 txqueuelen 10000
ip link set eth3 mtu 9000 txqueuelen 10000
#txqueuelen 1000 # default

#net.core.netdev_max_backlog = 1000 # default
net.core.netdev_max_backlog = 3000

Note. all the NICs on that network should be switched to that MTU.


Interestingly, when binding the endpoint to their respective interface using -B, LACP returns Retr retries to 0. That looks good. As for balance-rr that does not change anything. In fact, I got more shots with 5x the amount of normal retires when using -B.


balance-rr TCP 3 streams
[SUM]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec  2378             sender
[SUM]   0.00-10.04  sec  2.30 GBytes  1.97 Gbits/sec                  receiver

balance-rr UDP 3 streams
[SUM]   0.00-10.00  sec  3.76 MBytes  3.16 Mbits/sec  0.000 ms  0/441 (0%)  sender
[SUM]   0.00-10.04  sec  3.76 MBytes  3.14 Mbits/sec  0.037 ms  0/441 (0%)  receiver

balance-rr SCTP 3 streams
[SUM]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec                  sender
[SUM]   0.00-10.04  sec  1.15 GBytes   986 Mbits/sec                  receiver

balance-rr TCP 3 streams after changing the TCP window size and -w 8M
[SUM]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  3729             sender
[SUM]   0.00-10.04  sec  3.32 GBytes  2.84 Gbits/sec                  receiver


why tops 1Gbit/s per network flow?