Benchmarking Network Throughput

installation

ubuntu,

apt install iperf3

slackware, with SCTP which gets enabled at compile time without any specific flag, as long as you got the headers available,

sbopkg -i lksctp-tools
sbopkg -i iperf3

from source – same thing, about the SCTP headers,

pkill iperf3
ps aux | grep iperf3
removepkg iperf3

wget https://github.com/esnet/iperf/archive/3.6.tar.gz
tar xzf 3.6.tar.gz
cd iperf-3.6/
./configure
grep -i sctp config.log
make -j16
make -j16 install
ldconfig

operations

start the server,

iperf3 -s

or in daemon mode,

echo -n iperf3...
/usr/bin/iperf3 -s -D -I /var/run/iperf3.pid && echo done
#/usr/local/bin/iperf3

run the client test,

iperf3 -c SERVER_ADDRESS
iperf3 --udp -c SERVER_ADDRESS
iperf3 --sctp -c SERVER_ADDRESS

iperf3 -P 3 -c SERVER_ADDRESS #1.88 vs 2.68 Gbit/s with balance-rr
iperf3 --udp -P 3 -c SERVER_ADDRESS #3.15 Git/s with balance-rr
iperf3 --sctp -P 3 -c SERVER_ADDRESS

use all cores

on the server,

iperf3 -s -p 5101 -D; iperf3 -s -p 5102 -D; iperf3 -s -p 5103 -D

on the client,

server=server
ksh -c "iperf3 -c $server -T s1 -p 5101 &; iperf3 -c $server -T s2 -p 5102 &; iperf3 -c $server -T s3 -p 5103" | egrep 'receiver$|sender$'
unset server

balance-rr TCP re-ordering

#net.ipv4.tcp_reordering = 3 #default
net.ipv4.tcp_reordering = 127

adjusting TCP window size

check the default values first,

sysctl net.core.wmem_max
sysctl net.core.rmem_max
sysctl net.ipv4.tcp_rmem
sysctl net.ipv4.tcp_wmem

sysctl net.ipv4.tcp_reordering

then change those,

vi /etc/sysctl.conf

net.core.wmem_max = 4194304
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304
#net.ipv4.tcp_reordering = 3
net.ipv4.tcp_reordering = 127

sysctl -p

iperf3 -P3 -c $server -w 8M

jumbo frames

check the current values first,

ifconfig bond0
ifconfig xenbr0
ifconfig eth0
ifconfig eth2
ifconfig eth3
#mtu 1500
#txqueuelen 1000

sysctl net.core.netdev_max_backlog

then edit your init scripts,

ifconfig bond0 mtu 9000 txqueuelen 10000
ifconfig xenbr0 mtu 9000 txqueuelen 10000
ifconfig eth0 mtu 9000 txqueuelen 10000
ifconfig eth2 mtu 9000 txqueuelen 10000
ifconfig eth3 mtu 9000 txqueuelen 10000
#ip link set

vi /etc/sysctl.conf

#net.core.netdev_max_backlog = 1000
net.core.netdev_max_backlog = 3000

sysctl -p

Note. all the NICs on that network should be switched to that MTU.

troubelshooting

Interestingly, when binding the endpoint to their respective interface using -B, LACP returns Retr retries to 0. That looks good. As for balance-rr that does not change anything. In fact, I got more shots with 5x the amount of normal retires when using -B.

results

balance-rr TCP 3 streams
[SUM]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec  2378             sender
[SUM]   0.00-10.04  sec  2.30 GBytes  1.97 Gbits/sec                  receiver

balance-rr UDP 3 streams
[SUM]   0.00-10.00  sec  3.76 MBytes  3.16 Mbits/sec  0.000 ms  0/441 (0%)  sender
[SUM]   0.00-10.04  sec  3.76 MBytes  3.14 Mbits/sec  0.037 ms  0/441 (0%)  receiver

balance-rr SCTP 3 streams
[SUM]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec                  sender
[SUM]   0.00-10.04  sec  1.15 GBytes   986 Mbits/sec                  receiver

balance-rr TCP 3 streams after changing the TCP window size and -w 8M
[SUM]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  3729             sender
[SUM]   0.00-10.04  sec  3.32 GBytes  2.84 Gbits/sec                  receiver

references

why tops 1Gbit/s per network flow?

alternatives