In my new role as Technical Account Manager I work with the customers to evaluate new products and services to validate the fullfillment of the business requirements.

For a recent project I needed to test how some SDWAN appliances reacted to bad network events like packet drops and latency.

As usual, the timeline was very challenging so any solutions too complex or expensive was excluded.

For cases like this I usually search for some open source tools and some Linux magic and once again the solution was there with tc .

The lab

Hardware:

Software:

I will not discuss the details of the installation of Raspbian and tc, there are plenty of guides avaialble online.

For the SDWAN PoC I’ve used two RaspberryPi connected to WAN1 and WAN2 of the SDWAN appliance, each one to bridge a single WAN link to the CE routers.

With more USB/ethernet adapters available just one Raspberry would’ve been enough. When life gives you lemons …

Be the bridge

The usage of the tools is very simple: we create a bridge between eth0 and eth1 in the RaspberryPi and then apply some tc commands to add latency or drop packets going over the bridge.

Let’s start creating the bridge:

sudo brctl addbr br0
sudo brctl addif br0 eth0
sudo brctl addif br0 eth1
ifconfig br0 up

We can assign an IP address to the bridge via DHCP:

dhclient br0

In my case I’ve connected a monitor and keyboard to the RaspberryPi so an IP address was not required.

Enter tc

With the bridge in place we can start working with tc to add latency and drop some packets.

I share below some simple examples of tc from my notes that can be used as a starting point.

Add 400ms of latency:

sudo tc qdisc add dev eth0 root netem delay 400ms

Introduce a 25% packet loss:

sudo tc qdisc add dev eth0 root netem loss 25%

Some examples of policing and latency:

sudo tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms
sudo tc qdisc add dev eth0 root tbf rate 2mbit burst 32kbit latency 600ms
sudo tc qdisc add dev eth0 root tbf rate 50mbit burst 32kbit latency 400ms

Remove the tc configuration before applying the new settings:

sudo tc qdisc del dev eth0 root netem

Show the applied configuration:

tc qdisc show  dev eth0

Apply a variable delay and rate limit combined:

sudo tc qdisc add dev eth0 root netem delay 400ms 100ms 25% rate 2Mbit

Test results

A quick test to validate that tc works as expected.

Ping an host in the same network to get a baseline:

sudo hping3 --icmp -c 100 --fast 10.1.0.20
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 11.5/21.9/36.9 ms

Let’s add 400ms of latency and repeat the test:

100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 411.3/421.9/448.8 ms

Latency increased, good!

Now clear the config:

sudo tc qdisc del dev eth0 root

and apply 25% packet loss:

sudo tc qdisc add dev eth0 root netem loss 25%

Result:

100 packets transmitted, 75 packets received, 25% packet loss
round-trip min/avg/max = 11.2/21.5/33.5 ms

That’s pretty accurate!

I leave to the reader other combinations of rate-limiting and variable latency, you got the concept ✌

Conclusion

This post shows a basic use of tc applied to a specific use case to test SDWAN appliances. The thresholds of the appliances was tuned to react to specific values of latency or packet loss.

During the actual tests I’ve used PingPlotter to clearly show the moment when the traffic was re-routed to another tunnel to satisfy the configured SLA.

Update

Only after the POC I noticed the existence of tcconfig that would have made the process much easier.