Flowcontrol, port buffers and dropped packets

A few days ago a customer called worried by the flow control counters increasing on his Cisco 4510 switch, he just sent me this asking for an explanation:

CORE_4510#show interfaces flowcontrol
Port       Send FlowControl  Receive FlowControl  RxPause TxPause
admin    oper     admin    oper
---------  -------- -------- -------- --------    ------- -------
Gi1/1      on       on       desired  on          28972   107274064
Gi1/2      on       on       desired  on          9494    111534
Gi1/3      on       on       desired  on          32580   1406178
Gi1/4      on       on       desired  on          1278    84112
...(cut for brevity)...

RxPause: Number of Pause frames received.

TxPause: Number of Pause frames transmitted.

Flow control is not rocket science, wikipedia explains it clearly, the concept is: if you’re receiving more data than you can handle ask the sender to take a little break, if you receive a pause request be polite and stop sending data for a little while.

In our case the counter increasing fast was TxPause, that means the switch is receiving more data than it can handle.

Is the switch dropping packets?

We can check it with the command:

show interfaces gigabitEthernet {mod/port} counters detail

and look this counters:

Rx-No-Pkt-Buff

If the counter is 0 that means the sender saw the Pause frame sent by the switch and actually paused. It the counter increments that means the Pause frame was either not received or ignored and the sender kept sending traffic.

In this case all the Rx-No-Pkt-Buff were zero.

So the switch is not dropping packets, that is good.

What about performances?

We can say that since the sender is pausing from time to time it’s not getting the full throughput it could reach so we have somehow a bottleneck or at least non optimal network performances. In this case the ports are connected to iSCSI storage and the counters were increasing during backups at night.

Why is it happening?

The source cause is the switch module architecture, in this case a WS-X4548-RJ45:

We can see that this module has six group of 8 ports sharing a 1Gpbs ASIC per group, that is a 8:1 over subscription. So if the total received traffic of the eight ports reaches 1Gpbs the switch sends TxPause frames (if configured to do so) or eventually drops the frames when the buffers are full.

How can we fix that?

There’re two possible ways.

The first way is to distribute the links that are more likely to send or receive larger amounts of data in different port groups, served by different 1Gbit ASICs:

Ports 1, 2, 3, 4, 5, 6, 7, 8
Ports 9, 10, 11, 12, 13, 14, 15, 16
Ports 17, 18, 19, 20, 21, 22, 23, 24
Ports 25, 26, 27, 28, 29, 30, 31, 32
Ports 33, 34, 35, 36, 37, 38, 39, 40
Ports 41, 42, 43, 44, 45, 46, 47, 48

Since in this case there are many port channels, we distributed the links on different port groups and modules.

The second solution is to replace the 8:1 oversubscribed module with another module WS-x4748-RJ45-E that is non-blocking:

We should not forget to check the switch backplane to spot another possible over subscription point, in this case a WX-C4510R-E. The architecture of the chassis provides 24Gpbs per module:

Even with a 48Gpbs port non blocking module, the over subscription from module to chassis would be 2:1. Just be aware.

The last step is the supervisor. This chassis has a Sup7-E that provides up to 848 Gpbs so there’s plenty of space to grow:

Wrap up

Devices architecture is often seen as a topic for presales/network designer. Post-sales network engineers tend to focus on the CLI skipping all the boring slides full of numbers.

In this case knowing the architecture really helped to understand and explain a potential problem.