Poor network performance via cilium - is this expected?

I’ve been testing network performance using iperf3 between pods on our DigitalOcean Kubernetes cluster, to try and aid in debugging issues we have seen around database throughput.

When testing with hostNetwork: true on the pods, I see an average bitrate of around 1.9GBits/sec between the node IPs. When using the kubernetes network, via Cilium, I only see around 600MBits/sec between the pod IPs.

From reading articles such as on Cilium’s blog I got the impression that the performance disparity should not be anywhere near as much. What is Cilium even actually doing here? :-)

I’m using the docker image mlabbe/iperf3 deployed as a DaemonSet, with the command iperf3 -c <ip address> -t 100

Kubernetes v1.15.5 - tried on two different DigitalOcean clusters with very similar results.


Asked the same question in Cilium’s super-helpful Slack group. This problem doesn’t seem to be related to CPU power, as the CPUs are 90% idle while the slow transfer is happening.

We tested using tcpdump to look for e.g. fragmentation, and the size of the packets received was quite different - consistently 65226 bytes during the hostNetwork test, but much, much smaller when routed via Cilium (mostly 2000-7000 bytes). Not sure why this should be!

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!