I’ve been testing network performance using iperf3 between pods on our DigitalOcean Kubernetes cluster, to try and aid in debugging issues we have seen around database throughput.

When testing with hostNetwork: true on the pods, I see an average bitrate of around 1.9GBits/sec between the node IPs. When using the kubernetes network, via Cilium, I only see around 600MBits/sec between the pod IPs.

From reading articles such as on Cilium’s blog I got the impression that the performance disparity should not be anywhere near as much. What is Cilium even actually doing here? :-)

I’m using the docker image mlabbe/iperf3 deployed as a DaemonSet, with the command iperf3 -c <ip address> -t 100

Kubernetes v1.15.5 - tried on two different DigitalOcean clusters with very similar results.

1 comment
  • Asked the same question in Cilium’s super-helpful Slack group. This problem doesn’t seem to be related to CPU power, as the CPUs are 90% idle while the slow transfer is happening.

    We tested using tcpdump to look for e.g. fragmentation, and the size of the packets received was quite different - consistently 65226 bytes during the hostNetwork test, but much, much smaller when routed via Cilium (mostly 2000-7000 bytes). Not sure why this should be!

Submit an answer

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!