How to Diagnose and Fix Asymmetric Network Performance Issues

Introduction

Network connectivity issues can be challenging to diagnose, especially when dealing with asymmetric performance, latency variations, and unexpected bandwidth limitations. Identifying the root cause requires a structured approach using diagnostic tools and network analysis techniques. This tutorial walks you through the process of detecting and resolving network asymmetry issues by analyzing performance metrics, running network tests, and applying optimization techniques.

This tutorial is for system administrators, network engineers, DevOps professionals, and anyone responsible for maintaining network infrastructure. Whether you’re managing cloud resources, on-premises servers, or hybrid environments, you’ll learn practical skills to identify and fix asymmetric network issues that can impact application performance and user experience. By following this tutorial, you’ll be able to restore balanced network operations, reduce troubleshooting time, and ensure optimal performance across your infrastructure.

Prerequisites

Please ensure you have the following prerequisites before you begin:

• A basic understanding of networking concepts such as latency, bandwidth, and packet loss.
• Access to servers or cloud instances where network diagnostics can be run.
• Familiarity with network testing tools such as speedtest-cli, mtr, and iperf3.
• Administrator or root access to run diagnostic commands on the affected systems.

Step 1 - Identifying the Issue

When diagnosing network performance issues, the initial step is to establish a baseline. For demonstration purposes you can use a pure bare metal GPU node and a Premium Dedicated Droplet (PDD) in the same AMS region.

This involves measuring the maximum throughput of the network between the nodes.

The expected throughput is 10 Gbps, which is the limit on a Premium Dedicated Droplet (PDD).

Test Procedure:

1. Provision a pure bare metal GPU node in the AMS region.
2. Create a Premium Dedicated Droplet (PDD) in the AMS region.
3. Run an iperf3 server instance on bare metal GPU node.
4. Run an iperf3 client instance on Premium Dedicated Droplet (PDD).

Using iperf3 for Bandwidth Testing

To measure real-time bandwidth, use iperf3:

On the bare metal GPU node (acting as the iperf3 server) use the following command to start the server:

iperf3 -s

On Premium Dedicated Droplet (PDD) (acting as the iperf3 client) run the following command to start the client:

iperf3 -c <server-b-ip> -P 10 -t 30

Where:

• -c <server-b-ip> → Client mode: Specifies the target server for the test.

• -P 10 → Parallel connections: Runs 10 parallel streams for the test, simulating multiple simultaneous connections.

• -t 30 → Test duration: Runs the test for 30 seconds.

This will provide a detailed view of available bandwidth and potential bottlenecks.

Step 2 - Running Advanced Network Diagnostics

Using mtr for Network Tracing

To further investigate, use My Traceroute to identify network routing issues.

Example: From bare metal GPU node → Premium Dedicated Droplet (PDD)

mtr -rwbzc100 <server-a-ip>

Where:

• -r → Report mode: Runs mtr in report mode, meaning it collects statistics and exits instead of running interactively.

• -w → Wide report format: Displays additional statistics in the output.

• -b → Show both IPs and hostnames: Includes both IP addresses and resolved hostnames in the output.

• -z → Show AS numbers: Displays the Autonomous System (AS) numbers for each hop (if available).

• -c100 → Run 100 queries: Sends 100 probe packets to each hop and averages the results.

Look for packet loss or high latency at intermediate hops, which could indicate congestion or misrouting.

Example: From Premium Dedicated Droplet (PDD) → bare metal GPU node

mtr -rwbzc100 <server-b-ip>

If a loss occurs at a specific hop and continues beyond that point, it may indicate a problematic router in the path.

Step 3 - Diagnosing the Root Cause

Potential Causes for Asymmetry

• TCP Connection Pooling and Thread Limits: Speedtest clients often use more threads for downloading than for uploading, which can affect test results. For example, the default configuration of speedtest-cli uses 34 parallel TCP connections for downloads but only 6 for uploads.

• MTU (Maximum Transmission Unit) Misconfiguration: A high MTU setting can cause packet fragmentation, leading to performance degradation. Additionally, VPN tunnels like WireGuard introduce overhead, making an MTU reduction necessary.

• Network Provider Policies & Peering Agreements: Some ISPs and cloud providers implement traffic shaping that prioritizes downloads over uploads, while interconnection agreements between ISPs can also impact routing performance.

Step 4 - Implementing Fixes

• Fix 1: Adjust MTU Settings For VPN tunnels (WireGuard), reducing the MTU can improve performance. Ensure you set the optimum MTU first; You can refer to nr-wg-mtu-finder and Optimal MTU for WireGuard for guidance.

What is MTU?

MTU stands for Maximum Transmission Unit, which is the maximum size of a data packet that can be transmitted over a network interface. It is a critical parameter in network communication as it determines the maximum amount of data that can be sent in a single packet. A higher MTU allows for larger packets, which can improve network performance by reducing the number of packets sent. However, if the MTU is set too high, it can lead to packet fragmentation, which can negatively impact network performance.

What is WireGuard?

WireGuard is a fast, modern, and secure VPN (Virtual Private Network) solution that aims to replace traditional VPN solutions like OpenVPN and IPSec. It is designed to be easy to use, highly performant, and secure. WireGuard uses state-of-the-art cryptography and is optimized for modern network architectures. It is widely used for creating secure, point-to-point connections between devices over the internet.

How to set MTU for WireGuard?

To set the MTU for WireGuard, you need to run the following commands on both the bare metal GPU node and the Premium Dedicated Droplet (PDD).

ip link set dev wg0 mtu 1400

This command sets the Maximum Transmission Unit (MTU) of the network interface wg0 to 1400. The MTU is the maximum size of a data packet that can be transmitted over a network interface. Setting the MTU to 1400 ensures that packets sent over this interface do not exceed this size, which can help prevent packet fragmentation and improve network performance.

Or lower if necessary:

ip link set dev wg0 mtu 1300

This command sets the MTU of the network interface wg0 to 1300, which is a lower value than the previous command. This may be necessary if the network infrastructure or VPN tunnel requires a lower MTU to function properly.

• Fix 2: Use an Alternative Testing Tool (iPerf3) Instead of relying on speedtest-cli, use iperf3 for more controlled testing. Run the following command on the Premium Dedicated Droplet (PDD) as the client and the bare metal GPU node as the server:

On the client (PDD):

iperf3 -c <server-b-ip> -P 10 -t 60

This command runs iperf3 to perform a network bandwidth test. The options used are:

• -c <server-b-ip>: specifies the server to connect to, where <server-b-ip> is the IP address of the server.
• -P 10: sets the number of parallel streams to 10, which can help to saturate the network link and provide a more accurate measurement of bandwidth.
• -t 60: sets the duration of the test to 60 seconds, allowing for a longer test period to gather more data and ensure a stable measurement.

On the server (bare metal GPU node):

iperf3 -s

This provides more reliable results than commercial speed test servers.

• Fix 3: Contact ISP and Cloud Provider if the issue persists, check with the ISP for any traffic shaping policies that might be affecting upload speeds. Request a route optimization check to ensure the most efficient path between the data centers.

Step 5 - Re-testing Performance

After applying the fixes, run the tests again and observe:

• Upload speeds should improve significantly.
• Packet loss in mtr results should be minimized.
• Network asymmetry should be reduced.

FAQs

1. What is the ideal MTU for WireGuard?

The ideal MTU for WireGuard depends on the network infrastructure and VPN tunnel requirements. A common value is 1400, but it may need to be lowered to 1300 or even lower in some cases.

2. How do I know if I need to adjust the MTU for WireGuard?

If you’re experiencing packet fragmentation or network performance issues, adjusting the MTU might be necessary. A high MTU can lead to packet fragmentation, which can cause performance issues, while a low MTU can result in increased overhead due to smaller packet sizes. The optimal value for MTU is typically between 1300 and 1400, as this range balances packet size and fragmentation avoidance for optimal network performance and bottleneck minimization. However, consult your network infrastructure and VPN tunnel documentation for specific requirements to determine the optimal MTU for your setup.

3. What is the difference between speedtest-cli and iperf3 for network testing?

speedtest-cli is a tool for testing internet bandwidth using commercial speed test servers, which is ideal for measuring internet connectivity and speed. On the other hand, iperf3 is a more controlled tool for measuring network bandwidth between two points, making it suitable for internal network testing, such as measuring the performance of a VPN tunnel or a local network. iperf3 provides more reliable results for internal network testing due to its ability to control the test environment and eliminate external factors that might affect the results.

4. How do I run iperf3 for network testing?

Run iperf3 -c <server-b-ip> -P 10 -t 60 on the client side and iperf3 -s on the server side. This will perform a 60-second network bandwidth test with 10 parallel streams.

5. What should I do if the issue persists after applying the fixes?

If the issue persists, contact your ISP to investigate potential traffic shaping policies that might be impacting upload speeds. Request a route optimization check to ensure the most efficient path between data centers.

6. How do I know if the fixes have been successful?

After applying the fixes, re-run the tests and observe improvements in upload speeds, minimized packet loss in mtr results, and reduced network asymmetry. This can be done by comparing the results before and after the fixes. If the upload speeds have significantly improved, the packet loss has been minimized, and the network asymmetry has been reduced, the fixes have been successful.

Conclusion

This comprehensive tutorial has walked you through a structured approach to identify and address asymmetric network performance issues. By following the steps outlined, you’ve learned how to analyze network performance using tools like iperf3 and mtr, diagnose common issues such as packet loss and network asymmetry, and apply targeted fixes to improve upload speeds and overall network reliability. With these skills, you’re now equipped to tackle network performance problems with confidence, ensuring smoother data transfer and more efficient communication across your network.

For more in-depth knowledge on networking concepts and troubleshooting, check out these DigitalOcean tutorials:

1. An Introduction to Networking Terminology, Interfaces, and Protocols
2. Understanding IP Addresses, Subnets, and CIDR Notation
3. How To Use Traceroute and MTR to Diagnose Network Issues
4. How To Set Up WireGuard on Ubuntu
5. Network latency: common causes & proven tactics to reduce it

These resources provide valuable insights into network configuration, troubleshooting, and optimization techniques that complement the strategies discussed in this guide.