Crazy slow slow network performance from NYC2 to Amazon Oregon region

November 11, 2013 6.7k views
administrator
By:
administrator
I have a server in NYC2. I regularly back up 150 MB of data from NYC2, to Amazon S3, in their Oregon region. It’s very slow. I ran these tests AFTER this morning’s NTT network changes. There was no improvement. 150 MB upload from VPS to an Amazon S3 bucket in the us-west-2 (Oregon) region * Digital Ocean, from NYC2 (2 GB instance): 11:50 * Digital Ocean, from SFO1 (2 GB instance): 3:35 * Linode, from New Jersey (1 GB instance): 0:25 For this upload, DO/NYC2 is 25-30 times slower than Linode/NJ. What’s going on?
Log In to Comment
5 Answers
moisey MOD November 12, 2013
Hi,

Please provide the traceroutes from each location so we can review, without that information there isn't much info that we can troubleshoot against.

Thanks
Reply
bryan November 12, 2013
Google "bandwidth delay product" and tune your tcp/ip stacks for the desired performance that you want. DigitalOcean uses default settings. You can get any host to go faster or slower on transfers to anywhere with a little tuning.
Reply
administrator November 12, 2013
From DO/NYC2 --> Amazon S3 us-west-2

# traceroute [bucket name redacted].s3.amazonaws.com
traceroute to [bucket name redacted].s3.amazonaws.com (54.240.248.89), 30 hops max, 60 byte packets
1 192.241.180.253 (192.241.180.253) 3.921 ms 3.841 ms 13.240 ms
2 192.241.164.241 (192.241.164.241) 0.304 ms 0.291 ms 0.258 ms
3 xe-0-3-0-21.r05.nycmny01.us.bb.gin.ntt.net (129.250.192.17) 1.019 ms 2.080 ms 2.055 ms
4 ae-1.r22.nycmny01.us.bb.gin.ntt.net (129.250.4.172) 0.475 ms 0.471 ms 0.454 ms
5 ae-4.r21.sttlwa01.us.bb.gin.ntt.net (129.250.2.51) 74.890 ms 68.863 ms 74.843 ms
6 ae-2.r04.sttlwa01.us.bb.gin.ntt.net (129.250.5.45) 69.168 ms ae-2.r05.sttlwa01.us.bb.gin.ntt.net (129.250.5.49) 75.396 ms ae-2.r04.sttlwa01.us.bb.gin.ntt.net (129.250.5.45) 69.402 ms
7 ae-5.amazon.sttlwa01.us.bb.gin.ntt.net (198.104.202.190) 74.529 ms ae-3.amazon.sttlwa01.us.bb.gin.ntt.net (198.104.202.182) 80.125 ms ae-2.amazon.sttlwa01.us.bb.gin.ntt.net (129.250.201.178) 74.530 ms
8 205.251.226.231 (205.251.226.231) 91.704 ms 205.251.226.179 (205.251.226.179) 79.510 ms 205.251.226.191 (205.251.226.191) 79.062 ms
9 205.251.232.88 (205.251.232.88) 81.781 ms 87.901 ms 87.853 ms
10 205.251.232.159 (205.251.232.159) 87.892 ms 205.251.232.147 (205.251.232.147) 96.052 ms 205.251.232.141 (205.251.232.141) 79.796 ms
11 205.251.232.63 (205.251.232.63) 79.536 ms 79.614 ms 81.832 ms
12 54.240.230.165 (54.240.230.165) 79.738 ms 91.750 ms 54.240.230.167 (54.240.230.167) 88.081 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

From DO/SFO1 --> Amazon S3 us-west-2

# traceroute [bucket name redacted].s3.amazonaws.com
traceroute to [bucket name redacted].s3.amazonaws.com (54.240.252.97), 30 hops max, 60 byte packets
1 162.243.131.253 (162.243.131.253) 7.704 ms 7.800 ms 7.853 ms
2 ae5-401.cr1.sfo1.us.nlayer.net (69.22.130.37) 0.265 ms 0.256 ms 0.246 ms
3 ae3-70g.cr1.pao1.us.nlayer.net (69.22.143.170) 0.876 ms 0.863 ms 0.841 ms
4 paix01-sfo4.amazon.com (198.32.176.36) 6.105 ms 6.047 ms 1.062 ms
5 205.251.229.6 (205.251.229.6) 34.707 ms 205.251.229.2 (205.251.229.2) 23.583 ms 23.706 ms
6 205.251.232.64 (205.251.232.64) 25.654 ms 205.251.232.66 (205.251.232.66) 23.883 ms 205.251.232.64 (205.251.232.64) 25.550 ms
7 205.251.232.110 (205.251.232.110) 24.185 ms 205.251.232.151 (205.251.232.151) 25.906 ms 205.251.232.163 (205.251.232.163) 23.689 ms
8 205.251.232.169 (205.251.232.169) 27.781 ms 27.520 ms 23.785 ms
9 54.240.230.199 (54.240.230.199) 25.931 ms 54.240.230.197 (54.240.230.197) 25.873 ms 54.240.230.199 (54.240.230.199) 24.218 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *


From Linode/New Jersey --> Amazon S3 us-west-2

# traceroute [bucket name redacted].s3.amazonaws.com
traceroute to [bucket name redacted].s3.amazonaws.com (54.240.248.97), 30 hops max, 60 byte packets
1 router1-nac.linode.com (207.99.1.13) 0.449 ms 0.672 ms 0.805 ms
2 207.99.53.41 (207.99.53.41) 0.519 ms 0.488 ms 0.615 ms
3 vlan805.tbr2.mmu.nac.net (209.123.10.33) 0.272 ms 0.322 ms 0.415 ms
4 0.e1-2.tbr2.ewr.nac.net (209.123.10.113) 0.982 ms 0.962 ms 0.971 ms
5 0.e1-3.tbr1.ewr.nac.net (209.123.10.109) 0.900 ms 0.963 ms 0.947 ms
6 xe-1-3-1-3654.cr1.nyc2.us.nlayer.net (69.31.34.177) 1.870 ms 1.610 ms 1.538 ms
7 vlan-78.ar2.ewr1.us.nlayer.net (69.31.34.127) 5.698 ms 5.637 ms 5.619 ms
8 ewr-edge-10.inet.qwest.net (65.115.224.197) 2.320 ms 2.301 ms 2.282 ms
9 * * *
10 65-122-235-174.dia.static.qwest.net (65.122.235.174) 77.279 ms 65-122-235-170.dia.static.qwest.net (65.122.235.170) 78.373 ms 65-122-235-174.dia.static.qwest.net (65.122.235.174) 77.238 ms
11 205.251.225.4 (205.251.225.4) 86.135 ms 205.251.225.0 (205.251.225.0) 102.123 ms 205.251.225.40 (205.251.225.40) 85.401 ms
12 205.251.232.107 (205.251.232.107) 86.643 ms 205.251.232.105 (205.251.232.105) 93.597 ms 205.251.232.103 (205.251.232.103) 86.440 ms
13 205.251.232.215 (205.251.232.215) 101.147 ms 205.251.232.197 (205.251.232.197) 85.360 ms 85.300 ms
14 205.251.232.221 (205.251.232.221) 87.068 ms 205.251.232.61 (205.251.232.61) 111.026 ms 110.912 ms
15 54.240.230.169 (54.240.230.169) 85.766 ms 54.240.230.171 (54.240.230.171) 86.386 ms 86.385 ms
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Reply
administrator November 12, 2013
Note: Although all three traceroutes show a different IP address, the hostname was the same in all cases. Unfortunately, the full hostname had to be redacted, but it was just a CNAME for s3-us-west-2-w.amazonaws.com.
Reply
bryan November 12, 2013
It's just your stack tuning, sounds like linode just has some higher settings as defaults. This is why nothing changed with the addition of ntt, it's not the providers slowing you down, it's your droplets tcp/ip stack settings and normal "bandwidth delay product".
Reply
Have another answer? Share your knowledge.