Hi all;
I’m not even sure exactly how to describe my issue… I have an Ubuntu 18.04 server running Nginx. About once a month my local computer (also Ubuntu 18.04) seems to get “stuck” with a connection to my DO server. I didn’t notice it this week, and it has used 8GB of combined upload / download to my server. To what? I don’t know.
I use tcptrack, but I only see my server IP and port 443. I don’t know what it’s connected to on my server. At first I though it was my SSH sessions as sometimes they freeze and I close the window, but when I saw port 443 I knew it was a website. I’ve rebooted my local computer multiple times with no effect. Run tcptrack and there is the connection, using 28kb/s up and down.
Today, I did a reload of the Nginx server and my connection immediately showed “RESET” as the state and then disappeared completely. It’s been OK since. I need some help with what to use to actually track down the issue. It’s my server and it happened to me. I could reset Nginx. What if it’s happening to others that connect to my server, or is it during some of my administration? I left my local computer off all day and when I turned it on, the connection persisted. If it’s happening to others and they have no way of resetting the connection. The reason I even know is I’m in the country on very limited bandwidth and got a text notification, otherwise I might not have even noticed. It ends up costing me a small fortune.
Any help in pointing my to how to go about solving this would be appreciated. I’m going to dig through sever logs and see if I can find clues.
Thank you; Kyle
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
So, I spent the most of the weekend tracking down this issue and, ultimately, redoing my entire network. It ended up being a DNS issue.
Not sure why or how this all happened as I’ve been running the same DNS for the past few years, but a recent update must have triggered something (I recently upgraded my desktop to Ubuntu 18.04 and created a new droplet, also 18.04).
The issue, was my domain at home was kylestubbins.com (single sign on SSSD - LDAP / Kerberos), which was authoritive, so no requests for kylestubbins.com ever left my LAN. When I put www.kylestubbins.com on DO, I simply setup DNS records for it on my LAN. Everything was golden. What I’ve noticed now is that when I mistype any domain, it redirects to my DO server, which has records for my kylestubbins.com domain. Why? I haven’t yet figured that out. The same settings, with my new DNS domain, behaves exactly as it should and says “hey, can’t find that!” when I mistype something.
The cause: one of my printers kept posting to ipp/port1, which was being expanded to kylestubbins.com/ipp/port1 (I’m assuming by the expand-hosts setting of dnsmasq) and simply using bandwidth between my LAN and my DO server. 28kB/s on average.
The solution: was to redo the DNS for my LAN by adding the subdomain home.kylestubbins.com. (Yeah, I know there are those of you shaking your head right now… live and learn.)
On the plus side, I also solved an NFSv4 issue I was having, where I was getting long wait times opening NFS shares (this was happening with 16.04, but not as prevalent – it was painful in 18.04 for a while).
It was DNS. I should have known. It’s always DNS.
My LAN has never worked so well. Silver lining, I guess.
Kyle
Hi @kylestubbins,
I think that going through the server logs would be the best first step.
What I could suggest is using this script here to summarize your access logs.
The output that it would provide you with should look something like this:
This should give you some more information on what files and what IPs are hitting your server the most.
Regards, Boby
Crazy, but digging through my logs produces multiple POST requests to a printer from my IP.
[07/Dec/2019:02:21:21 +0000] “POST /ipp/port1 HTTP/1.1” 301 203 “-” “CUPS/2.2.7 (Linux 4.15.0-70-generic; x86_64) IPP/2.0”
Well, at least I’m getting somewhere.
I have spent hours upon hours trying to get somewhere here. I know this may not be the right place, but I’m begging for someone to help. I use tcptrack and see a connection, but that’s the extent of the information I have. I have no idea how to get any further information… even after hours of reading and installing application, trying and trying. Tcptrack, literally, shows my local IP with port 37604 (it’s always high and random) connected to my DO IP address at port 443. Yes, 443 is SSL, so I can assume it is Nginx.
What about locally? I search for how to see what applications are using a tcp connection and I get hundreds of “how to see what is listening on a port”… I don’t want listening, but it doesn’t seem to matter what keywords I use, I get “how to see what is listening on a port”. I try to use top and don’t see any instances of Firefox, or anything else that seems like it would be connected to my DO server via SSL.
I just turned my computer on to do a little more investigating, opened tcptrack and there it was, using 28kb/s up and down. This time I unplugged my network cable and let it sit until all activity stopped, plugged it back in and the connection is still there, but no longer using bandwidth.
If I can figure out what is actually using that tcp connection, maybe I can figure this out. Why is it so difficult to get the right command to pull this info? If I see 192.168.70.144:37604, how can I find out what is actually using this locally?
If you don’t know how to help, do you know where I can go to get help? I’m getting desperate.
Thank you; Kyle