Question

Long timeouts with sudo and systemctl

Posted December 25, 2019 264 views
Ubuntu

Here is my system:

bill@Bill2:~$ lsb_release -a
LSB Version: core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description: Ubuntu 18.10
Release: 18.10
Codename: cosmic

My Droplet has been running very smoothly for many months. I do a lot of software development on it as well as hosting a minimal web site.
Suddenly, this morning (Christmas morning, can you believe it), things started going haywire. Specifically, the sudo command now has a very long timeout, and systemctl does the same.

Here is a timed example of a simple systemctl command:

bill@Bill2:~$ time systemctl status caddy.service
Failed to get properties: Connection timed out

real 1m30.065s
user 0m0.003s
sys 0m0.007s

Notice the 1 ½ minute time out. Also, note the weak reference to a connection timing out (there is no indication of what connection that is).

Here is an example of a simple application of sudo:

bill@Bill2:~$ time sudo ls /root
[sudo] password for bill:
Desktop Documents Downloads Music Pictures Public Templates Videos snap

real 1m41.903s
user 0m0.010s
sys 0m0.009s

Again notice the long timeout. Below is a simple attempt to use strace for additional information. I have just included the tail end of the strace output because it refers to a timeout while attempting to access a server.

bill@Bill2:~$ strace systemctl status

writev(2, [{iovbase=“Failed to read server status: Co”…, iovlen=50}, {iovbase=“\n”, iovlen=1}], 2) = 51
close(3Failed to read server status: Connection timed out
) = 0
dup2(6, 1) = 1
close(6) = 0
dup2(7, 2) = 2
close(7) = 0
kill(7853, SIGCONT) = 0
waitid(PPID, 7853, {sisigno=SIGCHLD, sicode=CLDEXITED, sipid=7853, siuid=1000, sistatus=0, siutime=0, sistime=0}, WEXITED, NULL) = 0
— SIGCHLD {si
signo=SIGCHLD, sicode=CLDEXITED, sipid=7853, siuid=1000, sistatus=0, siutime=0, sistime=0} —
exit
group(0) = ?
+++ exited with 0 +++

Here is a strace example of a simple ls command:

bill@Bill2:~$ strace ls /root

sendmsg(3, {msgname=NULL, msgnamelen=0, msgiov=[{iovbase=“\0AUTH EXTERNAL ”, iovlen=15}, {iovbase=“31303030”, iovlen=8}, {iovbase=“\r\nNEGOTIATEUNIXFD\r\nBEGIN\r\n”, iovlen=28}], msgiovlen=3, msgcontrollen=0, msgflags=0}, MSGDONTWAIT|MSGNOSIGNAL) = 51
gettid() = 8146
getrandom(“\x93\xcd\x9d\x13\x0d\x32\xdd\x75\x57\x69\x54\x30\x3d\x9e\xfd\xb0”, 16, GRNDNONBLOCK) = 16
gettid() = 8146
ioctl(1, TCGETS, {B38400 opost isig icanon echo …}) = 0
ioctl(2, TCGETS, {B38400 opost isig icanon echo …}) = 0
ioctl(1, TIOCGWINSZ, {ws
row=35, wscol=100, wsxpixel=0, wsypixel=0}) = 0
ioctl(1, TIOCGWINSZ, {ws
row=35, wscol=100, wsxpixel=0, wsypixel=0}) = 0
pipe2([4, 5], O
CLOEXEC) = 0
rtsigprocmask(SIGSETMASK, ~[RTMIN RT1], [], 8) = 0
clone(child
stack=NULL, flags=CLONECHILDCLEARTID|CLONECHILDSETTID|SIGCHLD, childtidptr=0x7fbd25671c10) = 8147
rt
sigprocmask(SIGSETMASK, [], NULL, 8) = 0
fcntl(1, F
DUPFDCLOEXEC, 3) = 6
dup2(5, 1) = 1
fcntl(2, F
DUPFDCLOEXEC, 3) = 7
dup2(5, 2) = 2
close(4) = 0
close(5) = 0
prlimit64(0, RLIMIT
NOFILE, {rlimcur=16*1024, rlimmax=16*1024}, NULL) = 0
uname({sysname=“Linux”, nodename=“Bill2”, …}) = 0
recvmsg(3, {msgnamelen=0}, MSGDONTWAIT|MSGCMSGCLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
ppoll([{fd=3, events=POLLIN}], 1, {tvsec=89, tvnsec=997330000}, NULL, 8

While watching the output I noticed the terminal cursor sitting on the ppoll system call as if the sudo software, or some library, is waiting for an Internet response. I also notice a reference to OAUTH at the beginning of the dump.

I will continue to investigate, but I think I’m going to need your help.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello,

I think that you are headed in the right direction. What I could suggest is adding some more arguments to the strace command to get a bit more information, for example:

strace -vvv -s 10000 -f ls /root

Also, are you using LDAP? If so check your /var/log/auth.log log as well to see if you have some more information there. This looks like that your Droplet is unable to reach the LDAP server.

Another thing that comes to my mind is that if you’ve recently changed your hostname, if so make sure to update it in your /etc/hosts file as well:

127.0.0.1 localhost YOUR_HOSTNAME
::1 localhost YOUR_HOSTNAME

Let me know how it goes!
Regards,
Bobby

  • After several attempts at trying to account for this long timeout problem, I decided to install Wireshark and watch the packets. But using Wireshark brought out some error messages, including that libdbus was missing. I reinstalled it, but I still have the timeout.

    Then I installed gnome-shell-extensions to support the installation of the gnome-software package manager. But that installation was full of timeouts, and various components didn’t install, including ibus due to exessive timeouts.

    After all this, I decided to reboot to get the updates running, especially dbus. But now my Droplet won’t boot. I watched it attempting to boot on the DO console. It gets as far as installing ibus and then hangs.

    So this has gone too far. I’m not about to spend countless hours trying to patch up a sick Droplet. I would prefer to have my most recent backup recovered, and I will reinstall my development files from Github.

    I’m not sure if I ask for that support here, or if there is some other place to request a backup restore. I’m confident that someone will let me know.

    Thanks for listening,
    Bill Trenker

Submit an Answer