Unable to bypass sudo Password prompt on SSH

April 5, 2017 3.5k views
Networking Security DigitalOcean Ubuntu 16.04
SchrodingersCat
By:
SchrodingersCat

I have deployed my public key on to authorized keys.

All was fine till a couple of hours ago, when Logging in via SSH with RSA PK Auth

started prompting for a sudo password.

I checked Auth.log , and it says

Public Key Accepted.

One weird thing I noticed was, There were Millions of entries that logged

Opened a session for root

Immediately after It said

Public key accepted

Probably because it prompted me for a sudo password?

There are also millions of entries logging

Maximum login attempts reached for root @ port 472 from an IP I dont recognise

which were all blocked thanks to the firewall.

Also weird is, Once I do login,

When I run

ps -aux | grep ssh

I get a long list of root logins on the SSH process

I kill them, and they create a new one.

Why?

Log In to Comment
5 Answers
SchrodingersCat April 5, 2017
Accepted Answer

I Found the culprit!

It was a process called

hugepages

and it was doing

dev-hugepages.mount

on startup.

systemctl list-unit-files | grep hu

Gave it out

dev-hugepages.mount | static

sigpwr-container-shutdown.service | static

This, was running a shell script , to disable thp (Transparent huge page) on startup.

Like so:

sudo hugeadm --thp-never

This was one of the tweaks to memalloc for the Redis server on the VPS

** Solution : **

remove the shell script it ran from under \etc\init.d

reboot droplet

@jtittle

thanks for pointing me to this :

That being said, it's hard to tell what's going on. It looks like something is being executed after login (authentication) and it's attempting to run something that would require root or sudo, thus you're being prompted to authenticate before whatever command is running tries to execute.

Reply
jtittle1 April 5, 2017

@SchrodingersCat

If you're logged in as root and using sudo, i.e.

sudo [command]

... then you shouldn't be prompted for a password as you're already running as root, thus there's no higher escalation.

If you're logged in a as a non-root user and prefixing commands with sudo, then you'll be asked to confirm that users password before the command executes -- that's by design. Without prompting for the users password, you're effectively running as root and that defeats the purpose of a sudo user.

...

Public Key authentication has nothing to do with anything other than initial authentication between your computer and the server (or between two servers). If you're being prompted at initial login, that's because you've most likely put a passphrase on your key (which is ideal), but that's as far as that goes.

...

As for the failed logins, seeing failures is normal as the IP of your Droplet is public and most likely belonged to a user before you had it. It's common to see attempts to break in, and what this boils down to is security and properly securing your server.

Ideally, you should:

1). Create a sudo user;
2). Setup SSH keys for that user (with a passphrase on the key);
3). Set a password for the sudo user that differs from your passphrass;
4). Confirm that you're able to login as the sudo user using the key.
5). Confirm that you're able to run sudo [command] as the user, and it works;
6). Lock the root account so that it can't be used to login.

Reply
  • SchrodingersCat April 5, 2017

    Hey! thanks for looking in!

    BTW,

    I did go through the entire security setup in detail.

    1. Setup A User
    2. Put him in sudoers
    3. disabled root login via ssh
    4. configured firewalls
    5. disabled ssh via password auth
    6. enabled ssh via RSA PK Auth only

    I believe, All was fine till a couple of hours ago,

    I was able to authenticate myself via ssh+RSA.

    The actual issue Since earlier today is,

    when Logging in via SSH with RSA PK Auth.

    As soon as I hit return on

    ssh user@do-droplet

    It prompts me for the sudo password. Meaning, I can't get in without the password.

    I did put a passphrase in for the key, for extended security, but it doesn't seem to care about that passphrase at all.

    It for some reason, wants me to key in , the Sudo password for the user trying to get in.

    Which seems a bit off.

    And once I key that in, I have the sudo prompt by default, instead of the normal user prompt.

    Meaning, I do not need to prefix sudo anymore. That troubles me.
    it
    It is like If I login with

    ssh user@do-droplet

    I need to put in the sudo password, and then I can do stuff like

    bash $ init 6

    which normally requires a sudo prefixed

jtittle1 April 5, 2017

@SchrodingersCat

When it comes to SSH and PubKey Authentication, it's best to keep it as simple as possible unless you have a reason to drift from the norm. I can only speak from MacOS and Windows 10 at the moment as I don't have a Linux box setup locally right now -- only the remote Droplets.

I use Terminal on MacOS and PuTTy on Windows 10.

The known_hosts file on MacOS does store basic information and that, of course, prevents having to confirm the host that I'm connecting to. Beyond that, the key itself is not cached on either system, so to login, the passphrase for the key must still be used unless there's no passphrase setup for the key.

I personally wouldn't use anything that caches the passphrase of my keys, even on my own systems as that is substantially less secure and allows anyone that can gain access the ability to login.

That being said, it's hard to tell what's going on. It looks like something is being executed after login (authentication) and it's attempting to run something that would require root or sudo, thus you're being prompted to authenticate before whatever command is running tries to execute.

...

Using the details I provided above, your scenario is definitely not normal and shouldn't happen unless you do have something that is set to run or execute after authentication, in which case, you'd need to find out what that is. It's hard to tell when you're just being prompted as it doesn't provide any actual details on what's going on.

In any normal circumstance, you'd run:

ssh user@host

or

ssh user@host -i /path/to/privkey

Confirm (if it's the first connection), or enter your passphrase (if one is set), and you're in.

I can't think of anything off the top of my heard that would log you in and then immediately attempt to get you to authenticate your sudo user unless you've somehow been hacked and something is running in an attempt to gain your sudo password, thus allowing an attacker to get what they're after.

That's just a guess, not a claim, and is only based on the information you're providing.

To test, I would setup a simple 512MB Droplet and run though the configuration I provided. With it, you should confirm once, authenticate with or without a passphrase, and either login and be denied (if the details are not correct).

Reply
  • SchrodingersCat April 5, 2017

    Yeah! Now the thought of something hacking into the droplet is a scary one.

    maybe I 'd just revoke all users' auth and reconfigure the droplet, but it is kinda challenging to

    put some time into understanding why this happens.

    I did try a verbose connection with

    ssh user@server -vvv

    doesn't log anything of note , but definitely says

    debug1: Authentication succeeded (publickey).

    in the logs.

    Now sifting through the rest of the 500 lines of logs and SSL handshakes to see If something is fishy

    • jtittle1 April 5, 2017

      @SchrodingersCat

      auth.log isn't going to log anything that may run after the authentication process is over.

      As noted above, I'd retest the setup I recommended above and see if the same thing results, just as a test. This could be done quickly on a 512MB Droplet in just a few minutes.

      If you get the same results on a clean Droplet as you do with the current, I'd need to know more about how you set things up, in detail, as what I posted shouldn't result in the same issues. I've never had an issue with commands being auto-run on login unless that was the intention.

      You can also check to see if you have any .bash_* files. If there's anything in those files that is set to run, they will run on login.

      i.e.

      .bashrc
.bash_profile
.bash_aliases

      Or any other files that would be triggered on login.

      • jtittle1 April 5, 2017

        @SchrodingersCat

        As an example, if I created ~/.bash_profile, and within it I placed:

        echo "Hello"
echo "This is a test"

        Then exit and try to login again via SSH, the first thing I see after:

        Last login: Wed Apr  5 22:03:03 2017 from ...

        ... is:

        Hello
This is a test

        Now if a command is set to run that requires sudo within those files, boom, there's the issue. If not, then you'd need to dive a little deeper and see what's going on and from where.

        • SchrodingersCat April 5, 2017

          I just actually finished setting up a new droplet with ubuntu.

          I followed the digital ocean documentation at

          Setting up a fresh droplet

          to the letter.

          And configured the setup for users as you said earlier.

          I then also configured a simple firewall with

          This DO post

          double checked with the list of commands you wrote above for

          importing a public RSA key into

          /home/user/.ssh/authorized_keys

          And,

          I seem to have no trouble connecting to the new droplet.

          I am now checking the .bash* files as you said, (** On the Older Droplet with issues **) and there appears to be nothing that executes a shell command.

          I guess, earlier when you mentioned, that it might be something that runs immediately after SSH Authentication lets you in,

          I figured I would individually check all init.d processes .

          The problem definitely was with a package I installed earlier today.

          hugepages

          This disables huge Swappiness or huge page files on Ubuntu.

          This package, created a shell script, on install , and made it run on userauth.

          I deleted the script it runs.

          I can always run it before a redis session maybe?

          cheers

          Initial Server Setup with Ubuntu 14.04
          by Justin Ellingwood
          When you start a new server, there are a few steps that you should take every time to add some basic security and set a solid foundation. In this guide, we'll walk you through the basic steps necessary to hit the ground running with Ubuntu 14.04.
jtittle1 April 5, 2017

@SchrodingersCat

When you run:

ssh user@hostname

... it's not asking for a sudo password, it's asking for the password associated with your private key. When you created your public/private key pair, you must have set a passphrase on it (which is good), so you'd need to enter in that passphrase to login.

The above applies if you're using PuTTy on Windows or similar. On a Mac, you'd specify the path to your key using -i, i.e.

ssh user@hostname -i /path/to/privatekey

If it's not asking for a key file, then it's asking for the password of the user. If the user doesn't have a password, then you won't be able to login as that user and would need to login as root to change the users password using:

passwd username

Where username is the user you're trying to login with.

Reply
jtittle1 April 5, 2017

@SchrodingersCat

What tutorial/guide did you follow to setup the sudo user you're referencing?

Ideally, if we just use password-based logins, we'd do the following as root:

useradd -d /home/username username

usermod -aG sudo username

passwd username

Once the password is set for the user, you now have a sudo user and can login using SSH. If we need to setup SSH Keys for the user, it's a bit more involved.

...

mkdir -p /home/username/.ssh

touch /home/username/.ssh/authorized_keys

chown -R username:username /home/username/*

chmod 700 /home/username/.ssh

chmod 644 /home/username/.ssh/authorized_keys

usermod -aG sudo username

passwd username

You'd change username in the above to match that of the username you want to use as a sudo user. You would then add the public key to:

/home/username/.ssh/authorized_keys

Using:

nano /home/username/.ssh/authorized_keys

Paste and save.

As long as your SSH Configuration is setup to allow keys, then that should allow you to login using a public/private key pair as opposed to logging in with a password, but that won't prevent you from being prompted for the passphrase on your SSH key (two entirely different things).

With the above, you'd have to run:

sudo [command]

On each command you run as that user. This should be enforced. If you're able to run without sudo prefixing a command as a normal user added to the sudo group, then I'd say there's something wrong with the setup.

It's normal to not have to enter your password each time you run sudo [command] as authentication expires after a few minutes, so you can run a few commands without being prompted again.

Reply
  • SchrodingersCat April 5, 2017

    @jtittle

    Hey! I pretty much did it all myself, having done the setup and the firewall a dozen times n the past on various servers.

    Let me give you a picture of what is happening.

    1. I Know, that is it normal for the system, NOT to ask for a sudo password at times , as long as there is a file in the User's home directory, called,

    .sudo_as_admin_successful

    That However, is NOT the issue here.

    I am least worried about whether I am asked to SUDO once I am authenticated via SSH.

    Now, There is something called **gnome-key-ring **

    Once you log in to a remote host, from your local host, the first time , it caches your SSH Key's Passphrase for that particular remote host.

    That would mean,

    You do not need to enter your passphrase ( for the key you copied into .ssh/authorized_keys on the remote host)

    Now, the very first time you log in via ssh, you get this alert

    The authenticity of host 'abc.com ()' can't be established.
    RSA key fingerprint is SHA256:***************************
    Are you sure you want to continue connecting (yes/no)?

    you type in yes, it saves the host in .ssh/known_hosts.

    Then, it asks you for the passphrase for your public key cryptography?

    Warning: Permanently added 'abc.com , 1xx.abc.xyz.pqr' (RSA) to the list of known hosts.
    Enter passphrase for key '/home/sherlock/.ssh/id_rsa':

    you then key in your Passphrase for your key.

    My Issue is, on an attempt to log in ,

    when I run

    ssh sherlock@abc.com [return]

    1. The key ring cache seems to be ignored.
    2. An IRRELEVANT Password is being asked.

    like so:

    [sudo] password for sherlock:

    when it should actually be asking for a passphrase in case it ignores the key ring cache.

    cheers!

Have another answer? Share your knowledge.

Related Questions