Gitlab site says to configure Gitlab-CE installation, what can I do get back into my site?

February 13, 2018 1.7k views
Git Ubuntu 16.04

Hello,

I have a droplet set up for a while with Gitlab-CE from a one-click install. I believe Gitlab was at 9.2.5 when I created the droplet.

I have tried to keep the droplet up to date, but today I noticed that the site was not working. There was an error about unsecure connection. I have had this issue before, and it was easily fixed by updating my droplet. I went and did that but still could not get into my site.

I checked the Let's Encrypt certificate to see if it needed to be renewed with sudo certbot renew --dry-run but that showed these errors:

Attempting to renew cert (gitlab.devplateau.com) from /etc/letsencrypt/renewal/gitlab.devplateau.com.conf produced an unexpected error: Failed authorization procedure. gitlab.devplateau.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://gitlab.devplateau.com/.well-known/acme-challenge/HSNFfdwytBVlEdmalsrX1gGxfVn3WtNI0YK8Pm6JtPo: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.devplateau.com/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.devplateau.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gitlab.devplateau.com
   Type:   unauthorized
   Detail: Invalid response from
   http://gitlab.devplateau.com/.well-known/acme-challenge/HSNFfdwytBVlEdmalsrX1gGxfVn3WtNI0YK8Pm6JtPo:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I have made sure that the A record for gitlab.devplateau.com did not get removed somehow and it is still there. I even removed it and created it again just to be safe.

Can someone please help me get back into my Gitlab site? I have important code saved and would prefer not to have to start the server over.

3 Answers

Take a look at https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block - add this to the server block

location ^~ /.well-known/acme-challenge/ {
  default_type "type/plain";
  root /tmp/letsencrypt;
}

issue this command
mkdir /tmp/letsencrypt

Restart Gitlab and then try and renew the SSL

.. or you can reconfigure gitlab to not use SSL

  • Hi @jasonjpeters

    Thank you for replying!

    I tried that, but it did not work. I still get the unauthorized error for Let's Encrypt when trying to renew it.

    Here is my gitlab.rb file:

     external_url "https://gitlab.devplateau.com/"
    
     gitlab_rails['gitlab_email_from'] = "gitlab@gitlab.devplateau.com"
     gitlab_rails['gitlab_support_email'] = "support@gitlab.devplateau.com"
    
     #gitlab_rails['smtp_enable'] = true
     #gitlab_rails['smtp_address'] = "mail.devplateau.com"
     #gitlab_rails['smtp_port'] = 456
     #gitlab_rails['smtp_user_name'] = ""
     #gitlab_rails['smtp_password'] = ""
     #gitlab_rails['smtp_domain'] = "devplateau.com"
     #gitlab_rails['smtp_authentication'] = "login"
     #gitlab_rails['smtp_enable_starttls_auto'] = true
     #gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
    
     nginx['redirect_http_to_https'] = true
     nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab.devplateau.com/fullchain.pem"
     nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab.devplateau.com/privkey.pem"
     #nginx['custom_gitlab_server_config'] = "location ^~ /.well-known { root /var/www/letsencrypt; }"
     nginx['custom_gitlab_server_config'] = "location ^~ /.well-known/acme-challenge/ { default_type 'type/plain'; root /tmp/letsencrypt; }"
    #external_url "http://162.243.62.9"
    
    

    Does everything look right?

    I would rather not reconfigure gitlab to not use ssl.

    • I suspect that certbot is not able to connect properly to verify and that is why you are getting that error.

      Cerrtbot itself connect on port 80 however your Gitlab settings are redirecting all traffic to 443 with this line:

       nginx['redirect_http_to_https'] = true
      

      You can remove this line:

      nginx['custom_gitlab_server_config'] = "location ^~ /.well-known/acme-challenge/ { default_type 'type/plain'; root /tmp/letsencrypt; }"
      

      and update this line:

       #nginx['custom_gitlab_server_config'] = "location ^~ /.well-known { root /var/www/letsencrypt; }"
      

      to:

      nginx['custom_gitlab_server_config'] = "location ^~ /.well-known { root /var/www/letsencrypt; allow all; }"
      

      That's just a clean up as the lines were pretty much the same. However that does not solve the the SSL renewal issue.

      Here is an article with how to setup LetsEncrypt with GitLab.
      https://www.digitalocean.com/community/tutorials/how-to-secure-gitlab-with-let-s-encrypt-on-ubuntu-16-04

      Double check your settings with that.

      If you have Port 80 disabled/blocked in your firewall, you would need to have that open.

      by Justin Ellingwood
      GitLab, specifically GitLab CE (Community Edition), is an open source application primarily used to host Git repositories, with additional development-related features like issue tracking. The GitLab project makes it relatively straight forward to set up a GitLab instance on...
      • Hello again,

        Sorry for the late reply. I have changed the settings in gitlab.rb as suggested and went through the linked article to make sure all the settings are correct.

        Even after all that, I still get an error when trying to renew the certificate for my site. I have port 80 open on UFW.

        I would hate to have to start over, but is that my best option at this point or is there something else I can try?

That is quite alright.

If you would like to give it a shot. (backup your Gitlab configuration file)

This is how I run my Gitlab installation - non-bundled NGINX webserver

http://www.creativespacesbysherihannah.com/wp-sys/wp-content/uploads/staircase-before.jpg?TB_iframe=true&width=700&height=351

You will need to install NGINX

wget https://nginx.org/keys/nginx_signing.key -O - | sudo apt-key add -

echo "deb http://nginx.org/packages/mainline/ubuntu/ xenial nginx
deb-src http://nginx.org/packages/mainline/ubuntu/ xenial nginx" > /etc/apt/sources.list.d/nginx.list

apt-get -y update

apt-get -y install nginx

this will install NGINX Mainline

My NGINX configuration looks like this - replace YOURSERVERFQN with your server domain

upstream gitlab-workhorse {
  server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;
}


server {

  listen 0.0.0.0:80;

  server_name YOUR_SERVER_FQN;
  server_tokens off;

  location / {
    return 301 https://$http_host$request_uri;    
  }

  # Letsencrypt Verification
  #
  location ~ /.well-known {
    root /usr/share/nginx/html;
    allow all;
  } 

  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;

}


server {

  listen 0.0.0.0:443 ssl;

  server_name YOUR_SERVER_FQDN;
  server_tokens off;

  root /opt/gitlab/embedded/service/gitlab-rails/public; 

  ssl on;
  ssl_certificate /etc/letsencrypt/live/git.YOUR_SERVER_FQN/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/git.YOUR_SERVER_FQN/privkey.pem;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

  ssl_prefer_server_ciphers on;
  ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
  ssl_ecdh_curve secp384r1;

  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;

  ssl_stapling on;
  ssl_stapling_verify on;

  resolver 8.8.8.8 8.8.4.4 valid=300s;
  resolver_timeout 5s;

  add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;

  ssl_dhparam /etc/ssl/certs/dhparam.pem;  

  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;

  location / {
    client_max_body_size 0;
    gzip off;

    proxy_read_timeout      300;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_http_version 1.1;

    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-Ssl     on;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_pass http://gitlab-workhorse;
  }

}

I have the same problem, But still can't fix it.
Are there any idea.
Please help us. :(

  • Hey,

    Yeah I never figured out what was going on with this. I just decided to start over with a new server.

    Sorry I couldn't be of more help :(

Have another answer? Share your knowledge.