DigitalOcean Kubernetes: new control plane is faster and free, enable HA for 99.95% uptime SLA

Question

searx metasearch instance - refused to connect

Hi Community,

the good old “refused to connect” is driving me nuts in my particular setup and I hope somebody does see what I simply can’t anymore after fiddling around with almost everything I could possibly think off…

Here we go, with a system description as well as with all the glory details of the issue I’m facing…

System Details:

[user@host ~]$ head -1 /etc/*release*
# PRETTY_NAME="Debian GNU/Linux 10 (buster)"

## nginx version: nginx/1.14.2
built with OpenSSL 1.1.1c  28 May 2019 (running with OpenSSL 1.1.1d  10 Sep 2019)

I’m trying to figure out why on earth my “searx metasearch” instance suddenly stopped working, although I’ve even reverted back to a working snapshot 5 days back, when the same configuration was working as set up initially. Haven’t changed anything within my configuration at all… I even re-installed the entire application without any success.

Since I’m trying to access that searx instance running locally on my nginx on TCP_Port 8888 through uwsgi as described within this install wiki:

Example link

proxy or uWSGI is the relevant entry for my setup.

The uWSGI setup reflects this one:

Example link

1:1, no changes at all…

[user@host ~]$ sudo -H service uwsgi status searx
● uwsgi.service - LSB: Start/stop uWSGI server instance(s)
   Loaded: loaded (/etc/init.d/uwsgi; generated)
   Active: active (running) since Tue 2020-07-14 03:17:37 CEST; 2min 52s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1626 ExecStart=/etc/init.d/uwsgi start (code=exited, status=0/SUCCESS)
    Tasks: 4 (limit: 1167)
   Memory: 124.0M
   CGroup: /system.slice/uwsgi.service
           ├─1728 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
           ├─1735 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
           ├─1736 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
           └─1737 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log

Jul 14 03:17:36 host systemd[1]: Starting LSB: Start/stop uWSGI server instance(s)...
Jul 14 03:17:37 host uwsgi[1626]: Starting app server(s): uwsgi -> . done.
Jul 14 03:17:37 host systemd[1]: Started LSB: Start/stop uWSGI server instance(s).

The service is running as verified above and also accessible on the internal host listening on Port 8888

[user@host ~]$ curl --verbose --insecure --location http://127.0.0.1:8888
* Expire in 0 ms for 6 (transfer 0x5645278c2f50)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5645278c2f50)
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 10535
< Server-Timing: total;dur=11.569
<
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>

--- cut for brevity ---

    </noscript>
</body>
* Connection #0 to host 127.0.0.1 left intact
</html>%

The same test again, this time around with my subdomain;

[user@host ~]$ curl --verbose --insecure --location https://subdomain.domain
* Expire in 0 ms for 6 (transfer 0x560e66e73f50)
* Expire in 1 ms for 1 (transfer 0x560e66e73f50)

* Connected to subdomain.domain (xxxx:xxxx:x:xx::xxxx:xxxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=subdomain.domain
*  start date: Jul  1 17:16:32 2020 GMT
*  expire date: Sep 29 17:16:32 2020 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x560e66e73f50)
> GET / HTTP/2
> Host: subdomain.domain
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Tue, 14 Jul 2020 01:25:33 GMT


--- cut for brevity ---

    </noscript>
</body>
* Connection #0 to host subdomain.domain left intact
</html>%

Firewall allowing connections to 8888 and 443

[user@host ~]$ sudo ufw status numbered | grep 8888
[10] 8888/tcp                   ALLOW IN    Anywhere
[32] 8888/tcp (v6)              ALLOW IN    Anywhere (v6)


[user@host ~]$ sudo ufw status numbered | grep 443
[19] 443/tcp                    ALLOW IN    Anywhere
[23] 443/tcp (v6)               ALLOW IN    Anywhere (v6)

TCP sockets also hanging in there:

[user@host ~]$ sudo netstat -anp | grep 8888
tcp        0      0 127.0.0.1:8888          0.0.0.0:*               LISTEN      1728/uwsgi


[user@host ~]$ sudo netstat -anp | grep :443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      615/nginx: master p
tcp       32      0 138.197.209.92:44094    99.84.174.86:443        CLOSE_WAIT  1735/uwsgi
tcp       32      0 138.197.209.92:47040    99.84.243.60:443        CLOSE_WAIT  1735/uwsgi
tcp       32      0 138.197.209.92:55226    140.177.18.77:443       CLOSE_WAIT  1736/uwsgi
tcp       32      0 138.197.209.92:44096    99.84.174.86:443        CLOSE_WAIT  1736/uwsgi
tcp       32      0 138.197.209.92:47042    99.84.243.60:443        CLOSE_WAIT  1736/uwsgi
tcp       32      0 138.197.209.92:42792    140.177.16.37:443       CLOSE_WAIT  1735/uwsgi
tcp6       0      0 :::443                  :::*                    LISTEN      615/nginx: master p

and lastly, the relevant nginx config for the searx engine;

[user@host ~]$ sudo cat /etc/nginx/sites-available/searx
server {
    server_name searx.domain www.searx.domain;
    root /usr/local/searx;

    access_log /dev/null;
    error_log /dev/null;

    location / {
    proxy_pass http://127.0.0.1:8888;

    proxy_set_header Host $host;
    proxy_set_header Connection       $http_connection;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header X-Scheme         $scheme;
    proxy_buffering                   off;
    }

    listen 443 ssl http2; # managed by Certbot
    listen [::]:443 ssl http2;
    ssl_certificate 
    /etc/letsencrypt/live/searx.rsec.systems/fullchain.pem; # managed by Certbot
    ssl_certificate_key 
    /etc/letsencrypt/live/searx.rsec.systems/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = www.searx.domain) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = searx.domain) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name searx.domain www.searx.domain;
    return 404; # managed by Certbot

}

DNS resolution works just fine;

[user@host ~]$ dig searx.domain A,CNAME

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> searx.domain A,CNAME
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24508
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;searx.domain.        IN  A

;; ANSWER SECTION:
searx.domain. 278 IN  CNAME   domain.
domain.       278 IN  A   xxx.xxx.xxx.xxx

;; Query time: 21 msec
;; SERVER: 67.207.67.3#53(67.207.67.3)
;; WHEN: Tue Jul 14 03:40:59 CEST 2020
;; MSG SIZE  rcvd: 77

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54492
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;A,CNAME.           IN  A

;; AUTHORITY SECTION:
.           3600    IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2020071301 1800 900 604800 86400

;; Query time: 4 msec
;; SERVER: 67.207.67.3#53(67.207.67.3)
;; WHEN: Tue Jul 14 03:40:59 CEST 2020
;; MSG SIZE  rcvd: 111

Any help pointing me into the right direction would be much appreciated.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

Try DigitalOcean for free

Click below to sign up and get $100 of credit to try our products over 60 days!

Sign up