By zodiac72
PHD in Operating Systems
Hi Community,
the good old “refused to connect” is driving me nuts in my particular setup and I hope somebody does see what I simply can’t anymore after fiddling around with almost everything I could possibly think off…
Here we go, with a system description as well as with all the glory details of the issue I’m facing…
System Details:
[user@host ~]$ head -1 /etc/*release*
# PRETTY_NAME="Debian GNU/Linux 10 (buster)"
## nginx version: nginx/1.14.2
built with OpenSSL 1.1.1c 28 May 2019 (running with OpenSSL 1.1.1d 10 Sep 2019)
I’m trying to figure out why on earth my “searx metasearch” instance suddenly stopped working, although I’ve even reverted back to a working snapshot 5 days back, when the same configuration was working as set up initially. Haven’t changed anything within my configuration at all… I even re-installed the entire application without any success.
Since I’m trying to access that searx instance running locally on my nginx on TCP_Port 8888 through uwsgi as described within this install wiki:
proxy or uWSGI is the relevant entry for my setup.
The uWSGI setup reflects this one:
1:1, no changes at all…
[user@host ~]$ sudo -H service uwsgi status searx
● uwsgi.service - LSB: Start/stop uWSGI server instance(s)
Loaded: loaded (/etc/init.d/uwsgi; generated)
Active: active (running) since Tue 2020-07-14 03:17:37 CEST; 2min 52s ago
Docs: man:systemd-sysv-generator(8)
Process: 1626 ExecStart=/etc/init.d/uwsgi start (code=exited, status=0/SUCCESS)
Tasks: 4 (limit: 1167)
Memory: 124.0M
CGroup: /system.slice/uwsgi.service
├─1728 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
├─1735 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
├─1736 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
└─1737 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log
Jul 14 03:17:36 host systemd[1]: Starting LSB: Start/stop uWSGI server instance(s)...
Jul 14 03:17:37 host uwsgi[1626]: Starting app server(s): uwsgi -> . done.
Jul 14 03:17:37 host systemd[1]: Started LSB: Start/stop uWSGI server instance(s).
The service is running as verified above and also accessible on the internal host listening on Port 8888
[user@host ~]$ curl --verbose --insecure --location http://127.0.0.1:8888
* Expire in 0 ms for 6 (transfer 0x5645278c2f50)
* Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5645278c2f50)
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 10535
< Server-Timing: total;dur=11.569
<
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
--- cut for brevity ---
</noscript>
</body>
* Connection #0 to host 127.0.0.1 left intact
</html>%
The same test again, this time around with my subdomain;
[user@host ~]$ curl --verbose --insecure --location https://subdomain.domain
* Expire in 0 ms for 6 (transfer 0x560e66e73f50)
* Expire in 1 ms for 1 (transfer 0x560e66e73f50)
* Connected to subdomain.domain (xxxx:xxxx:x:xx::xxxx:xxxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=subdomain.domain
* start date: Jul 1 17:16:32 2020 GMT
* expire date: Sep 29 17:16:32 2020 GMT
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x560e66e73f50)
> GET / HTTP/2
> Host: subdomain.domain
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Tue, 14 Jul 2020 01:25:33 GMT
--- cut for brevity ---
</noscript>
</body>
* Connection #0 to host subdomain.domain left intact
</html>%
Firewall allowing connections to 8888 and 443
[user@host ~]$ sudo ufw status numbered | grep 8888
[10] 8888/tcp ALLOW IN Anywhere
[32] 8888/tcp (v6) ALLOW IN Anywhere (v6)
[user@host ~]$ sudo ufw status numbered | grep 443
[19] 443/tcp ALLOW IN Anywhere
[23] 443/tcp (v6) ALLOW IN Anywhere (v6)
TCP sockets also hanging in there:
[user@host ~]$ sudo netstat -anp | grep 8888
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 1728/uwsgi
[user@host ~]$ sudo netstat -anp | grep :443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 615/nginx: master p
tcp 32 0 138.197.209.92:44094 99.84.174.86:443 CLOSE_WAIT 1735/uwsgi
tcp 32 0 138.197.209.92:47040 99.84.243.60:443 CLOSE_WAIT 1735/uwsgi
tcp 32 0 138.197.209.92:55226 140.177.18.77:443 CLOSE_WAIT 1736/uwsgi
tcp 32 0 138.197.209.92:44096 99.84.174.86:443 CLOSE_WAIT 1736/uwsgi
tcp 32 0 138.197.209.92:47042 99.84.243.60:443 CLOSE_WAIT 1736/uwsgi
tcp 32 0 138.197.209.92:42792 140.177.16.37:443 CLOSE_WAIT 1735/uwsgi
tcp6 0 0 :::443 :::* LISTEN 615/nginx: master p
and lastly, the relevant nginx config for the searx engine;
[user@host ~]$ sudo cat /etc/nginx/sites-available/searx
server {
server_name searx.domain www.searx.domain;
root /usr/local/searx;
access_log /dev/null;
error_log /dev/null;
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
}
listen 443 ssl http2; # managed by Certbot
listen [::]:443 ssl http2;
ssl_certificate
/etc/letsencrypt/live/searx.rsec.systems/fullchain.pem; # managed by Certbot
ssl_certificate_key
/etc/letsencrypt/live/searx.rsec.systems/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = www.searx.domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = searx.domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name searx.domain www.searx.domain;
return 404; # managed by Certbot
}
DNS resolution works just fine;
[user@host ~]$ dig searx.domain A,CNAME
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> searx.domain A,CNAME
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24508
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;searx.domain. IN A
;; ANSWER SECTION:
searx.domain. 278 IN CNAME domain.
domain. 278 IN A xxx.xxx.xxx.xxx
;; Query time: 21 msec
;; SERVER: 67.207.67.3#53(67.207.67.3)
;; WHEN: Tue Jul 14 03:40:59 CEST 2020
;; MSG SIZE rcvd: 77
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54492
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;A,CNAME. IN A
;; AUTHORITY SECTION:
. 3600 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2020071301 1800 900 604800 86400
;; Query time: 4 msec
;; SERVER: 67.207.67.3#53(67.207.67.3)
;; WHEN: Tue Jul 14 03:40:59 CEST 2020
;; MSG SIZE rcvd: 111
Any help pointing me into the right direction would be much appreciated.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hi there,
It seems you’ve thoroughly checked the most common issues and your system and network setup appear correct. It’s difficult to pinpoint the exact issue from this information, but here are a few additional things you can check or try:
-
Check Application Logs: You have not mentioned anything about the searx logs. There might be some information in there that points towards the cause of the problem. You can check the log files for any specific error messages.
-
Nginx Config Test: Make sure to check your nginx configuration for any syntax errors. You can use
nginx -tto test your configuration. This should tell you if there’s any syntax error in your configuration. -
uWSGI Config: Check your uWSGI configurations. You can stop uWSGI and try running it manually in the foreground to see if it’s throwing any error. Use the command
uwsgi --ini /etc/uwsgi/apps-enabled/searx.inito run it in the foreground. -
Check SSL certificates: Since you’re using SSL, check the status of your SSL certificates. Expired certificates may cause issues.
-
Access via IP: Try to access searx directly via IP and port in the web browser to make sure that the problem isn’t DNS related.
-
Check Browser Console: The browser console may have more detailed error messages about why the site can’t be reached.
-
Nginx Access and Error Logs: Review the access and error logs for Nginx. They might have useful information about the request that can help troubleshoot the issue. The error log is especially useful for this. The location of these logs depends on your Nginx setup but commonly they are located at
/var/log/nginx/access.logand/var/log/nginx/error.log.
Best,
Bobby
Check Nginx Error Logs:
Even though you’ve configured your Nginx logs to /dev/null, you’ll need to turn the logging back on temporarily to get more insight into what’s going wrong. Re-enable the error log for the site by modifying your /etc/nginx/sites-available/searx configuration:
error_log /var/log/nginx/searx_error.log;
Then restart Nginx and check the log files for clues
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and AI-native businesses
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.