Last night, my server was under attack. This happens a lot, and shouldn’t be reason to panic. However, my node.js application sets up a connection with MongoDB, and this timed out. So this morning, the MongoDB was unreachable until I restarted the node.js process. At first, I figured out what was wrong because of this error in the nginx error.log file, there were lots of these errors:

upstream timed out (110: Connection timed out) while reading response header from upstream

Then, I started to inspect my console logs of node.js, and found the following:

[Sat, 09 May 2020 00:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.514 ms
[Sat, 09 May 2020 01:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.328 ms
[Sat, 09 May 2020 01:31:31 GMT] GET /solr/admin/info/system?wt=json 404 161 - ::ffff:127.0.0.1 - 0.283 ms
[Sat, 09 May 2020 01:39:47 GMT] GET /?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php> 200 382 - ::ffff:127.0.0.1 - 0.543 ms
[Sat, 09 May 2020 01:39:47 GMT] GET /?XDEBUG_SESSION_START=phpstorm 200 382 - ::ffff:127.0.0.1 - 0.501 ms
[Sat, 09 May 2020 01:42:40 GMT] GET /.git/config 404 150 - ::ffff:127.0.0.1 - 0.241 ms
[Sat, 09 May 2020 01:48:04 GMT] POST /GponForm/diag_Form?style/ 404 158 - ::ffff:127.0.0.1 - 0.499 ms
[Sat, 09 May 2020 01:54:18 GMT] GET /index.php?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP 404 148 - ::ffff:127.0.0.1 - 0.2$
[Sat, 09 May 2020 02:01:37 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 0.516 ms
[Sat, 09 May 2020 02:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.396 ms
[Sat, 09 May 2020 02:49:15 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.597 ms
[Sat, 09 May 2020 02:58:03 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.016 ms
events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: read ETIMEDOUT
    at _errnoException (util.js:1022:11)
    at TCP.onread (net.js:628:25)
error: Forever detected script exited with code: 1
error: Script restart attempt #127
Server is up and running
MongoDB connection error: { MongooseTimeoutError: Server selection timed out after 30000 ms
    at new MongooseTimeoutError (/root/node_modules/mongoose/lib/error/timeout.js:22:11)
    at NativeConnection.Connection.openUri (/root/node_modules/mongoose/lib/connection.js:803:19)
    at Mongoose.connect (/root/node_modules/mongoose/lib/index.js:332:15)
    at Object.<anonymous> (/root/index.js:131:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
  message: 'Server selection timed out after 30000 ms',
  name: 'MongooseTimeoutError',
  [Symbol(mongoErrorContextSymbol)]: {} }
(node:7896) UnhandledPromiseRejectionWarning: MongooseTimeoutError: Server selection timed out after 30000 ms
    at new MongooseTimeoutError (/root/node_modules/mongoose/lib/error/timeout.js:22:11)
    at NativeConnection.Connection.openUri (/root/node_modules/mongoose/lib/connection.js:803:19)
    at Mongoose.connect (/root/node_modules/mongoose/lib/index.js:332:15)
    at Object.<anonymous> (/root/index.js:131:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
(node:7896) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecti$
(node:7896) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process wit$
[Sat, 09 May 2020 03:19:59 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 7.701 ms
[Sat, 09 May 2020 03:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 1.523 ms
[Sat, 09 May 2020 04:06:50 GMT] POST /api/jsonws/invoke 404 157 - ::ffff:127.0.0.1 - 20.066 ms
[Sat, 09 May 2020 04:06:50 GMT] GET /?XDEBUG_SESSION_START=phpstorm 200 382 - ::ffff:127.0.0.1 - 0.729 ms
[Sat, 09 May 2020 04:06:50 GMT] GET /index.php?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP 404 148 - ::ffff:127.0.0.1 - 0.6$
[Sat, 09 May 2020 04:07:17 GMT] GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php 404 189 - ::ffff:127.0.0.1 - 1.356 ms
[Sat, 09 May 2020 04:07:19 GMT] POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php 404 190 - ::ffff:127.0.0.1 - 1.735 ms
[Sat, 09 May 2020 04:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.846 ms
[Sat, 09 May 2020 04:35:36 GMT] GET /api/jsonws/invoke 404 156 - ::ffff:127.0.0.1 - 0.365 ms
[Sat, 09 May 2020 05:06:04 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 0.706 ms
[Sat, 09 May 2020 05:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.366 ms
[Sat, 09 May 2020 05:26:19 GMT] GET /wp-login.php 404 151 - ::ffff:127.0.0.1 - 0.394 ms
[Sat, 09 May 2020 06:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.352 ms
[Sat, 09 May 2020 07:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.413 ms
[Sat, 09 May 2020 07:42:09 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.998 ms
[Sat, 09 May 2020 07:54:13 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.339 ms

So, while nothing got compromised, my MongoDB timed out, and was unable to connect for a couple of hours, until I reset the node.js connection myself. For your information, I am using 3 droplets: one for node.js (1 GB Memory / 25 GB Disk / AMS3 - Ubuntu 18.04.3 (LTS) x64), one for mysql database (2 GB Memory / 50 GB Disk / AMS3 - Ubuntu MySQL on 18.04) and one for mongodb database (1 GB Memory / 25 GB Disk / AMS3 - Ubuntu MongoDB 4.0.3 on 18.04).

How could I avoid this in the future?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hi there @PennyWise94,

A quick fix to this would be to use CloudFlare, that way you would ‘hide’ your servers behind their CDN, and attackers would not be able to flood the Nginx service directly.

Also, Cloudflare has DDoS protection which you could enable at any time for free. You can even use a script to enable and disable the DDoS protection on demand.

A couple of years ago I wrote a short script that checks your Droplet’s CPU and if it is higher than a certain value it enables your DDoS protection. I’ve not tested it in a while but you could take a look at it here. I’ll put it on my to-do list to review that script.

Hope that this helps!
Regards,
Bobby

Submit an Answer