Last night, my server was under attack. This happens a lot, and shouldn’t be reason to panic. However, my node.js application sets up a connection with MongoDB, and this timed out. So this morning, the MongoDB was unreachable until I restarted the node.js process. At first, I figured out what was wrong because of this error in the nginx error.log file, there were lots of these errors:

upstream timed out (110: Connection timed out) while reading response header from upstream

Then, I started to inspect my console logs of node.js, and found the following:

[Sat, 09 May 2020 00:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.514 ms
[Sat, 09 May 2020 01:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.328 ms
[Sat, 09 May 2020 01:31:31 GMT] GET /solr/admin/info/system?wt=json 404 161 - ::ffff:127.0.0.1 - 0.283 ms
[Sat, 09 May 2020 01:39:47 GMT] GET /?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php> 200 382 - ::ffff:127.0.0.1 - 0.543 ms
[Sat, 09 May 2020 01:39:47 GMT] GET /?XDEBUG_SESSION_START=phpstorm 200 382 - ::ffff:127.0.0.1 - 0.501 ms
[Sat, 09 May 2020 01:42:40 GMT] GET /.git/config 404 150 - ::ffff:127.0.0.1 - 0.241 ms
[Sat, 09 May 2020 01:48:04 GMT] POST /GponForm/diag_Form?style/ 404 158 - ::ffff:127.0.0.1 - 0.499 ms
[Sat, 09 May 2020 01:54:18 GMT] GET /index.php?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP 404 148 - ::ffff:127.0.0.1 - 0.2$
[Sat, 09 May 2020 02:01:37 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 0.516 ms
[Sat, 09 May 2020 02:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.396 ms
[Sat, 09 May 2020 02:49:15 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.597 ms
[Sat, 09 May 2020 02:58:03 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.016 ms
events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: read ETIMEDOUT
    at _errnoException (util.js:1022:11)
    at TCP.onread (net.js:628:25)
error: Forever detected script exited with code: 1
error: Script restart attempt #127
Server is up and running
MongoDB connection error: { MongooseTimeoutError: Server selection timed out after 30000 ms
    at new MongooseTimeoutError (/root/node_modules/mongoose/lib/error/timeout.js:22:11)
    at NativeConnection.Connection.openUri (/root/node_modules/mongoose/lib/connection.js:803:19)
    at Mongoose.connect (/root/node_modules/mongoose/lib/index.js:332:15)
    at Object.<anonymous> (/root/index.js:131:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
  message: 'Server selection timed out after 30000 ms',
  name: 'MongooseTimeoutError',
  [Symbol(mongoErrorContextSymbol)]: {} }
(node:7896) UnhandledPromiseRejectionWarning: MongooseTimeoutError: Server selection timed out after 30000 ms
    at new MongooseTimeoutError (/root/node_modules/mongoose/lib/error/timeout.js:22:11)
    at NativeConnection.Connection.openUri (/root/node_modules/mongoose/lib/connection.js:803:19)
    at Mongoose.connect (/root/node_modules/mongoose/lib/index.js:332:15)
    at Object.<anonymous> (/root/index.js:131:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
(node:7896) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecti$
(node:7896) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process wit$
[Sat, 09 May 2020 03:19:59 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 7.701 ms
[Sat, 09 May 2020 03:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 1.523 ms
[Sat, 09 May 2020 04:06:50 GMT] POST /api/jsonws/invoke 404 157 - ::ffff:127.0.0.1 - 20.066 ms
[Sat, 09 May 2020 04:06:50 GMT] GET /?XDEBUG_SESSION_START=phpstorm 200 382 - ::ffff:127.0.0.1 - 0.729 ms
[Sat, 09 May 2020 04:06:50 GMT] GET /index.php?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP 404 148 - ::ffff:127.0.0.1 - 0.6$
[Sat, 09 May 2020 04:07:17 GMT] GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php 404 189 - ::ffff:127.0.0.1 - 1.356 ms
[Sat, 09 May 2020 04:07:19 GMT] POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php 404 190 - ::ffff:127.0.0.1 - 1.735 ms
[Sat, 09 May 2020 04:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.846 ms
[Sat, 09 May 2020 04:35:36 GMT] GET /api/jsonws/invoke 404 156 - ::ffff:127.0.0.1 - 0.365 ms
[Sat, 09 May 2020 05:06:04 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 0.706 ms
[Sat, 09 May 2020 05:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.366 ms
[Sat, 09 May 2020 05:26:19 GMT] GET /wp-login.php 404 151 - ::ffff:127.0.0.1 - 0.394 ms
[Sat, 09 May 2020 06:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.352 ms
[Sat, 09 May 2020 07:23:55 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.413 ms
[Sat, 09 May 2020 07:42:09 GMT] GET / 200 382 - ::ffff:127.0.0.1 - 1.998 ms
[Sat, 09 May 2020 07:54:13 GMT] HEAD / 200 382 - ::ffff:127.0.0.1 - 0.339 ms

So, while nothing got compromised, my MongoDB timed out, and was unable to connect for a couple of hours, until I reset the node.js connection myself. For your information, I am using 3 droplets: one for node.js (1 GB Memory / 25 GB Disk / AMS3 - Ubuntu 18.04.3 (LTS) x64), one for mysql database (2 GB Memory / 50 GB Disk / AMS3 - Ubuntu MySQL on 18.04) and one for mongodb database (1 GB Memory / 25 GB Disk / AMS3 - Ubuntu MongoDB 4.0.3 on 18.04).

How could I avoid this in the future?