What is the /tmp/mule process?

November 28, 2017 114 views
PostgreSQL CentOS

In my company we have implemented a distributed cluster using the CitusData extension, the virtual machines we've using are located on the DigitalOcean premises. About two or three weeks ago we have noticed that one of the nodes where extremely slow (when doing queries to the master node), and the cores at full capacity. So the first move where to check the processes load and this was what we got issuing (only a few entries shown):

Issuing top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20398 postgres 20 0 462040 5088 836 S 10.3 0.1 20:00.38 mule
23139 postgres 20 0 462040 5156 864 S 10.3 0.1 303:25.07 mule
3361 postgres 20 0 462040 5056 784 S 10.0 0.1 52:28.02 mule
13159 postgres 20 0 462040 5076 772 S 10.0 0.1 113:03.39 mule
16580 postgres 20 0 396504 4792 796 S 10.0 0.1 27:28.25 mule
32372 postgres 20 0 265432 5144 920 S 9.6 0.1 0:03.29 mule
...........
...........

Issuing ps au|grep mule

postgres 1365 5.0 0.1 396504 4904 ? Ssl 13:04 8:12 /tmp/mule
postgres 1896 11.8 0.1 462040 5028 ? Ssl Nov24 262:02 /tmp/mule
postgres 2377 5.6 0.1 462040 4988 ? Ssl 04:03 39:43 /tmp/mule
postgres 3361 8.2 0.1 462040 5124 ? Ssl Nov24 141:37 /tmp/mule
postgres 3629 6.6 0.1 462040 4916 ? Ssl Nov24 82:51 /tmp/mule
postgres 4692 4.9 0.1 330968 4996 ? Ssl 14:04 5:00 /tmp/mule
postgres 5577 11.1 0.1 462040 5044 ? Ssl Nov24 240:06 /tmp/mule
......
......

We restarted that node and all went back to normal, but a few days later, the same ~160 mule processes where initiated by the postgres user. This happends only in one particular node. All of them are replicas (2cores and 8gb ram each) so all of them run postgresql-9.6 and CitusData_7.1 over CentOS 7.

We've done some research but with no luck and we can't find any docs or issues regarding these processes, only documentation about MuleSoft , but we never installed anything of such.

We also contacted the CitusData developers and they also have no idea what that process is. We've been using postgresql for quite a while but never seeing this. Is this any kind of malware ? Or maybe a background DigitalOcean's process. Feel free to ask for more details.

1 Answer

That looks like your node has been compromised, time for chkrootkit and rkhunter.

Have another answer? Share your knowledge.