CPU 100% usage caused by unknown postgres process..

2017-12-13 Thread Dinesh Chandra 12108 
Hi,

My CPU utilization is going to 100% in PostgreSQL because of one unknown 
process /x330341 is running from postgres user.

PID   USER  PR  NIVIRTRESSHR S %CPU %MEM   TIME+   COMMAND
19885 postgres 20   0  192684   3916   1420 S 99.3  0.1   5689:04  x330341

The same file is automatically created in Postgres Cluster also. I am using 
Postgresql-9.3.

Kindly suggest how can I resolve this issue.

Regar
Dinesh Chandra
|Database administrator (Oracle/PostgreSQL)| Cyient Ltd. Noida.
--
Mobile: +91-9953975849 | Ext 1078 
|[email protected]
Plot No. 7, NSEZ, Phase-II ,Noida-Dadri Road, Noida - 201 305,India.




DISCLAIMER:

This email message is for the sole use of the intended recipient(s) and may 
contain confidential and privileged information. Any unauthorized review, use, 
disclosure or distribution is prohibited. If you are not the intended 
recipient, please contact the sender by reply email and destroy all copies of 
the original message. Check all attachments for viruses before opening them. 
All views or opinions presented in this e-mail are those of the author and may 
not reflect the opinion of Cyient or those of our affiliates.

Re: CPU 100% usage caused by unknown postgres process..

2017-12-13 Thread Laurenz Albe 
Dinesh Chandra 12108 wrote:
> My CPU utilization is going to 100% in PostgreSQL because of one unknown 
> process /x330341 is running from postgres user.
>  
> PID   USER  PR  NIVIRTRESSHR S %CPU %MEM   TIME+   COMMAND
> 19885 postgres 20   0  192684   3916   1420 S 99.3  0.1   5689:04  
> x330341 
>  
> The same file is automatically created in Postgres Cluster also. I am using 
> Postgresql-9.3.
>  
> Kindly suggest how can I resolve this issue.

I don't know, but the same problem has been reported on Stackoverflow:
https://stackoverflow.com/q/46617329/6464308

If your queries look similar, then you might indeed be the victim of an attack.

Figure out where the function and the executable come from.

In case of doubt, disconnect the server from the network.

Yours,
Laurenz Albe

Re: CPU 100% usage caused by unknown postgres process..

2017-12-13 Thread Justin Pryzby 
On Wed, Dec 13, 2017 at 11:36:11AM +0100, Laurenz Albe wrote:
> Dinesh Chandra 12108 wrote:
> > My CPU utilization is going to 100% in PostgreSQL because of one unknown 
> > process /x330341 is running from postgres user.
> >  
> > PID   USER  PR  NIVIRTRESSHR S %CPU %MEM   TIME+   COMMAND
> > 19885 postgres 20   0  192684   3916   1420 S 99.3  0.1   5689:04  
> > x330341 
> >  
> > The same file is automatically created in Postgres Cluster also. I am using 
> > Postgresql-9.3.
> >  
> > Kindly suggest how can I resolve this issue.
> 
> I don't know, but the same problem has been reported on Stackoverflow:
> https://stackoverflow.com/q/46617329/6464308
> 
> If your queries look similar, then you might indeed be the victim of an 
> attack.
> 
> Figure out where the function and the executable come from.
> 
> In case of doubt, disconnect the server from the network.

Looks suspicious; I would look at (and save) things like these:

ls -l /proc/19885/exe
ls -l /proc/19885/fd
ls -l /proc/19885/cwd

sudo lsof -n -p 19885
sudo netstat -anpe |grep 19885

Stacktrace with gcore/gdb is a good idea.
Save a copy of your log/postgres logfiles and try to figure out where it came
from.  Since an attacker seems to control the postgres process, your data may
have been compromized (leaked or tampered with).

Justin

Re: CPU 100% usage caused by unknown postgres process..

2017-12-13 Thread Tomas Vondra 


On 12/13/2017 01:19 PM, Justin Pryzby wrote:
> On Wed, Dec 13, 2017 at 11:36:11AM +0100, Laurenz Albe wrote:
>> Dinesh Chandra 12108 wrote:
>>> My CPU utilization is going to 100% in PostgreSQL because of one unknown 
>>> process /x330341 is running from postgres user.
>>>  
>>> PID   USER  PR  NIVIRTRESSHR S %CPU %MEM   TIME+   COMMAND
>>> 19885 postgres 20   0  192684   3916   1420 S 99.3  0.1   5689:04  
>>> x330341 
>>>  
>>> The same file is automatically created in Postgres Cluster also. I am using 
>>> Postgresql-9.3.
>>>  
>>> Kindly suggest how can I resolve this issue.
>>
>> I don't know, but the same problem has been reported on Stackoverflow:
>> https://stackoverflow.com/q/46617329/6464308
>>
>> If your queries look similar, then you might indeed be the victim of an 
>> attack.
>>
>> Figure out where the function and the executable come from.
>>
>> In case of doubt, disconnect the server from the network.
> 
> Looks suspicious; I would look at (and save) things like these:
> 
> ls -l /proc/19885/exe
> ls -l /proc/19885/fd
> ls -l /proc/19885/cwd
> 
> sudo lsof -n -p 19885
> sudo netstat -anpe |grep 19885
> 
> Stacktrace with gcore/gdb is a good idea.
> Save a copy of your log/postgres logfiles and try to figure out where it came
> from.  Since an attacker seems to control the postgres process, your data may
> have been compromized (leaked or tampered with).
> 

Any details about the x330341 file (is it a shell script or some
kind of binary)?

FWIW the queries (listed in the stackoverflow post) are running under
postgres, which I assume is superuser. The backend has full access to
the data directory, of course, so it may create extra files (using
adminpack extension, for example).

If that's the case (and if it's indeed an attack), it either means the
attacker likely already has access to all the data. So presumably
x330341 is doing something else at the OS level.

regards

-- 
Tomas Vondra  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services