The memory occupied by bash has been increasing due to the fork bomb
Hello !
I am now doing a test on the fork bomb with the command `:(){:|:&};:` and
set `ulimit - c 1000`. And everything was well in bash-5.0, but there was a
problem in bash-5.1.
The main performance is that the system memory has been rising, and then the
kernel appears the oom, I found that the memory increase may be caused by
the following reasons:
First, systemd will send a sigterm to bash, but in bash-5.1, the bash process
will not be killed. this was caused by a change in bash 5.1. The modified
change information is as follows
```
sss. Fix a bug where receiving SIGTERM from a different process while readline
was active could cause the shell to terminate
```
Second, When bash fork is a child process, it always creates a memory to
manage the job, even if the maximum number set by the ulimit command is reached
```
int stop_pipeline(async, deferred)
int async;
COMMAND *deferred;
{
... ...
if (js.jobslots == 0) {
... ...
jobs = (JOB **)xmalloc(js.j_jobslots * sizeof(JOB
*));
... ...
}
... ...
if (i == js.jobslots) {
... ...
jobs = (JOB **)xrealloc(js.j_jobslots * sizeof(JOB
*));
... ...
}
}
```
After my own positioning, the above is probably the reason for the memory
increase. Could you tell me how to avoid or optimize memory increase ?
thanks ??
Re: The memory occupied by bash has been increasing due to the fork bomb
> First, systemd will send a sigterm to bash, but in bash-5.1, the bash process will not be killed. this was caused by a > > > change in bash 5.1. The modified change information is as follows > ``` > sss. Fix a bug where receiving SIGTERM from a different process while readline was active could cause the shell to terminate > ``` Sorry, it's wrong. The sigterm signal is sent by bash instead of systemd, Relevant log information is as follows ``` Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:[384404]) send SIGTERM to comm:bash exe:bash[384593].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384404](parent comm:systemd parent exe:systemd[1]) send SIGTERM to comm:bash exe:[384592]. Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:[384404]) send SIGTERM to comm:bash exe:bash[384592].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:bash[384404]) send SIGTERM to comm:bash exe:[384587].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384404](parent comm:systemd parent exe:systemd[1]) send SIGTERM to comm:bash exe:bash[384587]. Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:bash[384404]) send SIGTERM to comm:bash exe:[384586].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384404](parent comm:systemd parent exe:systemd[1]) send SIGTERM to comm:bash exe:bash[384586]. Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384404](parent comm:systemd parent exe:systemd[1]) send SIGTERM to comm:bash exe:[384584]. Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:[384404]) send SIGTERM to comm:bash exe:bash[384584].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:bash[384404]) send SIGTERM to comm:bash exe:[384583].(systemd[1]) Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384404](parent comm:systemd parent exe:systemd[1]) send SIGTERM to comm:bash exe:bash[384583]. Mar 10 08:52:58 localhost.localdomain sysmonitor[1191]: sysmonitor[1191]: comm:bash exe:bash[384578](parent comm:bash parent exe:bash[384404]) send SIGTERM to comm:bash exe:[384580].(systemd[1]) ```
Re: The memory occupied by bash has been increasing due to the fork bomb
On 3/10/23 5:04 AM, wang yuhang via Bug reports for the GNU Bourne Again SHell wrote: > First, systemd will send a sigterm to bash, but in bash-5.1, the bash process will not be killed. this was caused by a > > > change in bash 5.1. The modified change information is as follows > ``` > sss. Fix a bug where receiving SIGTERM from a different process while readline was active could cause the shell to terminate > ``` Sorry, it's wrong. The sigterm signal is sent by bash instead of systemd, Relevant log information is as follows Interactive shells always ignore SIGTERM. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: The memory occupied by bash has been increasing due to the fork bomb
> Interactive shells always ignore SIGTERM. I confirmed that the fork bomb through bash would cause the system oom! This indicates that anybody can use this flaw to crash the system.It is quite dangerous. If you think the behavior of ignoring the SIGTERM is reasonable. Maybe the only way to solve the problem is to deal with the increasing of the memory?
Re: The memory occupied by bash has been increasing due to the fork bomb
zju <21625...@zju.edu.cn> writes: >> Interactive shells always ignore SIGTERM. > > I confirmed that the fork bomb through bash would cause the system > oom! This indicates that anybody can use this flaw to crash the > system.It is quite dangerous. > > If you think the behavior of ignoring the SIGTERM is reasonable. Maybe > the only way to solve the problem is to deal with the increasing of > the memory? The Un*x convention has always been that SIGTERM kills the process but the process can override that, and SIGKILL kills the process and the process cannot override that. So if systemd isn't protecting the system adequately with its current operation, it should instead send SIGKILL. In regard to OOM, if the goal is to prevent fork bombs, the system administrator would need to set a hard limit on "ulimit -u", "The maximum number of processes available to a single user" as well as "ulimit -d", "The maximum size of a process's data segment". Changing the behavior of bash alone could not prevent an attacker from forcing OOM, it would just require the attacker to be more sophisticated. Dale
Re: The memory occupied by bash has been increasing due to the fork bomb
>The Un*x convention has always been that SIGTERM kills the process but >the process can override that, and SIGKILL kills the process and the >process cannot override that. So if systemd isn't protecting the system >adequately with its current operation, it should instead send SIGKILL. As yuhang said the sigterm signal is sent by bash instead of systemd. And I think ignore the SIGTERM is reasonable as well. >In regard to OOM, if the goal is to prevent fork bombs, the system >administrator would need to set a hard limit on "ulimit -u", “The >maximum number of processes available to a single user" as well as >"ulimit -d", "The maximum size of a process's data segment". Changing >the behavior of bash alone could not prevent an attacker from forcing >OOM, it would just require the attacker to be more sophisticated. I have already set the maximum number of processes available to a single user "ulimit -Su”. But the memory occupied by bashes were increasing all the time which would call oom.This is the key issue. As you can see the details below [root@fedora parallels]# ps aux | grep bash | grep paralle | head paralle+ 4640 0.0 0.0 6892 72 ? Ss 22:54 0:00 /usr/bin/ssh-agent /bin/sh -c exec -l /bin/bash -c "/usr/bin/gnome-session" paralle+ 5563 0.0 0.1 224516 3212 pts/0 Ss+ 22:54 0:00 bash paralle+ 6269 0.0 0.2 224252 4956 pts/1 Ss 22:57 0:00 bash paralle+ 51377 0.0 0.1 224820 2808 pts/0 S 23:14 0:00 bash paralle+ 51410 0.0 0.1 224964 2704 pts/0 S 23:14 0:00 bash paralle+ 51558 0.0 0.1 224804 2796 pts/0 S 23:14 0:00 bash paralle+ 51662 0.0 0.1 224800 2788 pts/0 S 23:14 0:00 bash paralle+ 51704 0.0 0.1 224796 2768 pts/0 S 23:14 0:00 bash paralle+ 51711 0.0 0.1 224816 2820 pts/0 S 23:14 0:00 bash paralle+ 52051 0.0 0.1 224952 2832 pts/0 S 23:14 0:00 bash [root@fedora parallels]# free -h total used free shared buff/cache available Mem: 1.9Gi 1.3Gi 44Mi 106Mi 590Mi 440Mi Swap: 1.9Gi 109Mi 1.8Gi [root@fedora parallels]# ps aux | grep bash | grep paralle | head paralle+ 4640 0.0 0.0 6892 1000 ? Ss 22:54 0:00 /usr/bin/ssh-agent /bin/sh -c exec -l /bin/bash -c "/usr/bin/gnome-session" paralle+ 5563 0.0 0.1 224516 3040 pts/0 Ss+ 22:54 0:00 bash paralle+ 6269 0.0 0.2 224252 4552 pts/1 Ss 22:57 0:00 bash paralle+ 104137 0.0 0.1 225632 2984 pts/0 S 23:26 0:00 bash paralle+ 104176 0.0 0.1 225668 3060 pts/0 S 23:26 0:00 bash paralle+ 104183 0.0 0.1 225668 3060 pts/0 S 23:26 0:00 bash paralle+ 104187 0.0 0.1 225664 2800 pts/0 S 23:26 0:00 bash paralle+ 104396 0.0 0.1 225628 3048 pts/0 S 23:26 0:00 bash paralle+ 104412 0.0 0.1 225628 3048 pts/0 S 23:26 0:00 bash paralle+ 104423 0.0 0.1 225668 3064 pts/0 S 23:26 0:00 bash [root@fedora parallels]# ps aux | grep bash | grep paralle | wc -l 714 >Second, When bash fork is a child process, it always creates a memory to >manage the job, even if the maximum number set by the ulimit command is reached Maybe as yuhang said at the very beginning that the memory occupied by the bash has been growing all the time.
Re: The memory occupied by bash has been increasing due to the fork bomb
On 3/10/23 4:28 AM, wang yuhang via Bug reports for the GNU Bourne Again SHell wrote: Second, When bash fork is a child process, it always creates a memory to manage the job, even if the maximum number set by the ulimit command is reached ``` If the maximum number of child processes is exceeded, fork(2) fails, and this code is not executed. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: The memory occupied by bash has been increasing due to the fork bomb
On 3/10/23 11:37 AM, zju wrote: I have already set the maximum number of processes available to a single user "ulimit -Su”. But the memory occupied by bashes were increasing all the time which would call oom.This is the key issue. If you don't see an error message from bash about fork failing, then fork hasn't failed, and the processes continue to run. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: The memory occupied by bash has been increasing due to the fork bomb
> 2023年3月11日 06:17，Chet Ramey 写道：
>
> On 3/10/23 11:37 AM, zju wrote:
>
>> I have already set the maximum number of processes available to a single
>> user "ulimit -Su”.
>> But the memory occupied by bashes were increasing all the time which would
>> call oom.This is the key issue.
>
> If you don't see an error message from bash about fork failing, then fork
> hasn't failed, and the processes continue to run.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Thanks for you reply:
I want to express this problem more clearly：
1.I set "ulimit -Su 30” to observe the situation;
2.exec
1.Actually there are many error messages already.
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
2.exec fork book in bash
[parallels@fedora ~]$ :() { :|:& };
3.The following error will appear soon
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
4.And I observe the situation in other terminal.
As you can see. The process 250229(bash) continues to pull new child processes
which the rss is larger than the old process(2096->2112). So the memory
occupied by the bashes keep growing even though the num of bash processes is
the same, if 8M is occupied by each bash process, there maybe 4G occupied by
500 bash processes totally.
And as the memory occupied by each bash process is not enough, the oom killer
would not take bash as the first target to kill.
So is it possible to optimize the continuous growth of memory occupied by child
processes？
> In regard to OOM, if the goal is to prevent fork bombs, the system
> administrator would need to set a hard limit on "ulimit -u", “The
> maximum number of processes available to a single user" as well as
> "ulimit -d", "The maximum size of a process's data segment". Changing
> the behavior of bash alone could not prevent an attacker from forcing
> OOM, it would just require the attacker to be more sophisticated.
Or is there anyway to avoid this problem？
As I used to use ulimit -Su to limit the process on bash-5.0 which dosen’t work
now.
I doubt that whether Worley said using "ulimit -d” with "ulimit -u” could avoid
this problem as the rss maybe occupied by the stack rather than the data
segment?
Looking forward to your reply!
[root@fedora ~]# ps aux | grep bash | grep paralle
paralle+ 250229 0.0 0.1 224500 3828 pts/0S+ 09:39 0:00 -bash
paralle+ 255620 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255621 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255622 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255623 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255624 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255625 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255626 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255627 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255628 0.0 0.1 224368 2104 pts/0S10:10 0:00 -bash
paralle+ 255629 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255630 0.0 0.1 224368 2104 pts/0S10:10 0:00 -bash
paralle+ 255631 0.0 0.1 224368 2100 pts/0S10:10 0:00 -bash
paralle+ 255632 0.0 0.1 224368 2100 pts/0S10:10 0:00 -bash
paralle+ 255633 0.0 0.1 224368 2100 pts/0S10:10 0:00 -bash
paralle+ 255634 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255635 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255636 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255637 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255638 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255639 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255640 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255641 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255642 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255643 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255644 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255645 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255646 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255647 0.0 0.1 224368 2096 pts/0S10:10 0:00 -bash
paralle+ 255648 0.0 0.1 224368 2104 pts/0S10:10 0:00 -bash
[root@fedora ~]# ps aux | grep bash | grep paralle
paralle+ 250229 0.0 0.1 224500 3828 pts/0S+ 09:39 0:00 -bash
paralle+ 255708 0.0 0.1 224368 2108 pts/0S10:10 0:00 -bash
paralle+ 255709 0.0 0.1 224368 2108 pts/0S
Re: The memory occupied by bash has been increasing due to the fork bomb
On Sat, Mar 11, 2023 at 10:50:21AM +0800, zju wrote: > So is it possible to optimize the continuous growth of memory occupied by > child processes？ Someone will have to find where the memory leak is occurring. You'll want a more controlled reproducer than an infinite fork bomb. > Or is there anyway to avoid this problem？ Depends on which problem you mean. If you're exclusively concerned about the memory leak, then finding and fixing the bug will take care of it. If you mean "prevent people from running a fork bomb", stop giving untrusted people shells on your system. That's basically it.
