On 02/15/2018 02:04 AM, John Hearns via Beowulf wrote:
Hmmm... I will also chip in with my favourite tip
Look at the sysctl for min_free_kbytes It is often set very low.
Increase this substantially. It will do no harm to your system (unless
you set it ti an absurd value!)
You should be looking at the vm dirty ratios etc. also
+1
vm.dirty_background_bytes
vm.dirty_bytes
(or the corresponding _ratios)
vm.min_free_kbytes
Defaults are low.
Increasing them improved a lot our compute nodes IO.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-tunables
On 15 February 2018 at 00:44, Kilian Cavalotti
<kilian.cavalotti.w...@gmail.com
<mailto:kilian.cavalotti.w...@gmail.com>> wrote:
On Wed, Feb 14, 2018 at 2:26 PM, David Mathog <mat...@caltech.edu
<mailto:mat...@caltech.edu>> wrote:
> Checked the hugepage settings and found a difference there. The two
systems
> that don't do this have /sys/kernel/mm/redhat_transparent_hugepage/defrag
>
> always madvise [never]
>
> whereas the system with the issue has:
>
> [always] madvise never
THP defragmentation is definitely something that has bitten us in the
past, when under memory pressure, and we now default to [madvise]
pretty much everywhere (we're too timid to disable it entirely).
A good way to see if that's really the issue is to "echo never >
/sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
is happening, while simultaneously monitoring the processes with htop,
for instance.
It's usually pretty instant: if the issue is really with THP defrag,
then CPU usage for your stalling process should drop pretty much
immediately and things go back to normal.
Cheers,
--
Kilian
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
<http://www.beowulf.org/mailman/listinfo/beowulf>
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
