> From: Kevin E. Thorpe [mailto:kevin.tho...@utoronto.ca] > Sent: Tuesday, August 13, 2013 2:25 PM > Subject: Re: [R] Memory limit on Linux? > > It appears that at the shell level, the differences are not to blame. > It has been a long time, but years ago in HP-UX, we needed to change an > actual kernel parameter (this was for S-Plus 5 rather than R back then). > Despite the ulimits being acceptable, there was a hard limit in the kernel. > I don't know whether such things have been (or can be) built in to your > "problem" machine. If it is a multiuser box, it could be that limits have > been set to prevent a user from gobbling up all the memory.
I thought about that too as I was not the admin who built the box (I took over for him), but I don't see anything at all in the kernel, climits, or anything else that suggests this is the case. > The other thing to check is if R has/can be compiled with memory limits. That was the second question of my original post. I looked but I don't see anything related to this except for various posts dealing with a windows box. > Sorry I can't be of more help. No worries. I do appreciate your help. I finally got access to the code and data set for my own testing so I did some more research while running a job. I looked around in the /proc/$taskid folder [on the box that seems to stop using memory around 5GB] but nothing really jumped out at me. Maybe someone else will catch something I missed. $ cat limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size unlimited unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1024 2066361 processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 2066361 2066361 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us $ cat statm 1347444 1312394 752 1 0 1312166 0 Memory Size=1347444~=5263M Resident memory=1312394~=5126M Libraries=1312166~=5125M $cat status Name: R State: R (running) Tgid: 23015 Pid: 23015 PPid: 23011 TracerPid: 0 Uid: 1701 1701 1701 1701 Gid: 680 680 680 680 Utrace: 0 FDSize: 256 Groups: 680 VmPeak: 5389776 kB VmSize: 5358464 kB VmLck: 0 kB VmHWM: 5249576 kB VmRSS: 5218420 kB VmData: 5217160 kB VmStk: 192 kB VmExe: 4 kB VmLib: 9200 kB VmPTE: 10352 kB VmSwap: 0 kB Threads: 1 SigQ: 0/2066361 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000000000 SigCgt: 0000000180001e4a CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffffff Cpus_allowed_list: 0-31 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f Mems_allowed_list: 0-3 voluntary_ctxt_switches: 98 nonvoluntary_ctxt_switches: 101913 On the system in which more memory allocates freely (this particular job isn't quite as big as the others, so ~10GB in size is correct): $ cat limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 773356 773356 processes Max open files 1024 1024 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 773356 773356 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 $ cat statm 2746521 2716426 841 1 0 2716928 0 Memory Size=2746521~=10728M Resident memory=2716426~=10611M Libraries=2716928~=10613M $ cat status Name: R State: R (running) SleepAVG: 0% Tgid: 17003 Pid: 17003 PPid: 16984 TracerPid: 0 Uid: 46877 46877 46877 46877 Gid: 25043 25043 25043 25043 FDSize: 256 Groups: 351 25043 48628 59110 VmPeak: 11109136 kB VmSize: 10986084 kB VmLck: 0 kB VmHWM: 10988628 kB VmRSS: 10865704 kB VmData: 10867572 kB VmStk: 140 kB VmExe: 4 kB VmLib: 19892 kB VmPTE: 21396 kB StaBrk: 1f1a5000 kB Brk: 1b310b000 kB StaStk: 7fff0ddf6ba0 kB Threads: 1 SigQ: 1/773356 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000000000 SigCgt: 0000000180001e4a CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00ffffff Mems_allowed: 00000000,00000003 Thanks! Chris Stackpole ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
