Hi all, I think this is only the second time that I have posted so I apologise
if my etiquette isn't quite correct.
I'm loading a large (~30GB) geojson file into R using readOGR on a HPC. I am
also loading a small shapefile, and then trying to undertake some processing on
the large geojson using gBuffer from the rgeos package.
I believe that the HPC is running Red Hat Enterprise Linux 7.4, and it
certainly has around 750 GB free for user jobs. I have allocated the full
amount of ram to the job.
I previously used the following modules to undertake this task and it ran
successfully, although only after tweaking the settings that I detail below -
otherwise I had the same error:
module load proj/5.0.0
module load gdal/2.3.1
module load geos/3.6.2
module load gcc/6.4.0
module load R/3.5.2
module load python # python 3 by default
module load numpy/1.14.0 # requires module load python
Settings at linux command line that previously allowed a successful run:
R_MAX_VSIZE=720G
R_GC_MEM_GROW=0
--min-nsize=50000k --min-vsize=12M --max-ppsize=500000 (when executing the R
script from command line)
However, the modules have now been updated on the HPC, and so I am now using:
module load proj/6.1.1
module load R/3.6.2
(other modules remain the same)
I get the following error whilst processing (loading the file into R is ok),
with gcinfo() turned on:
Garbage collection 144 = 86+22+36 (level 0) ...
288541.3 Mbytes of cons cells used (66%)
55320.1 Mbytes of vectors used (98%)
Garbage collection 145 = 86+23+36 (level 1) ...
66679.3 Mbytes of cons cells used (15%)
56447.7 Mbytes of vectors used (100%)
Garbage collection 146 = 86+23+37 (level 2) ...
39852.3 Mbytes of cons cells used (11%)
49032.4 Mbytes of vectors used (72%)
Garbage collection 147 = 87+23+37 (level 0) ...
124935.0 Mbytes of cons cells used (36%)
64961.0 Mbytes of vectors used (95%)
Garbage collection 148 = 87+24+37 (level 1) ...
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)
Error: cons memory exhausted (limit reached?)
In addition: Warning message:
Garbage collection 149 = 88+24+37 (level 0) ...
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)
Lost warning messages
Execution halted
Garbage collection 150 = 89+24+37 (level 0) ...
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)
Error: cons memory exhausted (limit reached?)
The job halts with Memory Utilized: 411.29 GB
I cannot understand why the job worked previously (just) but now does not when
seemingly the only change is an updated proj and R version (3.5.2 to 3.6.2).
Might anyone have any suggestions as to why this is the case? And/or how to
alter the memory management so that memory is not exhausted so easily?
Many thanks, Chris
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
