Another suggestion is to start with a subset of the data file to see how much memory is required for your processing. One of the misconceptions is that "memory is free". People think that with virtual memory and other such tools, that there is no restriction on what you can do. Instead of starting off, and getting "mad", when trying to read in all your data, try a small subset and then look at the memory usage of the R process. I would assume that you are running on a 32-bit Windows system, which if you are lucky, can address 3GB for a user process. My rule of thumb is that the largest object I can work with is 30% of the real memory I have available, so for my Windows system which lets me address almost 3GB, the biggest object I would attempt to work with is 1GB,
Working with a subset, you would understand how much memory an XXXMB file might require. This would then give you an idea of what the maximum size file you might be able to process. Every system has limits. If you have lots of money, then invest in a 64-bit system with 100GB of real memory and you probably won't hit its limits for a while. Otherwise, look at taking incremental steps and possibly determining if you can partition the data. You might consider a relational database to sotre the data so that it is easier to select a subset of data to process. 2012/3/17 Uwe Ligges <lig...@statistik.tu-dortmund.de>: > > > On 17.03.2012 19:27, David Winsemius wrote: >> >> >> On Mar 17, 2012, at 10:33 AM, Amen wrote: >> >>> I faced this problem when typing: >>> >>> temperature <- get.var.ncdf( ex.nc, 'Temperature' ) >>> >>> *unable to allocate a vector of size 2.8 GB* >> >> >> Read the R-Win-FAQ > >> >>> >>> >>> By the way my computer memory is 4G and the original size of the file is >>> 1.4G,netcdf file > > > ... and reading / storing the data in memory may require much more than > 4GB... > > Uwe Ligges > > >>> I don't know what is the problem.Any suggestion please >>> I tried also >>> memory limit(4000) >>> 4000 >>> but didnt solve the problem.any help >> >> > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
