You could also try read.csv.sql in sqldf. See examples on sqldf home page:
http://code.google.com/p/sqldf/#Example_13._read.csv.sql_and_read.csv2.sql On Tue, Jan 19, 2010 at 9:25 AM, <nabble.30.miller_2...@spamgourmet.com> wrote: > I'm sure this has gotten some attention before, but I have two CSV > files generated from vmstat and free that are roughly 6-8 Mb (about > 80,000 lines) each. When I try to use read.csv(), R allocates all > available memory (about 4.9 Gb) when loading the files, which is over > 300 times the size of the raw data. Here are the scripts used to > generate the CSV files as well as the R code: > > Scripts (run for roughly a 24-hour period): > vmstat -ant 1 | awk '$0 !~ /(proc|free)/ {FS=" "; OFS=","; print > strftime("%F %T %Z"),$6,$7,$12,$13,$14,$15,$16,$17;}' >> > ~/vmstat_20100118_133845.o; > free -ms 1 | awk '$0 ~ /Mem\:/ {FS=" "; OFS=","; print > strftime("%F %T %Z"),$2,$3,$4,$5,$6,$7}' >> > ~/memfree_20100118_140845.o; > > R code: > infile.vms <- "~/vmstat_20100118_133845.o"; > infile.mem <- "~/memfree_20100118_140845.o"; > vms.colnames <- > c("time","r","b","swpd","free","inact","active","si","so","bi","bo","in","cs","us","sy","id","wa","st"); > vms.colclass <- c("character",rep("integer",length(vms.colnames)-1)); > mem.colnames <- > c("time","total","used","free","shared","buffers","cached"); > mem.colclass <- c("character",rep("integer",length(mem.colnames)-1)); > vmsdf <- > (read.csv(infile.vms,header=FALSE,colClasses=vms.colclass,col.names=vms.colnames)); > memdf <- > (read.csv(infile.mem,header=FALSE,colClasses=mem.colclass,col.names=mem.colnames)); > > I am running R v2.10.0 on a 64-bit machine with Fedora 10 (Linux > version 2.6.27.41-170.2.117.fc10.x86_64 ) with 6Gb of memory. There > are no other significant programs running and `rm()` followed by ` > gc()` successfully frees the memory (followed by swapins after other > programs seek to used previously cached information swapped to disk). > I've incorporated the memory-saving suggestions in the `read.csv()` > manual page, excluding the limit on the lines read (which shouldn't > really be necessary here since we're only talking about < 20 Mb of raw > data. Any suggestions, or is the read.csv() code known to have memory > leak/ overcommit issues? > > Thanks > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
