Not sure if it applies to your file or not but if it does then the sqldf package facilitates reading a large file into an SQLite database. Its a front end to RSQLite which is a front end to SQLite and it reads the data straight into the database without going through R so R does not limit it in any way -- its only actuated from R. The code to do this is basically just two lines of code. You don;t have to install database software (its included with RSQLite package) and you don't have to set up a database at all -- it does that for you automatically.
See example 6e on the home page which creates a database transparently, reads in the data and extracts random rows from the database into R: http://sqldf.googlecode.com On Fri, Sep 26, 2008 at 3:55 PM, zerfetzen <[EMAIL PROTECTED]> wrote: > > Hi, > I'm sure that a large fixed width file, such as 300 million rows and 1,000 > columns, is too large for R to handle on a PC, but are there ways to deal > with it? > > For example, is there a way to combine some sampling method with read.fwf so > that you can read in a sample of 100,000 records, for example? > > Something like this may make analysis possible. > > Once analyzed, is there a way to, say, read in only x rows at a time, save > and score each subset separately, and finally append them back together? > > I haven't seen any information on this, if it is possible. Thank you for > reading, and sorry if the information was easily available and I simply > didn't find it. > -- > View this message in context: > http://www.nabble.com/Dealing-With-Extremely-Large-Files-tp19695311p19695311.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
