How big is the entire text file? What is the length of an average
line? Have you tried to use 'scan' to read in the data? How much
memory do you have? Are you paging? Here was a quick test I did with
a file with about 5M lines (12|this is some text|12345|more test):
> system.time(x <- scan('/tempxx.txt', what=list(0,'',0,''), sep="|"))
Read 4460544 records
user system elapsed
50.70 1.04 54.16
>
> str(x)
List of 4
$ : num [1:4460544] 12 12 12 12 12 12 12 12 12 12 ...
$ : chr [1:4460544] "this is some text" "this is some text" "this is
some text" "this is some text" ...
$ : num [1:4460544] 12345 12345 12345 12345 12345 ...
$ : chr [1:4460544] "more test" "more test" "more test" "more test" ...
> object.size(x)
107053288 bytes
So some more details might help to evaluate what your problem is.
Took less than a minute to read it in.
On Wed, Jul 8, 2009 at 4:00 PM, Amy Wesolowski<amywesolow...@gmail.com> wrote:
> Hi,
>
> I am currently working on reading large files into R. My files are text
> documents with four columns and around 10 million lines.
> Each line is set up as:
> string|integer|string|integer
>
> I have been trying to use read.table to read in the file, but I think I am
> reading too much into memory and the application quits.
>
> I want to be able to analyze the entire text document at once.
> I have thought about reading in the file, line by line, but I still want to
> store all the information together. I have also thought about writing each
> line of the file to a matrix, but I cannot seem to figure it out.
>
> Any help would be great.
> Thanks,
> Amy
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
--
Jim Holtman
Cincinnati, OH
+1 513 646 9390
What is the problem that you are trying to solve?
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
