Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-15 Thread Gabor Grothendieck
On Fri, Oct 15, 2010 at 6:14 AM, Chris Howden wrote: > Thanks for the advice Gabor, > > I was indeed not starting and finishing with sqldf(). Which was why it was > not working for me. Please forgive a blatantly obvious mistake. > > > I have tried what U suggested and unfortunately R is still havi

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-15 Thread Chris Howden
Solutions Tricky Solutions 4 Tricky Problems Evidence Based Strategic Development, IP development, Data Analysis, Modelling, and Training (mobile) 0410 689 945 (fax / office) (+618) 8952 7878 ch...@trickysolutions.com.au -Original Message- From: Gabor Grothendieck [mailto:ggrothendi...@gmail

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-14 Thread Gabor Grothendieck
On Thu, Oct 14, 2010 at 10:56 PM, Chris Howden wrote: > Thanks for the suggestion and code Gabor, > > I've tried creating 2 indices: > > 1) just for the variable I intend to merge on > 2) on the entire data set I am merging (which I think is the one I should > be using??) > > However neither seeme

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-14 Thread Chris Howden
lutions.com.au -Original Message- From: Gabor Grothendieck [mailto:ggrothendi...@gmail.com] Sent: Thursday, 14 October 2010 9:02 AM To: Chris Howden Cc: r-help@r-project.org Subject: Re: [R] merging and working with BIG data sets. Is sqldf the best way?? On Tue, Oct 12, 2010 at 2:39 AM, Chris Howden

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-13 Thread Gabor Grothendieck
On Tue, Oct 12, 2010 at 2:39 AM, Chris Howden wrote: > I’m working with some very big datasets (each dataset has 11 million rows > and 2 columns). My first step is to merge all my individual data sets > together (I have about 20) > > I’m using the following command from sqldf > >               dat

Re: [R] merging and working with big data sets

2010-10-12 Thread Jay Emerson
I can't speak for ff and filehash, but bigmemory's data structure doesn't allow "clever" merges (for actually good reasons). However, it is still probably less painful (and faster) than other options, though we don't implement it: we leave it to the user because details may vary depending on the e

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-11 Thread Jeff Newmiller
Chris Howden wrote: Hi everyone, I’m working with some very big datasets (each dataset has 11 million rows and 2 columns). My first step is to merge all my individual data sets together (I have about 20) I’m using the following command from sqldf data1 <- sqldf("select A.*, B.* f

[R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-11 Thread Chris Howden
Hi everyone, I’m working with some very big datasets (each dataset has 11 million rows and 2 columns). My first step is to merge all my individual data sets together (I have about 20) I’m using the following command from sqldf data1 <- sqldf("select A.*, B.* from A inner join