Re: Large RDBMS dataset

2011-12-29 Thread Alexey Serba
> The problem is that for each record in "fd", Solr makes three distinct SELECT > on the other three tables. Of course, this is absolutely inefficient. You can also try to use GROUP_CONCAT (it's MySQL function, but maybe there's something similar in MS SQL) to select all the nested 1-N entities i

Re: Large RDBMS dataset

2011-12-15 Thread Mikhail Khludnev
CachedSqlEntityProcessor joins you tables fine. But be aware that it works in the single thread only. On Thu, Dec 15, 2011 at 12:14 PM, Finotti Simone wrote: > CachedSqlEntityProcessor -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev

Re: Large RDBMS dataset

2011-12-15 Thread Finotti Simone
ch.com] Inviato: mercoledì 14 dicembre 2011 16.39 Fine: solr-user@lucene.apache.org Oggetto: Re: Large RDBMS dataset On Wed, Dec 14, 2011 at 3:48 PM, Finotti Simone wrote: > Hello, > I have a very large dataset (> 1 Mrecords) on the RDBMS which I want my Solr > application to pull data from

Re: Large RDBMS dataset

2011-12-14 Thread Erick Erickson
You can also consider using SolrJ to do this. I posted a small example a couple of days ago. Best Erick On Wed, Dec 14, 2011 at 10:39 AM, Gora Mohanty wrote: > On Wed, Dec 14, 2011 at 3:48 PM, Finotti Simone wrote: >> Hello, >> I have a very large dataset (> 1 Mrecords) on the RDBMS which I wan

Re: Large RDBMS dataset

2011-12-14 Thread Gora Mohanty
On Wed, Dec 14, 2011 at 3:48 PM, Finotti Simone wrote: > Hello, > I have a very large dataset (> 1 Mrecords) on the RDBMS which I want my Solr > application to pull data from. [...] > It works, but it takes 1'38" to parse 100 records: it means 1 rec/s! That > means that digesting the whole data

Re: Large RDBMS dataset

2011-12-14 Thread Martin Koch
Instead of handling it from within solr, I'd suggest writing an external application (e.g. in python using pysolr) that wraps the (fast) SQL query you like. Then retrieve a batch of documents, and write them to solr. For extra speed, don't commit until you're done. /Martin On Wed, Dec 14, 2011 at

Large RDBMS dataset

2011-12-14 Thread Finotti Simone
Hello, I have a very large dataset (> 1 Mrecords) on the RDBMS which I want my Solr application to pull data from. Problem is that the document fields which I have to index aren't in the same table, but I have to join records with two other tables. Well, in fact they are views, but I don't thin