Over the past few years, it is only on the Solr user mailing list that I
have read requests for greater than 2B documents in a single index - nobody
has requested this on the Lucene user list. The point is that this is
primarily a Lucene issue that just happens to get passed through to Solr
users,
bq: Are there any such structures?
Well, I thought there were, but I've got to admit I can't call any to mind
immediately.
bq: 2b is just the hard limit
Yeah, I'm always a little nervous as to when Moore's Law will make
everything I know about current systems' performance obsolete.
At any rate,
Erick Erickson [erickerick...@gmail.com] wrote:
> I guess my $0.02 is that you'd have to have strong evidence that extending
> Lucene to 64 bit is even useful. Or more generally, useful enough to pay the
> penalty. All the structures that allocate maxDoc id arrays would suddenly
> require twice th
I guess my $0.02 is that you'd have to have strong evidence that extending
Lucene to 64 bit is even useful. Or more generally, useful enough to pay the
penalty. All the structures that allocate maxDoc id arrays would suddenly
require twice the memory for instance, plus all the coding effort that co
On 2/4/2015 3:31 PM, Arumugam, Suresh wrote:
> We are trying to do a POC for searching our log files with a single node
> Solr(396 GB RAM with 14 TB Space).
> Since the server is powerful, added 2 Billion records successfully & search
> is working fine without much issues.
>
> Due to the restrict
Given that it is log entries, you might find it works to use a
collection per day, and then use collection aliasing to query over them
all. This way, you can have a different aliases that specify certain
ranges (e.g. week is an alias for the last 7 or 8 day's collections).
Upayavira
On Thu, Feb 5
On Wed, 2015-02-04 at 23:31 +0100, Arumugam, Suresh wrote:
> We are trying to do a POC for searching our log files with a single
> node Solr(396 GB RAM with 14 TB Space).
We're running 7 billion larger-than-typical-log-entries documents from a
machine of similar size and it serves our needs well:
: We are trying to do a POC for searching our log files with a single node
Solr(396 GB RAM with 14 TB Space).
: Since the server is powerful, added 2 Billion records successfully & search
is working fine without much issues.
how much CPU?
Assuming it's comparable to the amount of RAM you've go
ject: Re: Exception while loading 2 Billion + Documents in Solr 4.8.0
On 2/4/2015 2:54 PM, Arumugam, Suresh wrote:
>
> Hi All,
>
>
>
> We are trying to load 14+ Billion documents into Solr. But we are
> failing to load them into Solr.
>
>
>
> So
You can only put 2 billion documents in one core. This error message is the
clue:
Too many documents, composite IndexReaders cannot exceed 2147483647
You will need to shard the collection. You might have multiple shards per node,
but you will probably need 50-100 shards and lots of servers.
wu
On 2/4/2015 2:54 PM, Arumugam, Suresh wrote:
>
> Hi All,
>
>
>
> We are trying to load 14+ Billion documents into Solr. But we are
> failing to load them into Solr.
>
>
>
> Solr version: *4.8.0*
>
> Analyzer used: *ClassicTokenizer for index as well as query.*
>
>
>
> Can someone help me in g
What's your cluster size? The 2 billion limit is per-node.
My personal recommendation is that you don't load more than 100 million
documents per node. You need to do a proof of concept test to verify
whether your particular data would support a higher number or not.
Ultimately, it will not be a ma
.
Regards,
Suresh.A
From: Arumugam, Suresh [mailto:suresh.arumu...@emc.com]
Sent: Wednesday, February 04, 2015 1:54 PM
To: solr-user@lucene.apache.org
Cc: Habeeb, Anwar
Subject: Exception while loading 2 Billion + Documents in Solr 4.8.0
Hi All,
We are trying to load 14+ Billion documents into Solr. But
Hi All,
We are trying to load 14+ Billion documents into Solr. But we are failing to
load them into Solr.
Solr version: 4.8.0
Analyzer used: ClassicTokenizer for index as well as query.
Can someone help me in getting into the core of this issue?
For 14+ Billion document load, we are loading 2B
14 matches
Mail list logo
