Erick Erickson [erickerick...@gmail.com] wrote: > I guess that my main issue is that from everything I've seen so far, > this project is doomed. You simply cannot put 7B documents in a single > shard, period. Lucene has a 2B hard limit.
I would have agreed with you fully an hour ago and actually planned to ask Wilbur to check if he had corrupted his indexes. However, his latest post suggests that the scenario is more about having a larger amount of more resonably sized shards in play than building gigantic shards. > For instance, Wilburn is talking about only using 6G of memory. Even > at 2B docs/shard, I'd be surprised to see it function at all. Don't > try sorting on a timestamp for instance. I haven't understood Wilburns setup completely, as it seems to me that he will quickly run out of memory for starting new shards. But if we are looking at shards of 30GB and 160M documents, 6GB sounds a lot better. Regards, Toke Eskildsen
