EFFs work fine with millions of objects. Under the
covers, the float values are just read into memory
so looking them up in order to use in a function
is very fast.
Best
Erick
On Thu, Aug 16, 2012 at 9:52 PM, Tirthankar Chatterjee
wrote:
> Awesome, you rock !!!. Thanks to Eric too for coming
Awesome, you rock !!!. Thanks to Eric too for coming up with the ideas. But
honestly I am more interested to know this EFF concept. I will do my reading
and then throw up questions if I don't get it.
External File Field and function queries are something which i have not
evaluated yet, but I a
"*Solr's sorting reads all the unique values into memory whether or not they
satisfy the query...*"
Is it not a overhead for any query? Any reason why it is done on entire
result set and not on the query? Is there a way to avoid this?
--
View this message in context:
http://lucene.472066.n3.nab
Awesome thanks a lot, I am already on it with option 1. We need to track delete
to flip the previous one as the current.
Erick Erickson wrote:
No, sharding into multiple cores on the same machine still
is limited by the physical memory available. It's still lots
of stuf on a limited box.
But.
: 2> Use external file fields (EFF) for the same purpose, that
: won't require you to re-index the doc. The trick
: here is you use the value in the EFF as a multiplier
: for the score (that's what function queries do). So older
: versions of the doc have scores of 0 and just d
No, sharding into multiple cores on the same machine still
is limited by the physical memory available. It's still lots
of stuf on a limited box.
But try backing up and re-thinking the problem a bit.
Some possibilities off the top of my head:
1> have a new field "current". when you update a d
Hi Erick,
You are so right on the memory calculations. I am happy that I know now that I
was doing something wrong. Yes I am getting confused with SQL.
I will back up and let you know the use case. I am tracking file versions. And
I want to give an option to browse your system for the latest fil
You'r putting a lot of data on a single box, then
asking to group on what I presume is a string
field. That's just going to eat up a _bunch_ of
memory.
let's say your average file name is 16 bytes long. Each
unique value will take up 58 + 32 bytes (58 bytes
of overhead, I'm presuming Solr 3.X and
Editing the query...remove I don't know where it came from while I
did copy/paste
Tirthankar Chatterjee wrote:
Hi,
I have a beefy box with 24Gb RAM (12GB for Tomcat7 which houses SOLR3.6) 2
Processors Intel Xeon 64 bit Server, 30TB HDD. JDK 1.7.0_03 x64 bit
Data Index Dir Size: 400GB
Hi,
I have a beefy box with 24Gb RAM (12GB for Tomcat7 which houses SOLR3.6) 2
Processors Intel Xeon 64 bit Server, 30TB HDD. JDK 1.7.0_03 x64 bit
Data Index Dir Size: 400GB
Metadata of files is stored in it. I have around 15 schema fields.
Total number of items:150million approx.
I have a sce
10 matches
Mail list logo
