Hi Aki,
IMO this is underuse of Solr (not to mention SolrCloud). I would
recommend doing in memory document parsin (if you need something from
Lucene/Solr analysis classes, use it) and use some other cache like
solution to store term/total frequency pairs (you can try Redis).
That way you will have updatable, fast total frequency lookups.
Thanks,
Emir
On 26.10.2015 14:43, Aki Balogh wrote:
Hi Emir,
This is correct. This is the only way we use the index.
Thanks,
Aki
On Mon, Oct 26, 2015 at 9:31 AM, Emir Arnautovic <
emir.arnauto...@sematext.com> wrote:
If I got it right, you are using term query, use function to get TF as
score, iterate all documents in results and sum up total number of
occurrences of specific term in index? Is this only way you use index or
this is side functionality?
Thanks,
Emir
On 24.10.2015 22:28, Aki Balogh wrote:
Certainly, yes. I'm just doing a word count, ie how often does a specific
term come up in the corpus?
On Oct 24, 2015 4:20 PM, "Upayavira" <u...@odoko.co.uk> wrote:
yes, but what do you want to do with the TF? What problem are you
solving with it? If you are able to share that...
On Sat, Oct 24, 2015, at 09:05 PM, Aki Balogh wrote:
Yes, sorry, I am not being clear.
We are not even doing scoring, just getting the raw TF values. We're
doing
this in solr because it can scale well.
But with large corpora, retrieving the word counts takes some time, in
part
because solr is splitting up word count by document and generating a
large
request. We then get the request and just sum it all up. I'm wondering
if
there's a more direct way.
On Oct 24, 2015 4:00 PM, "Upayavira" <u...@odoko.co.uk> wrote:
Can you explain more what you are using TF for? Because it sounds
rather
like scoring. You could disable field norms and IDF and scoring would
be
mostly TF, no?
Upayavira
On Sat, Oct 24, 2015, at 07:28 PM, Aki Balogh wrote:
Thanks, let me think about that.
We're using termfreq to get the TF score, but we don't know which
term
we'll need the TF for. So we'd have to do a corpuswide summing of
termfreq
for each potential term across all documents in the corpus. It seems
like
it'd require some development work to compute that, and our code
would be
fragile.
Let me think about that more.
It might make sense to just move to solrcloud, it's the right
architectural
decision anyway.
On Sat, Oct 24, 2015 at 1:54 PM, Upayavira <u...@odoko.co.uk> wrote:
If you just want word length, then do work during indexing - index
a
field for the word length. Then, I believe you can do faceting -
e.g.
with the json faceting API I believe you can do a sum()
calculation on
a
field rather than the more traditional count.
Thinking aloud, there might be an easier way - index a field that
is
the
same for all documents, and facet on it. Instead of counting the
number
of documents, calculate the sum() of your word count field.
I *think* that should work.
Upayavira
On Sat, Oct 24, 2015, at 04:24 PM, Aki Balogh wrote:
Hi Jack,
I'm just using solr to get word count across a large number of
documents.
It's somewhat non-standard, because we're ignoring relevance,
but it
seems
to work well for this use case otherwise.
My understanding then is:
1) since termfreq is pre-processed and fetched, there's no good
way
to
speed it up (except by caching earlier calculations)
2) there's no way to have solr sum up all of the termfreqs
across all
documents in a search and just return one number for total
termfreqs
Are these correct?
Thanks,
Aki
On Sat, Oct 24, 2015 at 11:20 AM, Jack Krupansky
<jack.krupan...@gmail.com>
wrote:
That's what a normal query does - Lucene takes all the terms
used
in
the
query and sums them up for each document in the response,
producing a
single number, the score, for each document. That's the way
Solr is
designed to be used. You still haven't elaborated why you are
trying
to use
Solr in a way other than it was intended.
-- Jack Krupansky
On Sat, Oct 24, 2015 at 11:13 AM, Aki Balogh <
a...@marketmuse.com>
wrote:
Gotcha - that's disheartening.
One idea: when I run termfreq, I get all of the termfreqs for
each
document
one-by-one.
Is there a way to have solr sum it up before creating the
request,
so I
only receive one number in the response?
On Sat, Oct 24, 2015 at 11:05 AM, Upayavira <u...@odoko.co.uk>
wrote:
If you mean using the term frequency function query, then
I'm
not
sure
there's a huge amount you can do to improve performance.
The term frequency is a number that is used often, so it is
stored
in
the index pre-calculated. Perhaps, if your data is not
changing,
optimising your index would reduce it to one segment, and
thus
might
ever so slightly speed the aggregation of term frequencies,
but I
doubt
it'd make enough difference to make it worth doing.
Upayavira
On Sat, Oct 24, 2015, at 03:37 PM, Aki Balogh wrote:
Thanks, Jack. I did some more research and found similar
results.
In our application, we are making multiple (think: 50)
concurrent
requests
to calculate term frequency on a set of documents in
"real-time". The
faster that results return, the better.
Most of these requests are unique, so cache only helps
slightly.
This analysis is happening on a single solr instance.
Other than moving to solr cloud and splitting out the
processing
onto
multiple servers, do you have any suggestions for what
might
speed up
termfreq at query time?
Thanks,
Aki
On Fri, Oct 23, 2015 at 7:21 PM, Jack Krupansky
<jack.krupan...@gmail.com>
wrote:
Term frequency applies only to the indexed terms of a
tokenized
field.
DocValues is really just a copy of the original source
text
and is
not
tokenized into terms.
Maybe you could explain how exactly you are using term
frequency in
function queries. More importantly, what is so "heavy"
about
your
usage?
Generally, moderate use of a feature is much more
advisable to
heavy
usage,
unless you don't care about performance.
-- Jack Krupansky
On Fri, Oct 23, 2015 at 8:19 AM, Aki Balogh <
a...@marketmuse.com>
wrote:
Hello,
In our solr application, we use a Function Query
(termfreq)
very
heavily.
Index time and disk space are not important, but
we're
looking to
improve
performance on termfreq at query time.
I've been reading up on docValues. Would this be a
way to
improve
performance?
I had read that Lucene uses Field Cache for Function
Queries, so
performance may not be affected.
And, any general suggestions for improving query
performance
on
Function
Queries?
Thanks,
Aki
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
<https://t.yesware.com/tl/506312808dab13214164f92fbcf5714d3ce38c6b/92f5492fd055692ff7f03b2888be3b50/7a8fd1f72b93af5d79583420b3483a7d?ytl=http%3A%2F%2Fsematext.com%2F>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
