Hello Developers,
I just want to ask don't you think that response streaming can be useful
for things like OLAP, e.g. is you have sharded index presorted and
pre-joined by BJQ way you can calculate counts in many cube cells in
parallel?
Essential distributed test for response streaming just passed
Hello,
Small update - reading streamed response is done via callback. No
SolrDocumentList in memory.
https://github.com/m-khl/solr-patches/tree/streaming
here is the test
https://github.com/m-khl/solr-patches/blob/d028d4fabe0c20cb23f16098637e2961e9e2366e/solr/core/src/test/org/apache/solr/response
@All
Why nobody desires such a pretty cool feature?
Nicholas,
I have a tiny progress: I'm able to stream in javabin codec format while
searching, It implies sorting by _docid_
here is the diff
https://github.com/m-khl/solr-patches/commit/2f9ff068c379b3008bb983d0df69dff714ddde95
The current issue
Mikhail & Ludovic,
Thanks for both your replies, very helpful indeed!
Ludovic, I was actually looking into just that and did some tests with
SolrJ, it does work well but needs some changes on the Solr server if we
want to send out individual documents a various times. This could be done
with a w
Ludovic,
I looked through. First of all, it seems to me you don't amend regular
"servlet" solr server, but the only embedded one.
Anyway, the difference is that you stream DocList via callback, but it
means that you've instantiated it in memory and keep it there until it will
be completely consume
Hi,
I was looking for something similar.
I tried this patch :
https://issues.apache.org/jira/browse/SOLR-2112
it's working quite well (I've back-ported the code in Solr 3.5.0...).
Is it really different from what you are trying to achieve ?
Ludovic.
-
Jouve
France.
--
View this message
Hello Nicholas,
Looks like we are around the same point. Here is my branch
https://github.com/m-khl/solr-patches/tree/streaming there are only two
commits on top of it. And here is the test
https://github.com/m-khl/solr-patches/blob/streaming/solr/core/src/test/org/apache/solr/response/ResponseSt
Thanks for the reply Erik,
Yep, the project is working with distributed Solr applications (i.e.
shards) but not with the Solr supplied shard implementation, rather a
custom version (not very different to it to be honest).
I understand that Solr has scoring at it's heart which is something we are
Somehow you'd have to create a custom collector, probably queue off the docs
that made it to the collector and have some asynchronous thread consuming
those docs and sending them in bits...
But this is so antithetical to how Solr operates that I suspect my hand-waving
wouldn't really work out. The
