Streaming Expression can be sent to any SolrCloud node in any collection. You can setup collections that have no data and just execute the expressions. The expressions reference other collections that hold data.
Collections that only execute expressions we can call "Worker Collections". Collections that hold data we can call "Search Collections". This allows you to have servers with a different specification for workers and search nodes and offload the the workloads onto different collections. The parallel function sends a streaming expression to N worker nodes. Each worker node executes the expression and processes a partition of the data. Both the search function and the topic function can be parallelized like this. So operations like joins that wrap a search can be parallelized. If you just use the Java API directly, the code executes in the VM where the code is run. You could use the ParallelStream to send the code to a SolrCloud worker to execute the code as well. In this scenario the code is serialized to a Streaming Expression and sent across the wire to the Solr node. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jul 22, 2016 at 11:23 AM, tedsolr <tsm...@sciquest.com> wrote: > The streaming API looks like it's meant to be run from the client app > server > - very similar to a standard Solr search. When I run a basic streaming > operation the memory consumption occurs on the app server jvm, not the solr > server jvm. The opposite of what I was expecting. > > (pseudo code) > Stream A = new CloudSolrStream(); > Stream B = new CloudSolrStream(); > Stream C = new HashJoinStream(A, B); > Stream D = new SortStream(C); > Stream E = new ReducerStream(D); > E.open(); > > The SortStream is processed in memory when open() is called. Can the > processing be pushed off to the Solr cluster? Is that what the Parallel > stream will do - using worker collections? > > confused, > Ted > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Should-streaming-place-load-on-the-app-server-tp4288466.html > Sent from the Solr - User mailing list archive at Nabble.com. >
