Hi Joel, It works when I started off with just one expression.
Could it be that the data size is too big for export after the join, which causes the error? Regards, Edwin On 4 May 2017 at 02:53, Joel Bernstein <joels...@gmail.com> wrote: > I was just testing with the query below and it worked for me. Some of the > error messages I was getting with the syntax was not what I was expecting > though, so I'll look into the error handling. But the joins do work when > the syntax correct. The query below is joining to the same collection three > times, but the mechanics are exactly the same joining three different > tables. In this example each join narrows down the result set. > > hashJoin(parallel(collection2, > workers=3, > sort="id asc", > innerJoin(search(collection2, q="*:*", fl="id", > sort="id asc", qt="/export", partitionKeys="id"), > search(collection2, > q="year_i:42", fl="id, year_i", sort="id asc", qt="/export", > partitionKeys="id"), > on="id")), > hashed=search(collection2, q="day_i:7", fl="id, day_i", > sort="id asc", qt="/export"), > on="id") > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Wed, May 3, 2017 at 1:29 PM, Joel Bernstein <joels...@gmail.com> wrote: > > > Start off with just this expression: > > > > search(collection2, > > q=*:*, > > fl="a_s,b_s,c_s,d_s,e_s", > > sort="a_s asc", > > qt="/export") > > > > And then check the logs for exceptions. > > > > Joel Bernstein > > http://joelsolr.blogspot.com/ > > > > On Wed, May 3, 2017 at 12:35 PM, Zheng Lin Edwin Yeo < > edwinye...@gmail.com > > > wrote: > > > >> Hi Joel, > >> > >> I am getting this error after I change add qt=/export and removed the > rows > >> param. Do you know what could be the reason? > >> > >> { > >> "error":{ > >> "metadata":[ > >> "error-class","org.apache.solr.common.SolrException", > >> "root-error-class","org.apache.http.MalformedChunkCodingExce > >> ption"], > >> "msg":"org.apache.http.MalformedChunkCodingException: CRLF expected > >> at > >> end of chunk", > >> "trace":"org.apache.solr.common.SolrException: > >> org.apache.http.MalformedChunkCodingException: CRLF expected at end of > >> chunk\r\n\tat > >> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr > >> iteMap$0(TupleStream.java:79)\r\n\tat > >> org.apache.solr.response.JSONWriter.writeIterator(JSONRespon > >> seWriter.java:523)\r\n\tat > >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes > >> ponseWriter.java:175)\r\n\tat > >> org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter > >> .java:559)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.TupleStream.writeMap( > >> TupleStream.java:64)\r\n\tat > >> org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri > >> ter.java:547)\r\n\tat > >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes > >> ponseWriter.java:193)\r\n\tat > >> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD > >> ups(JSONResponseWriter.java:209)\r\n\tat > >> org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo > >> nseWriter.java:325)\r\n\tat > >> org.apache.solr.response.JSONWriter.writeResponse(JSONRespon > >> seWriter.java:120)\r\n\tat > >> org.apache.solr.response.JSONResponseWriter.write(JSONRespon > >> seWriter.java:71)\r\n\tat > >> org.apache.solr.response.QueryResponseWriterUtil.writeQueryR > >> esponse(QueryResponseWriterUtil.java:65)\r\n\tat > >> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC > >> all.java:732)\r\n\tat > >> org.apache.solr.servlet.HttpSolrCall.call( > HttpSolrCall.java:473)\r\n\tat > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp > >> atchFilter.java:345)\r\n\tat > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp > >> atchFilter.java:296)\r\n\tat > >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte > >> r(ServletHandler.java:1691)\r\n\tat > >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan > >> dler.java:582)\r\n\tat > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped > >> Handler.java:143)\r\n\tat > >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa > >> ndler.java:548)\r\n\tat > >> org.eclipse.jetty.server.session.SessionHandler.doHandle( > >> SessionHandler.java:226)\r\n\tat > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle( > >> ContextHandler.java:1180)\r\n\tat > >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand > >> ler.java:512)\r\n\tat > >> org.eclipse.jetty.server.session.SessionHandler.doScope( > >> SessionHandler.java:185)\r\n\tat > >> org.eclipse.jetty.server.handler.ContextHandler.doScope( > >> ContextHandler.java:1112)\r\n\tat > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped > >> Handler.java:141)\r\n\tat > >> org.eclipse.jetty.server.handler.ContextHandlerCollection.ha > >> ndle(ContextHandlerCollection.java:213)\r\n\tat > >> org.eclipse.jetty.server.handler.HandlerCollection.handle( > >> HandlerCollection.java:119)\r\n\tat > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl > >> erWrapper.java:134)\r\n\tat > >> org.eclipse.jetty.server.Server.handle(Server.java:534)\r\n\tat > >> org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:320)\r\n\tat > >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne > >> ction.java:251)\r\n\tat > >> org.eclipse.jetty.io.AbstractConnection$ReadCallback. > >> succeeded(AbstractConnection.java:273)\r\n\tat > >> org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:95)\r\n\tat > >> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann > >> elEndPoint.java:93)\r\n\tat > >> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume > >> .executeProduceConsume(ExecuteProduceConsume.java:303)\r\n\tat > >> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume > >> .produceConsume(ExecuteProduceConsume.java:148)\r\n\tat > >> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume > >> .run(ExecuteProduceConsume.java:136)\r\n\tat > >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued > >> ThreadPool.java:671)\r\n\tat > >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT > >> hreadPool.java:589)\r\n\tat > >> java.lang.Thread.run(Thread.java:745)\r\nCaused by: > >> org.apache.http.MalformedChunkCodingException: CRLF expected at end of > >> chunk\r\n\tat > >> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(Chun > >> kedInputStream.java:255)\r\n\tat > >> org.apache.http.impl.io.ChunkedInputStream.nextChunk(Chunked > >> InputStream.java:227)\r\n\tat > >> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInput > >> Stream.java:186)\r\n\tat > >> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInput > >> Stream.java:215)\r\n\tat > >> org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInpu > >> tStream.java:316)\r\n\tat > >> org.apache.http.conn.BasicManagedEntity.streamClosed(BasicMa > >> nagedEntity.java:164)\r\n\tat > >> org.apache.http.conn.EofSensorInputStream.checkClose(EofSens > >> orInputStream.java:228)\r\n\tat > >> org.apache.http.conn.EofSensorInputStream.close(EofSensorInp > >> utStream.java:174)\r\n\tat > >> sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:378)\r\n\tat > >> sun.nio.cs.StreamDecoder.close(StreamDecoder.java:193)\r\n\tat > >> java.io.InputStreamReader.close(InputStreamReader.java:199)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.JSONTupleStream.close > >> (JSONTupleStream.java:92)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.SolrStream.close(Solr > >> Stream.java:193)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.CloudSolrStream.close > >> (CloudSolrStream.java:464)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.HashJoinStream.close( > >> HashJoinStream.java:231)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.ExceptionStream.close > >> (ExceptionStream.java:93)\r\n\tat > >> org.apache.solr.handler.StreamHandler$TimerStream.close( > >> StreamHandler.java:452)\r\n\tat > >> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr > >> iteMap$0(TupleStream.java:71)\r\n\t... > >> 40 more\r\n", > >> "code":500}} > >> > >> > >> Regards, > >> Edwin > >> > >> > >> On 4 May 2017 at 00:00, Joel Bernstein <joels...@gmail.com> wrote: > >> > >> > I've reformatted the expression below and made a few changes. You have > >> put > >> > things together properly. But these are MapReduce joins that require > >> > exporting the entire result sets. So you will need to add qt=/export > to > >> all > >> > the searches and remove the rows param. In Solr 6.6. there is a new > >> > "shuffle" expression that does this automatically. > >> > > >> > To test things you'll want to break down each expression and make sure > >> it's > >> > behaving as expected. > >> > > >> > For example first run each search. Then run the innerJoin, not in > >> parallel > >> > mode. Then run it in parallel mode. Then try the whole thing. > >> > > >> > hashJoin(parallel(collection2, > >> > innerJoin(search(collection2, > >> > q=*:*, > >> > > >> > fl="a_s,b_s,c_s,d_s,e_s", > >> > sort="a_s asc", > >> > > >> partitionKeys="a_s", > >> > qt="/export"), > >> > search(collection1, > >> > q=*:*, > >> > > >> > fl="a_s,f_s,g_s,h_s,i_s,j_s", > >> > sort="a_s asc", > >> > > >> partitionKeys="a_s", > >> > qt="/export"), > >> > on="a_s"), > >> > workers="2", > >> > sort="a_s asc"), > >> > hashed=search(collection3, > >> > q=*:*, > >> > fl="a_s,k_s,l_s", > >> > sort="a_s asc", > >> > qt="/export"), > >> > on="a_s") > >> > > >> > Joel Bernstein > >> > http://joelsolr.blogspot.com/ > >> > > >> > On Wed, May 3, 2017 at 11:26 AM, Zheng Lin Edwin Yeo < > >> edwinye...@gmail.com > >> > > > >> > wrote: > >> > > >> > > Hi Joel, > >> > > > >> > > Thanks for the clarification. > >> > > > >> > > Would like to check, is this the correct way to do the join? > >> Currently, I > >> > > could not get any results after putting in the hashJoin for the 3rd, > >> > > smallerStream collection (collection3). > >> > > > >> > > http://localhost:8983/solr/collection1/stream?expr= > >> > > hashJoin(parallel(collection2 > >> > > , > >> > > innerJoin( > >> > > search(collection2, > >> > > q=*:*, > >> > > fl="a_s,b_s,c_s,d_s,e_s", > >> > > sort="a_s asc", > >> > > partitionKeys="a_s", > >> > > rows=200), > >> > > search(collection1, > >> > > q=*:*, > >> > > fl="a_s,f_s,g_s,h_s,i_s,j_s", > >> > > sort="a_s asc", > >> > > partitionKeys="a_s", > >> > > rows=200), > >> > > on="a_s"), > >> > > workers="2", > >> > > sort="a_s asc"), > >> > > hashed=search(collection3, > >> > > q=*:*, > >> > > fl="a_s,k_s,l_s", > >> > > sort="a_s asc", > >> > > rows=200), > >> > > on="a_s") > >> > > &indent=true > >> > > > >> > > > >> > > Regards, > >> > > Edwin > >> > > > >> > > > >> > > On 3 May 2017 at 20:59, Joel Bernstein <joels...@gmail.com> wrote: > >> > > > >> > > > Sorry, it's just called hashJoin > >> > > > > >> > > > Joel Bernstein > >> > > > http://joelsolr.blogspot.com/ > >> > > > > >> > > > On Wed, May 3, 2017 at 2:45 AM, Zheng Lin Edwin Yeo < > >> > > edwinye...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Hi Joel, > >> > > > > > >> > > > > I am getting this error when I used the innerHashJoin. > >> > > > > > >> > > > > "EXCEPTION":"Invalid stream expression innerHashJoin(parallel( > >> > > innerJoin > >> > > > > > >> > > > > I also can't find the documentation on innerHashJoin for the > >> > Streaming > >> > > > > Expressions. > >> > > > > > >> > > > > Are you referring to hashJoin? > >> > > > > > >> > > > > Regards, > >> > > > > Edwin > >> > > > > > >> > > > > > >> > > > > On 3 May 2017 at 13:20, Zheng Lin Edwin Yeo < > edwinye...@gmail.com > >> > > >> > > > wrote: > >> > > > > > >> > > > > > Hi Joel, > >> > > > > > > >> > > > > > Thanks for the info. > >> > > > > > > >> > > > > > Regards, > >> > > > > > Edwin > >> > > > > > > >> > > > > > > >> > > > > > On 3 May 2017 at 02:04, Joel Bernstein <joels...@gmail.com> > >> wrote: > >> > > > > > > >> > > > > >> Also take a look at the documentation for the "fetch" > streaming > >> > > > > >> expression. > >> > > > > >> > >> > > > > >> Joel Bernstein > >> > > > > >> http://joelsolr.blogspot.com/ > >> > > > > >> > >> > > > > >> On Tue, May 2, 2017 at 2:03 PM, Joel Bernstein < > >> > joels...@gmail.com> > >> > > > > >> wrote: > >> > > > > >> > >> > > > > >> > Yes you join more then one collection with Streaming > >> > Expressions. > >> > > > Here > >> > > > > >> are > >> > > > > >> > a few things to keep in mind. > >> > > > > >> > > >> > > > > >> > * You'll likely want to use the parallel function around > the > >> > > largest > >> > > > > >> join. > >> > > > > >> > You'll need to use the join keys as the partitionKeys. > >> > > > > >> > * innerJoin: requires that the streams be sorted on the > join > >> > keys. > >> > > > > >> > * innerHashJoin: has no sorting requirement. > >> > > > > >> > > >> > > > > >> > So a strategy for a three collection join might look like > >> this: > >> > > > > >> > > >> > > > > >> > innerHashJoin(parallel(innerJoin(bigStream, bigStream)), > >> > > > > smallerStream) > >> > > > > >> > > >> > > > > >> > The largest join can be done in parallel using an > innerJoin. > >> You > >> > > can > >> > > > > >> then > >> > > > > >> > wrap the stream coming out of the parallel function in an > >> > > > > innerHashJoin > >> > > > > >> to > >> > > > > >> > join it to another stream. > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > Joel Bernstein > >> > > > > >> > http://joelsolr.blogspot.com/ > >> > > > > >> > > >> > > > > >> > On Mon, May 1, 2017 at 9:42 PM, Zheng Lin Edwin Yeo < > >> > > > > >> edwinye...@gmail.com> > >> > > > > >> > wrote: > >> > > > > >> > > >> > > > > >> >> Hi, > >> > > > > >> >> > >> > > > > >> >> Is it possible to join more than 2 collections using one > of > >> the > >> > > > > >> streaming > >> > > > > >> >> expressions (Eg: innerJoin)? If not, is there other ways > we > >> can > >> > > do > >> > > > > it? > >> > > > > >> >> > >> > > > > >> >> Currently, I may need to join 3 or 4 collections together, > >> and > >> > to > >> > > > > >> output > >> > > > > >> >> selected fields from all these collections together. > >> > > > > >> >> > >> > > > > >> >> I'm using Solr 6.4.2. > >> > > > > >> >> > >> > > > > >> >> Regards, > >> > > > > >> >> Edwin > >> > > > > >> >> > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
