Hi, I've ended up processing the doclist in the response twice. Once in the write method and once in getContent. Its a bit inefficient but i'm only looking at top doc each time so probably ok. Is their a better way to do this ?
Cheers lee C On 22 April 2018 at 13:26, Lee Carroll <lee.a.carr...@googlemail.com> wrote: > Hi, > That works a treat. The raw response writer has a configurable base class > which executes when no content stream is present in the response so I just > delegate to that. I do still have an issue with writing content type on the > http response from a value in the top document however. Although I do have > a getContentType method which returns a value from a field in the top > document this is called before the response writer has performed its write > method. > > Whats the best way to set response headers using values from documents in > the search result? In particular Content-Type. > > Cheers Lee C > > On 20 April 2018 at 20:03, Chris Hostetter <hossman_luc...@fucit.org> > wrote: > >> >> Invariant really means "invariant" ... nothing can change them >> >> In the case of "wt" this may seem weird and unhelpful, but the code that >> handles defaults/appends/invariants is ignorant of what the params are. >> >> Since your writting custom code anyway, my suggestion would be that >> perhaps you could make your custom ResponseWriter delegate to the javabin >> responsewriter if/when you see that this is an "isShard=true" request? >> >> >> >> : Date: Thu, 19 Apr 2018 18:42:58 +0100 >> : From: Lee Carroll <lee.a.carr...@googlemail.com> >> : Reply-To: solr-user@lucene.apache.org >> : To: solr-user@lucene.apache.org >> : Subject: Re: custom response writer which extends RawResponseWriter >> fails when >> : shards > 1 >> : >> : Hi, >> : >> : I rewrote all of my tests to use SolrCloudTestCase rather than >> SolrTestCaseJ4 >> : and was able to replicate the responsewriter issue and debug with a >> sharded >> : collection. It turned out the issue was not with my response writer >> really >> : but rather my config. >> : >> : <requestHandler name="/content" class="solr.SearchHandler"> >> : .... >> : >> : <lst name="invariants"> >> : <str name="wt">content</str> >> : </lst> >> : >> : </requestHandler> >> : >> : In cloud mode having wt as an invariant breaks the collation of results >> : from shards. Now I'm sure this is a common mistake which I've repeated >> : (blush) but I do sort of want to actually implement my request handler >> in >> : this way. Is their a way to have a request handler support a single >> : response writer but still work in cloud mode ? >> : >> : Could this be considered a bug ? >> : >> : Lee C >> : >> : On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: >> : >> : > Injecting headers might require deeper customisation up to >> establishing own >> : > filter or so. >> : > Speaking regarding your own WT, there might be some issues because >> usually >> : > it's not a big deal to use one wt for responding user query like >> (wt=csv) >> : > and wt=javabin in internal communication between aggregator and >> slaves like >> : > it happens in wt=csv query. >> : > >> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < >> lee.a.carr...@googlemail.com >> : > > >> : > wrote: >> : > >> : > > Inventive. I need to control content-type of the response from the >> : > document >> : > > field value. I have the actual content field and the content-type >> field >> : > to >> : > > use configured in the response writer. I've just noticed that the >> xslt >> : > > transformer allows you to do this but not controlled by document >> values. >> : > I >> : > > may also need to set some headers based on content-type and perhaps >> : > content >> : > > size, accept ranges comes to mind. Although I might be getting >> ahead of >> : > > myself. >> : > > >> : > > >> : > > >> : > > On 18 April 2018 at 12:05, Mikhail Khludnev <m...@apache.org> >> wrote: >> : > > >> : > > > well .. >> : > > > what if >> : > > > http://localhost:8983/solr/images/select?fl=content&q=id: >> : > > 1&start=1&wt=csv& >> : > > > csv.separator=&csv.encapsulator&csv.null=null >> : > > > ? >> : > > > >> : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < >> : > > lee.a.carr...@googlemail.com >> : > > > > >> : > > > wrote: >> : > > > >> : > > > > sorry cut n paste error i'd get >> : > > > > >> : > > > > { >> : > > > > "responseHeader":{ >> : > > > > "zkConnected":true, >> : > > > > "status":0, >> : > > > > "QTime":0, >> : > > > > "params":{ >> : > > > > "q":"*:*", >> : > > > > "fl":"content", >> : > > > > "rows":"1"}}, >> : > > > > "response":{"numFound":1,"start":0,"docs":[ >> : > > > > { >> : > > > > "content":"my-content-value"}] >> : > > > > }} >> : > > > > >> : > > > > >> : > > > > but you get my point >> : > > > > >> : > > > > >> : > > > > >> : > > > > On 18 April 2018 at 11:13, Lee Carroll < >> lee.a.carr...@googlemail.com >> : > > >> : > > > > wrote: >> : > > > > >> : > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id: >> : > > 1&start=1 >> : > > > > > >> : > > > > > I'd get >> : > > > > > >> : > > > > > { >> : > > > > > "responseHeader":{ >> : > > > > > "zkConnected":true, >> : > > > > > "status":0, >> : > > > > > "QTime":1, >> : > > > > > "params":{ >> : > > > > > "q":"*:*", >> : > > > > > "_":"1524046333220"}}, >> : > > > > > "response":{"numFound":1,"start":0,"docs":[ >> : > > > > > { >> : > > > > > "id":"1", >> : > > > > > "content":"my-content-value", >> : > > > > > "*content-type*":"text/plain"}] >> : > > > > > }} >> : > > > > > >> : > > > > > when i want >> : > > > > > >> : > > > > > my-content-value >> : > > > > > >> : > > > > > >> : > > > > > >> : > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <m...@apache.org> >> : > wrote: >> : > > > > > >> : > > > > >> Lee, from this description I don see why it can't be >> addressed by >> : > > > > fl,rows >> : > > > > >> params. What makes it different form the typical Solr usage? >> : > > > > >> >> : > > > > >> >> : > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < >> : > > > > >> lee.a.carr...@googlemail.com> >> : > > > > >> wrote: >> : > > > > >> >> : > > > > >> > Sure, we want to return a single field's value for the top >> : > > matching >> : > > > > >> > document for a given query. Bare content rather than a full >> : > search >> : > > > > >> result >> : > > > > >> > listing. >> : > > > > >> > >> : > > > > >> > To be concrete: >> : > > > > >> > >> : > > > > >> > For a schema of fields id [unique key], >> : > > > content[stored],content-type[ >> : > > > > >> > stored] >> : > > > > >> > For a request: >> : > > > > >> > >> : > > > > >> > 1. Request URL: >> : > > > > >> > https://localhost/solr/content?q=id:1 >> : > > > > >> > 2. Request Method: >> : > > > > >> > GET >> : > > > > >> > >> : > > > > >> > We get a response >> : > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: >> : > [content-type >> : > > > > value] >> : > > > > >> > >> : > > > > >> > and the body to be the raw value of content >> : > > > > >> > >> : > > > > >> > In short clients consume directly the most relevant >> "content" >> : > > > returned >> : > > > > >> from >> : > > > > >> > solr queries they construct. >> : > > > > >> > >> : > > > > >> > Naively I've implemented a subclass of RawResponseWriter >> which >> : > > takes >> : > > > > the >> : > > > > >> > first docs values and adds them to the appended "content" >> : > stream. >> : > > > > >> Should I >> : > > > > >> > selectively add the content stream depending on if this is >> the >> : > > final >> : > > > > >> > aggregation of cloud results (and provide a base class >> writer to >> : > > act >> : > > > > if >> : > > > > >> > not), if so how do I know its the final aggregation. Or is >> : > adding >> : > > > the >> : > > > > >> > content stream within the response writer a bad idea. >> Should >> : > that >> : > > be >> : > > > > >> being >> : > > > > >> > added to the response somewhere else? >> : > > > > >> > >> : > > > > >> > Failing all of the above is asking about response writer >> an X / >> : > Y >> : > > > > >> problem. >> : > > > > >> > Is their a better way to achieve the above. I'd looked at >> : > > > transforming >> : > > > > >> > response xml but that seemed not to offer a complete bare >> slate. >> : > > > > >> > >> : > > > > >> > Cheers Lee C >> : > > > > >> > >> : > > > > >> > >> : > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev < >> m...@apache.org> >> : > > > wrote: >> : > > > > >> > >> : > > > > >> > > In distributed search response writer is used twice >> : > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed- >> : > > > requests.html >> : > > > > >> > > once slave node that's where response writer yields >> "json" >> : > > content >> : > > > > >> and it >> : > > > > >> > > upset aggregator node which is expect only javabin. >> : > > > > >> > > I hardly can comment on rrw, it's probably used for >> responding >> : > > > > >> separate >> : > > > > >> > > files in distrib=false mode. >> : > > > > >> > > You can start from describing why you need to create own >> : > > response >> : > > > > >> writer. >> : > > > > >> > > >> : > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < >> : > > > > >> > lee.a.carr...@googlemail.com >> : > > > > >> > > > >> : > > > > >> > > wrote: >> : > > > > >> > > >> : > > > > >> > > > Ok. My expectation was the response writer would not >> be used >> : > > > until >> : > > > > >> the >> : > > > > >> > > > final serialization of the result. If my response >> writer >> : > > breaks >> : > > > > the >> : > > > > >> > > > response writer contract, exactly the way >> rawResponseWriter >> : > > does >> : > > > > and >> : > > > > >> > just >> : > > > > >> > > > out puts a filed value how does that work? Does >> : > > > rawResponseWriter >> : > > > > >> > support >> : > > > > >> > > > cloud mode? >> : > > > > >> > > > >> : > > > > >> > > > >> : > > > > >> > > > >> : > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev < >> : > m...@apache.org> >> : > > > > >> wrote: >> : > > > > >> > > > >> : > > > > >> > > > > That's what should happen. >> : > > > > >> > > > > >> : > > > > >> > > > > Expected mime type application/octet-stream but got >> : > > > > >> application/json. >> : > > > > >> > > > > >> : > > > > >> > > > > Distributed search coordinator expect to merge slave >> : > > responses >> : > > > > in >> : > > > > >> > > javabin >> : > > > > >> > > > > format. But slave's wt indicated json. >> : > > > > >> > > > > As far as I know only javabin might be used to >> distributed >> : > > > > search >> : > > > > >> > > > > underneath. Coordinator itself might yield json. >> : > > > > >> > > > > >> : > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < >> : > > > > >> > > > lee.a.carr...@googlemail.com >> : > > > > >> > > > > > >> : > > > > >> > > > > wrote: >> : > > > > >> > > > > >> : > > > > >> > > > > > Sure >> : > > > > >> > > > > > >> : > > > > >> > > > > > with 1 shard 1 replica this request works fine >> : > > > > >> > > > > > >> : > > > > >> > > > > > 1. Request URL: >> : > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1 >> : > > > > >> > > > > > 2. Request Method: >> : > > > > >> > > > > > GET >> : > > > > >> > > > > > 3. Status Code: >> : > > > > >> > > > > > 200 OK >> : > > > > >> > > > > > >> : > > > > >> > > > > > logs are clean >> : > > > > >> > > > > > >> : > > > > >> > > > > > with 2 shards 2 replicas the same request fails >> and in >> : > the >> : > > > > logs >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 >> : > > > > r:core_node7 >> : > > > > >> > > > > > x:images_shard2_replica_n4] >> : > org.apache.solr.core.SolrCore; >> : > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr >> path=/image >> : > > > > >> > > > > > params={df=text&distrib=false& >> qt=/image&fl=id&fl=score& >> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url= >> : > > > > >> > > > > > http://10.224.30.207:8983/ >> : > solr/images_shard2_replica_n4/ >> : > > > > >> > > > > > |http://10.224.30.207:7574/ >> : > solr/images_shard2_replica_n6/ >> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW= >> : > > > 1523971232039&isShard=true&wt= >> : > > > > >> > javabin} >> : > > > > >> > > > > > hits=0 status=0 QTime=0 >> : > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 >> : > > > > r:core_node3 >> : > > > > >> > > > > > x:images_shard1_replica_n1] >> : > org.apache.solr.common.SolrExc >> : > > > > >> eption; >> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$ >> : > > > > >> > > RemoteSolrException: >> : > > > > >> > > > > > Error >> : > > > > >> > > > > > from server at http://10.224.30.207:8983/ >> : > > > > >> > > solr/images_shard2_replica_n4 >> : > > > > >> > > > : >> : > > > > >> > > > > > Expected mime type application/octet-stream but got >> : > > > > >> > application/json. >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient. >> : > > > > >> > > > > > executeMethod(HttpSolrClient.java:607) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.client.solrj. >> : > impl.HttpSolrClient.request( >> : > > > > >> > > > > > HttpSolrClient.java:255) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.client.solrj. >> : > impl.HttpSolrClient.request( >> : > > > > >> > > > > > HttpSolrClient.java:244) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.client.solrj.i >> mpl.LBHttpSolrClient. >> : > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.client.solrj. >> : > > impl.LBHttpSolrClient.request( >> : > > > > >> > > > > > LBHttpSolrClient.java:413) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.handler.component. >> : > > HttpShardHandlerFactory. >> : > > > > >> > > > > > makeLoadBalancedRequest(HttpSh >> ardHandlerFactory.java: >> : > 273) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.handler.compon >> ent.HttpShardHandler. >> : > > > > >> > lambda$submit$0( >> : > > > > >> > > > > > HttpShardHandler.java:175) >> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask. >> : > > java:266) >> : > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter. >> : > > > > >> > > > > call(Executors.java:511) >> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask. >> : > > java:266) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$ >> : > > > > >> > > > > InstrumentedRunnable.run( >> : > > > > >> > > > > > InstrumentedExecutorService.java:176) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$ >> : > > > > >> > MDCAwareThreadPoolExecutor. >> : > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( >> : > > > > >> > > > > > ThreadPoolExecutor.java:1142) >> : > > > > >> > > > > > at >> : > > > > >> > > > > > java.util.concurrent.ThreadPoo >> lExecutor$Worker.run( >> : > > > > >> > > > > > ThreadPoolExecutor.java:617) >> : > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745) >> : > > > > >> > > > > > >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 >> : > > > > r:core_node3 >> : > > > > >> > > > > > x:images_shard1_replica_n1] >> : > org.apache.solr.core.SolrCore; >> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr >> path=/image >> : > > > > >> > params={q=id:1} >> : > > > > >> > > > > > status=200 QTime=17 >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 >> : > > > > r:core_node3 >> : > > > > >> > > > > > x:images_shard1_replica_n1] >> : > org.apache.solr.core.SolrCore; >> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr >> path=/image >> : > > > > >> > > > > > params={df=text&distrib=false& >> qt=/image&fl=id&fl=score& >> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url= >> : > > > > >> > > > > > http://10.224.30.207:8983/ >> : > solr/images_shard1_replica_n1/ >> : > > > > >> > > > > > |http://10.224.30.207:7574/ >> : > solr/images_shard1_replica_n2/ >> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW= >> : > > > 1523971232039&isShard=true&wt= >> : > > > > >> > javabin} >> : > > > > >> > > > > > hits=1 status=0 QTime=2 >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > I've implemented getcontenttype simply as >> : > > > > >> > > > > > >> : > > > > >> > > > > > @Override >> : > > > > >> > > > > > public String getContentType(SolrQueryRequest >> request, >> : > > > > >> > > > > > SolrQueryResponse response) { >> : > > > > >> > > > > > >> : > > > > >> > > > > > return "application/json;charset=utf-8"; >> : > > > > >> > > > > > } >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev < >> : > > > m...@apache.org> >> : > > > > >> > wrote: >> : > > > > >> > > > > > >> : > > > > >> > > > > > > Lee, >> : > > > > >> > > > > > > It's worth to send a stacktrace for such kind of >> : > > > inquiries. >> : > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() >> or so. >> : > > > Shard >> : > > > > >> > > response >> : > > > > >> > > > > > should >> : > > > > >> > > > > > > contains <uniqueKey> from schema.xml field. >> : > > > > >> > > > > > > I encounter something like this while >> troubleshooting >> : > > > > >> > > > > > > https://lucene.apache.org/ >> : > solr/guide/6_6/transforming- >> : > > > > >> > > > > > > result-documents.html#Transfor >> mingResultDocuments- >> : > > > > >> > > > > > > CoresandCollectionsinSolrCloud >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll < >> : > > > > >> > > > > > lee.a.carr...@googlemail.com >> : > > > > >> > > > > > > > >> : > > > > >> > > > > > > wrote: >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > > I've created a custom response writer which >> extends >> : > > > > >> > > > > RawResponseWriter. >> : > > > > >> > > > > > > The >> : > > > > >> > > > > > > > basic operation is to output a single field >> value >> : > from >> : > > > the >> : > > > > >> top >> : > > > > >> > > > > matching >> : > > > > >> > > > > > > doc >> : > > > > >> > > > > > > > as the entire response. This works when shards >> = 1 >> : > but >> : > > > > fails >> : > > > > >> > when >> : > > > > >> > > > > > shards >> : > > > > >> > > > > > > > are greater than 1. >> : > > > > >> > > > > > > > >> : > > > > >> > > > > > > > I throw an error if the field in question is >> missing >> : > > > from >> : > > > > >> the >> : > > > > >> > top >> : > > > > >> > > > > doc. >> : > > > > >> > > > > > > This >> : > > > > >> > > > > > > > happens when individual shards are being >> searched >> : > and >> : > > > only >> : > > > > >> id >> : > > > > >> > and >> : > > > > >> > > > > score >> : > > > > >> > > > > > > are >> : > > > > >> > > > > > > > returned. I'm sure I've committed a basic >> error. >> : > > > > >> > > > > > > > >> : > > > > >> > > > > > > > Lee C >> : > > > > >> > > > > > > > >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > >> : > > > > >> > > > > > > -- >> : > > > > >> > > > > > > Sincerely yours >> : > > > > >> > > > > > > Mikhail Khludnev >> : > > > > >> > > > > > > >> : > > > > >> > > > > > >> : > > > > >> > > > > >> : > > > > >> > > > > >> : > > > > >> > > > > >> : > > > > >> > > > > -- >> : > > > > >> > > > > Sincerely yours >> : > > > > >> > > > > Mikhail Khludnev >> : > > > > >> > > > > >> : > > > > >> > > > >> : > > > > >> > > >> : > > > > >> > > >> : > > > > >> > > >> : > > > > >> > > -- >> : > > > > >> > > Sincerely yours >> : > > > > >> > > Mikhail Khludnev >> : > > > > >> > > >> : > > > > >> > >> : > > > > >> >> : > > > > >> >> : > > > > >> >> : > > > > >> -- >> : > > > > >> Sincerely yours >> : > > > > >> Mikhail Khludnev >> : > > > > >> >> : > > > > > >> : > > > > > >> : > > > > >> : > > > >> : > > > >> : > > > >> : > > > -- >> : > > > Sincerely yours >> : > > > Mikhail Khludnev >> : > > > >> : > > >> : > >> : > >> : > >> : > -- >> : > Sincerely yours >> : > Mikhail Khludnev >> : > >> : >> >> -Hoss >> http://www.lucidworks.com/ >> > >
