Dockerized Solr service stops responding

2018-10-15 Thread Christian Spitzlay
Hi,

we are running Solr Cloud with a single instance of zookeeper and solr
in a docker setup.  We are experiencing occasional hangs where the solr 
service stops responding: clients time out but the docker containers are still
reported as healthy by the rancher administration UI.

"docker exec" can still connect to the solr container and open a shell but 
a local connect to solr using curl from inside the solr container times out, 
too.

A restart of the Solr container fixes the issue temporarily.

The last log entry we could find from before the hang is an exception
of the type I described in 
https://lists.apache.org/thread.html/eafcdc4c3c620f093f1292f9062badc4001b4957450ba69e4e668d1a@%3Csolr-user.lucene.apache.org%3E
(streaming expressions returning the correct result but throwing an exception 
anyway;
as described, we cannot switch to the export handler in all places as not all 
types we 
work with support docValues).

Can the two things be related, e.g. can lots of exceptions have a side effect
that leads to solr not responding any more?
Do you have any hints as to what to check for?


Best regards
Christian Spitzlay


--  

Christian Spitzlay
Senior Software-Entwickler

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Str. 15
D-60439 Frankfurt am Main

T: +49 69 348 739 116
christian.spitz...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico 
Just
Firmensitz:  Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945











Escaping in streaming expression

2018-05-24 Thread Christian Spitzlay
Hello,

I’m experimenting with streaming expressions and I wonder how to escape a 
double quote in a value.
I am on 7.3.0 and trying with the text area on 
http://localhost:8983/solr/#/collname/stream

The following expression works for me and returns results:
search(kmm, q="sds_endpoint_name:F2", fl="sds_endpoint_name", 
sort="sds_endpoint_name ASC", qt="/export“)

When I try to add a double quote to the value quoted with a backslash like 
this: 
search(kmm, q="sds_endpoint_name:F\"2", fl="sds_endpoint_name", 
sort="sds_endpoint_name ASC", qt="/export")
I get an exception with message:

org.apache.solr.search.SyntaxError:  Cannot parse 'sds_endpoint_name:F\"2': 
Lexical error at line 1, column 22.  Encountered:  after : \"\\\"2\“",

I tried several more levels of escaping with backslashes but none worked so far
(only the error message was different as sometimes the expression was broken in 
different ways)


On http://localhost:8983/solr/#/collname/query, entering
sds_endpoint_name:F\“2
as the q parameter does not throw a syntax error and an empty result is returned
(which is to be expected as there is no document with a quote in the name at 
the moment).


Is there a correct way to escape the double quote in a streaming expression?


Best regards
Christian



Re: Escaping in streaming expression

2018-05-25 Thread Christian Spitzlay
Thanks for your help.
Yes, I think SOLR-10894 is exactly about the issue I have seen.

So if I understand correctly there is currently no way to create a method
in client code (like Drupal’s search_api_solr module) that takes arbitrary user 
input 
and escapes it to get *always* a valid expression for a search for literal 
string values.

The streaming expression builder in that module uses the normal escaping method 
from 
the Solarium library.  I assume those work correctly for non-streaming queries.
But given this unresolved issue I guess the Solarium library and the Drupal 
module 
will inherit the problem.

Is this only about double quotes or are there other meta characters that will 
work 
with backslash-escaping in non-streaming queries but will not parse as part of 
streaming expressions?


Christian Spitzlay



> Am 24.05.2018 um 18:55 schrieb Joel Bernstein :
> 
> I just confirmed that the following query works as expected:
> 
> search(collection2, q="test_s:\"hello world\"", fl="id", sort="id desc")
> 
> In this case the double quotes are used to specify a phrase query.
> 
> But this fails:
> 
> search(collection2, q="test_s:\"hello world", fl="id", sort="id desc")
> 
> In this case the double quote is used as part of the term, which is what I
> believe you had in mind.
> 
> SOLR-10894, I believe was created to address this issue but as of yet no
> patch is available.
> 
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Thu, May 24, 2018 at 12:38 PM, Joel Bernstein  wrote:
> 
>> Also while looking at you're query it looks like you are getting error
>> from the solr query parser. I believe the this is the issue you are facing:
>> 
>> https://issues.apache.org/jira/browse/SOLR-10894
>> 
>> I'll confirm, but I believe this query should work:
>> 
>> search(collection1, q="test \"hello world\""...)
>> 
>> In the query about the double quotes are escaped and send to Solr
>> unescaped to form the query: test "hello world". The query parse has no
>> problem parsing this.
>> 
>> But you're using a double quote not as part of query syntax, but as part
>> of the query term. This is where I believe SOLR-10894 comes into play.
>> 
>> 
>> 
>> 
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>> 
>> On Thu, May 24, 2018 at 11:23 AM, Christian Spitzlay <
>> christian.spitz...@biologis.com> wrote:
>> 
>>> Hello,
>>> 
>>> I’m experimenting with streaming expressions and I wonder how to escape a
>>> double quote in a value.
>>> I am on 7.3.0 and trying with the text area on
>>> http://localhost:8983/solr/#/collname/stream
>>> 
>>> The following expression works for me and returns results:
>>> search(kmm, q="sds_endpoint_name:F2", fl="sds_endpoint_name",
>>> sort="sds_endpoint_name ASC", qt="/export“)
>>> 
>>> When I try to add a double quote to the value quoted with a backslash
>>> like this:
>>> search(kmm, q="sds_endpoint_name:F\"2", fl="sds_endpoint_name",
>>> sort="sds_endpoint_name ASC", qt="/export")
>>> I get an exception with message:
>>> 
>>> org.apache.solr.search.SyntaxError:  Cannot parse
>>> 'sds_endpoint_name:F\"2': Lexical error at line 1, column 22.
>>> Encountered:  after : \"\\\"2\“",
>>> 
>>> I tried several more levels of escaping with backslashes but none worked
>>> so far
>>> (only the error message was different as sometimes the expression was
>>> broken in different ways)
>>> 
>>> 
>>> On http://localhost:8983/solr/#/collname/query, entering
>>> sds_endpoint_name:F\“2
>>> as the q parameter does not throw a syntax error and an empty result is
>>> returned
>>> (which is to be expected as there is no document with a quote in the name
>>> at the moment).
>>> 
>>> 
>>> Is there a correct way to escape the double quote in a streaming
>>> expression?
>>> 
>>> 
>>> Best regards
>>> Christian
>>> 
>>> 
>> 



Streaming Expression intersect() behaviour

2018-06-06 Thread Christian Spitzlay
Hi,

I don’t seem to get the behaviour of the intersect() stream decorator.
I only ever get one doc from the left stream when I would have expected 
more than one.

I constructed a test case that does not depend on my concrete index:

intersect(
cartesianProduct(tuple(fieldA=array(c,c,a,b,d,d)), fieldA, productSort="fieldA 
asc"),
cartesianProduct(tuple(fieldB=array(c,c,a,d,d)), fieldB, productSort="fieldB 
asc"),
on="fieldA=fieldB“
)


The result:

{
  "result-set": {
"docs": [
  {
"fieldA": "a"
  },
  {
"EOF": true,
"RESPONSE_TIME": 0
  }
]
  }
}


I would have expected all the docs from the left stream with fieldA values a, 
c, d
and only the docs with fieldA == b missing.  Do I have a fundamental 
misunderstanding?


Best regards
Christian Spitzlay




Graph traversal: Bypass cycle detection?

2018-06-06 Thread Christian Spitzlay
Hi,

is it possible to bypass the cycle detection so a traversal 
can revisit nodes?

The documentation at
https://lucene.apache.org/solr/guide/7_3/graph-traversal.html#cycle-detection
does not mention any and lists reasons why the cycle detection is in place.
But if I were willing to live with the consequences would it be possible?


Best regards
Christian Spitzlay



Re: Graph traversal: Bypass cycle detection?

2018-06-07 Thread Christian Spitzlay
Hi,


> Am 07.06.2018 um 03:20 schrieb Joel Bernstein :
> 
> Hi,
> 
> At this time cycle detection is built into the nodes expression and cannot
> be turned off. The nodes expression is really designed to do a traditional
> breadth first search through a graph where cycle detection is needed so you
> don't continually walk the same nodes.
> 
> Are you looking to do random walk analysis?
> I've been meaning to add a
> function that supports random walks on a graph that would not do cycle
> detection.


No, this is not about random walks. We have an application that knows
different types of entities and links betweens them. Both entities and links 
are indexed and we create additional documents to represent relations 
between the entities to prepare a network we can search on.
A regular walk with nodes() is part of that.

There is an issue in a situation where one of the entities in the original 
system is linked to itself.  I’m haven’t finished analysing the problem yet
but I wondered whether there was an easy way to rule out that 
cycle detection is causing it.


Best regards,
Christian Spitzlay






> christian.spitz...@biologis.com> wrote:
> 
>> Hi,
>> 
>> is it possible to bypass the cycle detection so a traversal
>> can revisit nodes?
>> 
>> The documentation at
>> https://lucene.apache.org/solr/guide/7_3/graph-
>> traversal.html#cycle-detection
>> does not mention any and lists reasons why the cycle detection is in place.
>> But if I were willing to live with the consequences would it be possible?
>> 
>> 
>> Best regards
>> Christian Spitzlay
>> 
>> 



Re: Streaming Expression intersect() behaviour

2018-06-07 Thread Christian Spitzlay
Hi,

I noticed that my mail program broke the test case by replacing a double
quote with a different UTF-8 character.

Here is the test case again and I hope it will work this time:

intersect(
cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA, productSort="fieldA 
asc"),
cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB asc"),
on="fieldA=fieldB"
)

I simplified it a bit, too. I still get one document with fieldA == a.
I would have expected three documents in the output, one with fieldA == a and 
two with fieldB == c.
Did I misunderstand the docs of the intersect decorator or have I come across a 
bug?


Best regards,
Christian Spitzlay



> Am 06.06.2018 um 10:18 schrieb Christian Spitzlay 
> :
> 
> Hi,
> 
> I don’t seem to get the behaviour of the intersect() stream decorator.
> I only ever get one doc from the left stream when I would have expected 
> more than one.
> 
> I constructed a test case that does not depend on my concrete index:
> 
> intersect(
> cartesianProduct(tuple(fieldA=array(c,c,a,b,d,d)), fieldA, 
> productSort="fieldA asc"),
> cartesianProduct(tuple(fieldB=array(c,c,a,d,d)), fieldB, productSort="fieldB 
> asc"),
> on="fieldA=fieldB“
> )
> 
> 
> The result:
> 
> {
>  "result-set": {
>"docs": [
>  {
>"fieldA": "a"
>  },
>  {
>"EOF": true,
>"RESPONSE_TIME": 0
>  }
>]
>  }
> }
> 
> 
> I would have expected all the docs from the left stream with fieldA values a, 
> c, d
> and only the docs with fieldA == b missing.  Do I have a fundamental 
> misunderstanding?
> 
> 
> Best regards
> Christian Spitzlay
> 
> 



Re: Streaming Expression intersect() behaviour

2018-06-07 Thread Christian Spitzlay



> Am 07.06.2018 um 11:34 schrieb Christian Spitzlay 
> :
> 
> intersect(
> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA, productSort="fieldA 
> asc"),
> cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB asc"),
> on="fieldA=fieldB"
> )
> 
> I simplified it a bit, too. I still get one document with fieldA == a.
> I would have expected three documents in the output, one with fieldA == a and 
> two with fieldB == c.

That should have read „… and two with fieldA == c“ of course.





Re: Streaming Expression intersect() behaviour

2018-06-08 Thread Christian Spitzlay
Hi,


> Am 08.06.2018 um 03:42 schrieb Joel Bernstein :
> 
> And when you transpose the "on" fields like this:
> 
> intersect(
> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA, productSort="fieldA
> asc"),
> cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB
> asc"),
> on="fieldB=fieldA"
> )
> 
> It also works.


No, IIUC this does not work correctly. 

I had tried this before posting my original question.
That version emits the documents from the left stream
but does not filter out the document with fieldA == b.

This might be due to the fact that fieldB is not present in the left stream
and fieldA is not present in the right stream; it compares two
empty values (null?) and comes to the conclusion that they are equal.
Could that be the reason?



> So, yes there is a bug where the fields are being transposed with intersect
> function's "on" fields. The same issue was happening with joins and may
> have been resolved. I'll do little more research into this.

Thanks for your work on this!


Best regards
Christian Spitzlay





> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Thu, Jun 7, 2018 at 9:29 AM, Christian Spitzlay <
> christian.spitz...@biologis.com> wrote:
> 
>> 
>> 
>>> Am 07.06.2018 um 11:34 schrieb Christian Spitzlay <
>> christian.spitz...@biologis.com>:
>>> 
>>> intersect(
>>> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA,
>> productSort="fieldA asc"),
>>> cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB
>> asc"),
>>> on="fieldA=fieldB"
>>> )
>>> 
>>> I simplified it a bit, too. I still get one document with fieldA == a.
>>> I would have expected three documents in the output, one with fieldA ==
>> a and two with fieldB == c.
>> 
>> That should have read „… and two with fieldA == c“ of course.
>> 
>> 
>> 
>> 



Re: Streaming Expression intersect() behaviour

2018-06-08 Thread Christian Spitzlay
As a temporary workaround until that issue is fixed 
one could wrap the right stream with a select that renames the field:

intersect(
cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA, productSort="fieldA 
asc"),
select(cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB 
asc"), fieldB as fieldA),
on=fieldA
)



> Am 08.06.2018 um 14:42 schrieb Joel Bernstein :
> 
> You're correct, after testing again the only way that this works correctly
> appears to be:
> 
> intersect(
> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA, productSort="fieldA
> asc"),
> cartesianProduct(tuple(fieldA=array(a,c)), fieldA, productSort="fieldA
> asc"),
> on="fieldA"
> )
> 
> I suspect that there are only test cases that cover this scenario as well.
> I'll create a jira issue for this.
> 
> 
> 
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Fri, Jun 8, 2018 at 3:41 AM, Christian Spitzlay <
> christian.spitz...@biologis.com> wrote:
> 
>> Hi,
>> 
>> 
>>> Am 08.06.2018 um 03:42 schrieb Joel Bernstein :
>>> 
>>> And when you transpose the "on" fields like this:
>>> 
>>> intersect(
>>> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA,
>> productSort="fieldA
>>> asc"),
>>> cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB
>>> asc"),
>>> on="fieldB=fieldA"
>>> )
>>> 
>>> It also works.
>> 
>> 
>> No, IIUC this does not work correctly.
>> 
>> I had tried this before posting my original question.
>> That version emits the documents from the left stream
>> but does not filter out the document with fieldA == b.
>> 
>> This might be due to the fact that fieldB is not present in the left stream
>> and fieldA is not present in the right stream; it compares two
>> empty values (null?) and comes to the conclusion that they are equal.
>> Could that be the reason?
>> 
>> 
>> 
>>> So, yes there is a bug where the fields are being transposed with
>> intersect
>>> function's "on" fields. The same issue was happening with joins and may
>>> have been resolved. I'll do little more research into this.
>> 
>> Thanks for your work on this!
>> 
>> 
>> Best regards
>> Christian Spitzlay
>> 
>> 
>> 
>> 
>> 
>>> Joel Bernstein
>>> http://joelsolr.blogspot.com/
>>> 
>>> On Thu, Jun 7, 2018 at 9:29 AM, Christian Spitzlay <
>>> christian.spitz...@biologis.com> wrote:
>>> 
>>>> 
>>>> 
>>>>> Am 07.06.2018 um 11:34 schrieb Christian Spitzlay <
>>>> christian.spitz...@biologis.com>:
>>>>> 
>>>>> intersect(
>>>>> cartesianProduct(tuple(fieldA=array(a,b,c,c)), fieldA,
>>>> productSort="fieldA asc"),
>>>>> cartesianProduct(tuple(fieldB=array(a,c)), fieldB, productSort="fieldB
>>>> asc"),
>>>>> on="fieldA=fieldB"
>>>>> )
>>>>> 
>>>>> I simplified it a bit, too. I still get one document with fieldA == a.
>>>>> I would have expected three documents in the output, one with fieldA ==
>>>> a and two with fieldB == c.
>>>> 
>>>> That should have read „… and two with fieldA == c“ of course.
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 



Exception when processing streaming expression

2018-06-13 Thread Christian Spitzlay
Hi,

I am seeing a lot of (reproducible) exceptions in my solr log file
when I execute streaming expressions:

o.a.s.s.HttpSolrCall  Unable to write response, client closed connection or we 
are shutting down
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:292)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
at 
org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:372)
at 
org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:794)
[…]
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at 
java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
at 
java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:506)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:272)
... 69 more


I have read up on the exception message and found 
http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-connection-or-we-are-shutting-down-tt4350349.html#a4350947
but I don’t understand how an early client connect can cause what I am seeing:

What puzzles me is that the response has been delivered in full to the 
client library, including the document with EOF.

So Solr must have already processed the streaming expression and returned the 
result.
It’s just that the log is filled with stacktraces of this exception that 
suggests something went wrong.
I don’t understand why this happens when the query seems to have succeeded.


Best regards,
Christian




Re: Exception when processing streaming expression

2018-06-14 Thread Christian Spitzlay
Here ist one I stripped down as far as I could:

innerJoin(sort(search(kmm, 
q="sds_endpoint_uuid:(2f927a0b\-fe38\-451e\-9103\-580914a77e82)", 
fl="sds_endpoint_uuid,sds_to_endpoint_uuid", sort="sds_to_endpoint_uuid ASC", 
qt="/export"), by="sds_endpoint_uuid ASC"), search(kmm, 
q=ss_search_api_datasource:entity\:as_metadata, fl="sds_metadata_of_uuid", 
sort="sds_metadata_of_uuid ASC", qt="/select", rows=1), 
on="sds_endpoint_uuid=sds_metadata_of_uuid")

The exception happens both via PHP (search_api_solr / Solarium) and via the 
Solr admin UI.
(version: Solr 7.3.1 on macOS High Sierra 10.13.5)

It seems to be related to the fact that the second stream uses "select“. 
- If I use "export“ the exception doesn’t occur.
- If I set the rows parameter "low enough“ so I do not get any results
  the exception doesn’t occur either.


BTW: Do you know of any tool for formatting and/or syntax highlighting 
these expressions?


Christian Spitzlay





> Am 13.06.2018 um 23:02 schrieb Joel Bernstein :
> 
> Can your provide some example expressions that are causing these exceptions?
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
> christian.spitz...@biologis.com> wrote:
> 
>> Hi,
>> 
>> I am seeing a lot of (reproducible) exceptions in my solr log file
>> when I execute streaming expressions:
>> 
>> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
>> or we are shutting down
>> org.eclipse.jetty.io.EofException
>>at org.eclipse.jetty.io.ChannelEndPoint.flush(
>> ChannelEndPoint.java:292)
>>at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>>at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>>at org.eclipse.jetty.io.AbstractEndPoint.write(
>> AbstractEndPoint.java:372)
>>at org.eclipse.jetty.server.HttpConnection$SendCallback.
>> process(HttpConnection.java:794)
>> […]
>>at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
>> EatWhatYouKill.java:131)
>>at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
>> ReservedThread.run(ReservedThreadExecutor.java:382)
>>at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:708)
>>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:626)
>>at java.base/java.lang.Thread.run(Thread.java:844)
>> Caused by: java.io.IOException: Broken pipe
>>at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>>at java.base/sun.nio.ch.SocketDispatcher.writev(
>> SocketDispatcher.java:51)
>>at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>>at java.base/sun.nio.ch.SocketChannelImpl.write(
>> SocketChannelImpl.java:506)
>>at org.eclipse.jetty.io.ChannelEndPoint.flush(
>> ChannelEndPoint.java:272)
>>... 69 more
>> 
>> 
>> I have read up on the exception message and found
>> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
>> but I don’t understand how an early client connect can cause what I am
>> seeing:
>> 
>> What puzzles me is that the response has been delivered in full to the
>> client library, including the document with EOF.
>> 
>> So Solr must have already processed the streaming expression and returned
>> the result.
>> It’s just that the log is filled with stacktraces of this exception that
>> suggests something went wrong.
>> I don’t understand why this happens when the query seems to have succeeded.
>> 
>> 
>> Best regards,
>> Christian
>> 
>> 
>> 



Streaming Expressions: Merge array values? Inverse of cartesianProduct()

2018-06-14 Thread Christian Spitzlay
Hi,

is there a way to merge array values?

Something that transforms

{
  "k1": "1",
  "k2": ["a", "b"] 
},
{
  "k1": "2",
  "k2": ["c", "d"] 
},
{
  "k1": "2",
  "k2": ["e", "f"] 
}

into 

{
  "k1": "1",
  "k2": ["a", "b"] 
},
{
  "k1": "2",
  "k2": ["c", "d", "e", "f"] 
}


And an inverse of cartesianProduct() that transforms

{
  "k1": "1",
  "k2": "a"
},
{
  "k1": "2",
  "k2": "b"
},
{
  "k1": "2",
  "k2": "c"
}

into 

{
  "k1": "1",
  "k2": ["a"]
},
{
  "k1": "2",
  "k2": ["b", "c"] 
}


Christian




Re: Exception when processing streaming expression

2018-06-14 Thread Christian Spitzlay
What does that mean exactly?  If I set the rows parameter to 10
the exception still occurs.  AFAICT all this happens internally during the 
processing of the streaming expression.  Why wouldn't the select send 
the EOF tuple when it reaches the end of the documents? 
Or why wouldn't the receiving end wait for it to appear?  
Due to an incredibly low timeout used internally?


Christian Spitzlay



> Am 14.06.2018 um 19:18 schrieb Susmit :
> 
> Hi, 
> This may be expected if one of the streams is closed early - does not reach 
> to EOF tuple
> 
> Sent from my iPhone
> 
>> On Jun 14, 2018, at 9:53 AM, Christian Spitzlay 
>>  wrote:
>> 
>> Here ist one I stripped down as far as I could:
>> 
>> innerJoin(sort(search(kmm, 
>> q="sds_endpoint_uuid:(2f927a0b\-fe38\-451e\-9103\-580914a77e82)", 
>> fl="sds_endpoint_uuid,sds_to_endpoint_uuid", sort="sds_to_endpoint_uuid 
>> ASC", qt="/export"), by="sds_endpoint_uuid ASC"), search(kmm, 
>> q=ss_search_api_datasource:entity\:as_metadata, fl="sds_metadata_of_uuid", 
>> sort="sds_metadata_of_uuid ASC", qt="/select", rows=1), 
>> on="sds_endpoint_uuid=sds_metadata_of_uuid")
>> 
>> The exception happens both via PHP (search_api_solr / Solarium) and via the 
>> Solr admin UI.
>> (version: Solr 7.3.1 on macOS High Sierra 10.13.5)
>> 
>> It seems to be related to the fact that the second stream uses "select“. 
>> - If I use "export“ the exception doesn’t occur.
>> - If I set the rows parameter "low enough“ so I do not get any results
>> the exception doesn’t occur either.
>> 
>> 
>> BTW: Do you know of any tool for formatting and/or syntax highlighting 
>> these expressions?
>> 
>> 
>> Christian Spitzlay
>> 
>> 
>> 
>> 
>> 
>>> Am 13.06.2018 um 23:02 schrieb Joel Bernstein :
>>> 
>>> Can your provide some example expressions that are causing these exceptions?
>>> 
>>> Joel Bernstein
>>> http://joelsolr.blogspot.com/
>>> 
>>> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
>>> christian.spitz...@biologis.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I am seeing a lot of (reproducible) exceptions in my solr log file
>>>> when I execute streaming expressions:
>>>> 
>>>> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
>>>> or we are shutting down
>>>> org.eclipse.jetty.io.EofException
>>>>  at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>>> ChannelEndPoint.java:292)
>>>>  at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>>>>  at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>>>>  at org.eclipse.jetty.io.AbstractEndPoint.write(
>>>> AbstractEndPoint.java:372)
>>>>  at org.eclipse.jetty.server.HttpConnection$SendCallback.
>>>> process(HttpConnection.java:794)
>>>> […]
>>>>  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
>>>> EatWhatYouKill.java:131)
>>>>  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
>>>> ReservedThread.run(ReservedThreadExecutor.java:382)
>>>>  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>> QueuedThreadPool.java:708)
>>>>  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>>>> QueuedThreadPool.java:626)
>>>>  at java.base/java.lang.Thread.run(Thread.java:844)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>  at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>>>>  at java.base/sun.nio.ch.SocketDispatcher.writev(
>>>> SocketDispatcher.java:51)
>>>>  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>>>>  at java.base/sun.nio.ch.SocketChannelImpl.write(
>>>> SocketChannelImpl.java:506)
>>>>  at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>>> ChannelEndPoint.java:272)
>>>>  ... 69 more
>>>> 
>>>> 
>>>> I have read up on the exception message and found
>>>> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
>>>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
>>>> but I don’t understand how an early client connect can cause what I am
>>>> seeing:
>>>> 
>>>> What puzzles me is that the response has been delivered in full to the
>>>> client library, including the document with EOF.
>>>> 
>>>> So Solr must have already processed the streaming expression and returned
>>>> the result.
>>>> It’s just that the log is filled with stacktraces of this exception that
>>>> suggests something went wrong.
>>>> I don’t understand why this happens when the query seems to have succeeded.
>>>> 
>>>> 
>>>> Best regards,
>>>> Christian
>>>> 
>>>> 
>>>> 
>> 



Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()

2018-06-15 Thread Christian Spitzlay
Hi,

I had come across the reduce function in the docs but
I have a hard time getting it to work; I haven't found any documentation on it
or its parameters, and the source code of the GroupOperation doesn't explain it 
either ...
For example, what is the "n" parameter about?


I constructed a source stream to produce the input from my second example:

merge(
sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1 
asc"), by="k1 asc"),
sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2, productSort="k1 
asc"), by="k1 asc"),
on="k1 asc"
  )

---> 

{
  "result-set": {
"docs": [
  {
"k1": "1",
"k2": "a"
  },
  {
"k1": "2",
"k2": "b"
  },
  {
"k1": "2",
"k2": "c"
  },
  {
"EOF": true,
"RESPONSE_TIME": 0
  }
]
  }
}



Then wrapped in a reduce function:


reduce(
  merge(
sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1 
asc"), by="k1 asc"),
sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2, productSort="k1 
asc"), by="k1 asc"),
on="k1 asc"
  ),
  by="k1",
  group(sort="k1 asc", n="10")
)

---> 

{
  "result-set": {
"docs": [
  {
"k1": "1",
"k2": "a",
"group": [
  {
"k1": "1",
"k2": "a"
  }
]
  },
  {
"k1": "2",
"k2": "c",
"group": [
  {
"k1": "2",
"k2": "c"
  },
  {
"k1": "2",
"k2": "b"
  }
]
  },
  {
"EOF": true,
"RESPONSE_TIME": 0
  }
]
  }
}


It adds a field "group" that contains an array of the unchanged input documents 
with the same "by" value, 
not grouped values.

{
  "result-set": {
"docs": [
  {
"k1": "1",
"k2": "a",
"group": [
  {
"k1": "1",
"k2": "a"
  }
]
  },
  {
"k1": "2",
"k2": "c",
"group": [
  {
"k1": "2",
"k2": "c"
  },
  {
"k1": "2",
"k2": "b"
  }
]
  },
  {
"EOF": true,
"RESPONSE_TIME": 0
  }
]
  }
}


Or am I doing it wrong?

Christian Spitzlay










> Am 15.06.2018 um 01:48 schrieb Joel Bernstein :
> 
> Actually you're second example is probably a straight forward:
> 
> reduce(select(...), group(...), by="k1")
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Thu, Jun 14, 2018 at 7:33 PM, Joel Bernstein  wrote:
> 
>> Take a look at the reduce() function. You'll have to write a custom reduce
>> operation but you can follow the example here:
>> 
>> https://github.com/apache/lucene-solr/blob/master/solr/
>> solrj/src/java/org/apache/solr/client/solrj/io/ops/GroupOperation.java
>> 
>> You can plug in your custom reduce operation in the solrconfig.xml and use
>> it like any other function. If you're interested in working on this you
>> could create a ticket and I can provide guidance.
>> 
>> 
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>> 
>> 2018-06-14 13:13 GMT-04:00 Christian Spitzlay <
>> christian.spitz...@biologis.com>:
>> 
>>> Hi,
>>> 
>>> is there a way to merge array values?
>>> 
>>> Something that transforms
>>> 
>>> {
>>>  "k1": "1",
>>>  "k2": ["a", "b"]
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": ["c", "d"]
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": ["e", "f"]
>>> }
>>> 
>>> into
>>> 
>>> {
>>>  "k1": "1",
>>>  "k2": ["a", "b"]
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": ["c", "d", "e", "f"]
>>> }
>>> 
>>> 
>>> And an inverse of cartesianProduct() that transforms
>>> 
>>> {
>>>  "k1": "1",
>>>  "k2": "a"
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": "b"
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": "c"
>>> }
>>> 
>>> into
>>> 
>>> {
>>>  "k1": "1",
>>>  "k2": ["a"]
>>> },
>>> {
>>>  "k1": "2",
>>>  "k2": ["b", "c"]
>>> }
>>> 
>>> 
>>> Christian
>>> 
>>> 
>>> 
>> 



Re: Exception when processing streaming expression

2018-06-15 Thread Christian Spitzlay


> Am 15.06.2018 um 01:23 schrieb Joel Bernstein :
> 
> We have to check the behavior of the innerJoin. I suspect that its closing
> the second stream when the first stream his finished. This would cause a
> broken pipe with the second stream. The export handler has specific code
> that eats the broken pipe exception so it doesn't end up in the logs. The
> select hander does not have this code.

Ah, I see.  The stack trace in my original mail has the "broken pipe" message:

[...]
Caused by: java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
[...]


> In general you never want to use the select handler and set the rows to
> such a big number. If you have that many rows you'll want to use the export
> and handler which is designed to export the entire result set.


We started out with the export handler but we are updating documents using 
streaming expressions and we had fields that had types 
that do not support docValues, according to the documentation at
https://lucene.apache.org/solr/guide/7_3/docvalues.html#enabling-docvalues

We switched to the select handler in some places and it worked. 
We set the rows parameter to a large value:
"If you want to tell Solr to return all possible results from the query without 
an 
upper bound, specify rows to be 1000 or some other ridiculously 
large value that is higher than the possible number of rows that are expected."
From:
https://wiki.apache.org/solr/CommonQueryParameters#rows


Christian Spitzlay





Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()

2018-06-19 Thread Christian Spitzlay



> Am 18.06.2018 um 15:30 schrieb Joel Bernstein :
> 
> You are doing things correctly. I was incorrect about the behavior of the
> group() operation.
> 
> I think the behavior you are looking for should be done using reduce() but
> we'll need to create a reduce operation that does this. If you want to
> create a ticket we can work through exactly how the operation would work.
> 


I'll create an issue tonight at the latest.
Should we take further discussions off the user list
or is it acceptable here?


Christian Spitzlay


--  

Christian Spitzlay
Diplom-Physiker,
Senior Software-Entwickler

Tel: +49 69 / 348739116
E-Mail: christian.spitz...@biologis.com

bio.logis Genetic Information Management GmbH
Altenhöferallee 3
60438 Frankfurt am Main

Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico 
Just
Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
Umsatzsteuer-Identifikationsnummer DE293587677








Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()

2018-06-19 Thread Christian Spitzlay
Ok. I'm about to create the issue and I have a draft version of what I had in 
mind 
in a branch on github.

Christian Spitzlay


> Am 19.06.2018 um 15:27 schrieb Joel Bernstein :
> 
> Let's move the discussion to the jira ticket.
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Tue, Jun 19, 2018 at 3:42 AM, Christian Spitzlay <
> christian.spitz...@biologis.com> wrote:
> 
>> 
>> 
>>> Am 18.06.2018 um 15:30 schrieb Joel Bernstein :
>>> 
>>> You are doing things correctly. I was incorrect about the behavior of the
>>> group() operation.
>>> 
>>> I think the behavior you are looking for should be done using reduce()
>> but
>>> we'll need to create a reduce operation that does this. If you want to
>>> create a ticket we can work through exactly how the operation would work.
>>> 
>> 
>> 
>> I'll create an issue tonight at the latest.
>> Should we take further discussions off the user list
>> or is it acceptable here?
>> 
>> 
>> Christian Spitzlay
>> 
>> 
>> --
>> 
>> Christian Spitzlay
>> Diplom-Physiker,
>> Senior Software-Entwickler
>> 
>> Tel: +49 69 / 348739116
>> E-Mail: christian.spitz...@biologis.com
>> 
>> bio.logis Genetic Information Management GmbH
>> Altenhöferallee 3
>> 60438 Frankfurt am Main
>> 
>> Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt
>> Enrico Just
>> Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
>> Umsatzsteuer-Identifikationsnummer DE293587677
>> 
>> 
>> 
>> 
>> 
>> 
>> 



Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()

2018-06-19 Thread Christian Spitzlay
Here it is: https://issues.apache.org/jira/browse/SOLR-12499


--  

Christian Spitzlay
Diplom-Physiker,
Senior Software-Entwickler

Tel: +49 69 / 348739116
E-Mail: christian.spitz...@biologis.com

bio.logis Genetic Information Management GmbH
Altenhöferallee 3
60438 Frankfurt am Main

Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico 
Just
Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
Umsatzsteuer-Identifikationsnummer DE293587677




> Am 19.06.2018 um 15:30 schrieb Christian Spitzlay 
> :
> 
> Ok. I'm about to create the issue and I have a draft version of what I had in 
> mind 
> in a branch on github.
> 
> Christian Spitzlay
> 
> 
>> Am 19.06.2018 um 15:27 schrieb Joel Bernstein :
>> 
>> Let's move the discussion to the jira ticket.
>> 
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>> 
>> On Tue, Jun 19, 2018 at 3:42 AM, Christian Spitzlay <
>> christian.spitz...@biologis.com> wrote:
>> 
>>> 
>>> 
>>>> Am 18.06.2018 um 15:30 schrieb Joel Bernstein :
>>>> 
>>>> You are doing things correctly. I was incorrect about the behavior of the
>>>> group() operation.
>>>> 
>>>> I think the behavior you are looking for should be done using reduce()
>>> but
>>>> we'll need to create a reduce operation that does this. If you want to
>>>> create a ticket we can work through exactly how the operation would work.
>>>> 
>>> 
>>> 
>>> I'll create an issue tonight at the latest.
>>> Should we take further discussions off the user list
>>> or is it acceptable here?
>>> 
>>> 
>>> Christian Spitzlay
>>> 
>>> 
>>> --
>>> 
>>> Christian Spitzlay
>>> Diplom-Physiker,
>>> Senior Software-Entwickler
>>> 
>>> Tel: +49 69 / 348739116
>>> E-Mail: christian.spitz...@biologis.com
>>> 
>>> bio.logis Genetic Information Management GmbH
>>> Altenhöferallee 3
>>> 60438 Frankfurt am Main
>>> 
>>> Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt
>>> Enrico Just
>>> Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
>>> Umsatzsteuer-Identifikationsnummer DE293587677
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
> 



Re: Preferred PHP Client Library

2018-07-17 Thread Christian Spitzlay
Hi,

we're using Solarium, too, via Drupal's search_api / search_api_solr modules,
and I hear there are plans by the Typo3 project to move to Solarium.

So there are at least two major CMS projects that will be relying on it in the 
future.

I don't have any experience with wordpress but a web search 
returned wpsolr (I assume you are aware of this):
https://www.wpsolr.com/knowledgebase/what-is-your-solr-php-client-library/
So someone from the wordpress  world seems to be using it successfully, too.

I haven't used the standard extension so I cannot comment on the differences
but here is the Solarium project's description in their own words:
https://solarium.readthedocs.io/en/latest/

Solarium is under active development. They recently added support
for Solr cloud streaming expressions and for the JSON Facet API
(the latter is in the beta version).


Best regards
Christian Spitzlay


> Am 16.07.2018 um 21:19 schrieb Zimmermann, Thomas 
> :
> 
> Hi,
> 
> We're in the midst of our first major Solr upgrade in years and are trying to 
> run some cleanup across
> all of our client codebases. We're currently using the standard PHP Solr 
> Extension when communicating
> with our cluster from our Wordpress installs. 
> http://php.net/manual/en/book.solr.php
> 
> Few questions.
> 
> Should we have any concerns about communicating with a Solr 7 cloud from that 
> client?
> Is anyone using another client they prefer? If so what are the benefits of 
> switching to it?
> 
> Thanks!
> TZ



--  

Christian Spitzlay
Diplom-Physiker,
Senior Software-Entwickler

E-Mail: christian.spitz...@biologis.com

bio.logis Genetic Information Management GmbH
Altenhöferallee 3
60438 Frankfurt am Main

Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico 
Just
Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
Umsatzsteuer-Identifikationsnummer DE293587677




Re: Exception when processing streaming expression

2018-07-24 Thread Christian Spitzlay
Hi,


> Am 15.06.2018 um 14:54 schrieb Christian Spitzlay 
> :
> 
> 
>> Am 15.06.2018 um 01:23 schrieb Joel Bernstein :
>> 
>> We have to check the behavior of the innerJoin. I suspect that its closing
>> the second stream when the first stream his finished. This would cause a
>> broken pipe with the second stream. The export handler has specific code
>> that eats the broken pipe exception so it doesn't end up in the logs. The
>> select hander does not have this code.
> 
> Ah, I see.  The stack trace in my original mail has the "broken pipe" message:
> 
> [...]
> Caused by: java.io.IOException: Broken pipe
>   at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
> [...]



Should I open a Jira ticket about the innerJoin issue?




>> In general you never want to use the select handler and set the rows to
>> such a big number. If you have that many rows you'll want to use the export
>> and handler which is designed to export the entire result set.
> 
> 
> We started out with the export handler but we are updating documents using 
> streaming expressions and we had fields that had types 
> that do not support docValues, according to the documentation at
> https://lucene.apache.org/solr/guide/7_3/docvalues.html#enabling-docvalues
> 
> We switched to the select handler in some places and it worked. 
> We set the rows parameter to a large value:
> "If you want to tell Solr to return all possible results from the query 
> without an 
> upper bound, specify rows to be 1000 or some other ridiculously 
> large value that is higher than the possible number of rows that are 
> expected."
> From:
> https://wiki.apache.org/solr/CommonQueryParameters#rows



Since we have trouble switching back to the export handler,
do you have any ideas how we could temporarily keep this exception from 
filling the solr log file when I run my code?


Christian



--  

Christian Spitzlay
Diplom-Physiker,
Senior Software-Entwickler

Tel: +49 69 / 348739116
E-Mail: christian.spitz...@biologis.com

bio.logis Genetic Information Management GmbH
Altenhöferallee 3
60438 Frankfurt am Main

Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico 
Just
Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945
Umsatzsteuer-Identifikationsnummer DE293587677




Re: How do I add my own Streaming Expressions?

2019-11-18 Thread Christian Spitzlay
HI,

I assume you are trying to add a custom extension, 
not preparing a general purpose contribution for solr itself ...

We succeeded in loading a custom reduce operation by adding the jar
with our code:



And then defining an "expressible":


Both changes were made in solrconfig_extra.xml, a file included from our 
solrconfig.xml.


HTH,
Christian


> Am 16.11.2019 um 03:43 schrieb Eric Pugh :
> 
> What is the process for adding new Streaming Expressions?   
> 
> It appears that the org.apache.solr.client.solrj.io.Lang method statically 
> loads all the streaming expressions?
> 
> Eric
> 
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> http://www.opensourceconnections.com <http://www.opensourceconnections.com/> 
> | My Free/Busy <http://tinyurl.com/eric-cal>  
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>   
> This e-mail and all contents, including attachments, is considered to be 
> Company Confidential unless explicitly stated otherwise, regardless of 
> whether attachments are marked as such.


--  

Christian Spitzlay
Senior Software-Entwickler

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Str. 15
D-60439 Frankfurt am Main

T: +49 69 348 739 116
christian.spitz...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Firmensitz:  Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945



Re: How to tell which core was used based on Json or XML response from Solr

2019-11-25 Thread Christian Spitzlay
Hi rhys,

if you are taking the PHP route for the mentioned server part then I would 
suggest 
using a client library, not plain curl.  There is solarium, for instance:

https://solarium.readthedocs.io/en/stable/
https://github.com/solariumphp/solarium

It can use curl under the hood but you can program your stuff on a higher level,
against an API.


Christian


> Am 25.11.2019 um 07:09 schrieb Paras Lehana :
> 
> Hey rhys,
> 
> What David suggested is what we do for querying Solr. You can figure out
> our frontend implementation of Auto-Suggest by seeing the AJAX requests
> fired when you type in the search box on www.indiamart.com.
> 
> Why are you using two jQuery files? If you have a web server, you already
> know that which core you queried from. Just convert the Solr JSON response
> and add the key "core" and return the modified JSON response. Keep your
> front-end query simple - just describe your query. All the other parameters
> can be added on the web server side. Anyways, why do you want to know the
> core name?
> 
> On Sat, 23 Nov 2019 at 00:23, David Hastings 
> wrote:
> 
>> i personally dont like php, but it may just be the easiest way to do what
>> you need assuming you have a basic web server,
>> send your search query to php, and use $_GET or $_POST to read it into a
>> variable:
>> https://www.php.net/manual/en/reserved.variables.get.php
>> 
>> then send that to the solr server in the same piece of php with curl
>> 
>> https://phpenthusiast.com/blog/five-php-curl-examples
>> 
>> and return the raw result if you want.  at the very least it hides its url,
>> but with this you can block the solr port to outside ip's and only allow 80
>> or whatever your webserver is using
>> 
>> 
>> On Fri, Nov 22, 2019 at 1:43 PM rhys J  wrote:
>> 
>>> On Fri, Nov 22, 2019 at 1:39 PM David Hastings <
>>> hastings.recurs...@gmail.com>
>>> wrote:
>>> 
>>>> 2 things (maybe 3):
>>>> 1.  dont have this code facing a client thats not you, otherwise anyone
>>>> could view the source and see where the solr server is, which means
>> they
>>>> can destroy your index or anything they want.  put at the very least a
>>>> simple api/front end in between the javascript page for the user and
>> the
>>>> solr server
>>>> 
>>> 
>>> Is there a way I can fix this?
>>> 
>>> 
>>>> 2. i dont think there is a way, you would be better off indexing an
>>>> indicator of sorts into your documents
>>>> 
>>> 
>>> Oh this is a good idea.
>>> 
>>> Thanks!
>>> 
>>> 3. the jquery in your example already has the core identified, not sure
>> why
>>>> the receiving javascript wouldn't be able to read that variable unless
>> im
>>>> missing something.
>>>> 
>>>> 
>>> There's another function on_data that is being called by the url, which
>>> does not have any indication of what the core was, only the response from
>>> the url.
>>> 
>>> Thanks,
>>> 
>>> Rhys
>>> 
>> 
> 
> 
> -- 
> -- 
> Regards,
> 
> *Paras Lehana* [65871]
> Development Engineer, Auto-Suggest,
> IndiaMART Intermesh Ltd.
> 
> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> Noida, UP, IN - 201303
> 
> Mob.: +91-9560911996
> Work: 01203916600 | Extn:  *8173*
> 
> -- 
> IMPORTANT: 
> NEVER share your IndiaMART OTP/ Password with anyone.


--  

Christian Spitzlay
Senior Software-Entwickler

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Str. 15
D-60439 Frankfurt am Main

T: +49 69 348 739 116
christian.spitz...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Firmensitz:  Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945






Re: How to tell which core was used based on Json or XML response from Solr

2019-11-25 Thread Christian Spitzlay
I was referring to David Hastings' suggestion of shielding solr from direct 
access
which is something I strongly agree with.

If you're not going with a PHP-based server-side application
as to not expose your solr directly to the javascript application (and thus to 
possible 
manipulation by an end user) then you obviously won't need solarium.

As Paras Lehana said:
"Keep your front-end query simple - just describe your query. All the other 
parameters
can be added on the web server side."

... that could then be implemented in your Perl code.


Christian



> Am 25.11.2019 um 16:32 schrieb rhys J :
> 
>> if you are taking the PHP route for the mentioned server part then I would
>> suggest
>> using a client library, not plain curl.  There is solarium, for instance:
>> 
>> https://solarium.readthedocs.io/en/stable/
>> https://github.com/solariumphp/solarium
>> 
>> It can use curl under the hood but you can program your stuff on a higher
>> level,
>> against an API.
>> 
>> 
> I am using jquery, so I am using the json package to send and decode the
> json that solr sends. I hope that makes sense?
> 
> Thanks for your tip!
> 
> Our pages are a combo of jquery, javascript, and perl.


--  

Christian Spitzlay
Senior Software-Entwickler

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Str. 15
D-60439 Frankfurt am Main

T: +49 69 348 739 116
christian.spitz...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Firmensitz:  Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945







Re: How to control the number of grouped results [DRUPAL]

2019-12-03 Thread Christian Spitzlay
Hi alee2,

judging from the first screenshot I think you are not on the Solr level at all
but on the level of a "Views" integration of a "SearchAPI"-based search
(which happens to be a Solr search in your setup).

I suggest you ask that question on https://drupalchat.me/ in the 
#drupal-support channel.  
There is also a dedicated #search channel but in the general one the audience 
is larger
and this may be a "Views" question rather than purely a search-related one.


HTH
Christian



> Am 02.12.2019 um 17:18 schrieb Erick Erickson :
> 
> This is a Solr parameter, how Drupal allows you to pass it on would probably 
> be a better asked of Drupal.
> 
>> On Dec 2, 2019, at 10:01 AM, Diego Ceccarelli (BLOOMBERG/ LONDON) 
>>  wrote:
>> 
>> This parameter referers to the Solr request, for example: 
>> 
>> https://lucene.apache.org/solr/guide/7_0/result-grouping.html#grouping-by-query
>> 
>> Drupal should expose it in the API, I guess? 
>> 
>> Cheers,
>> diego
>> 
>> From: solr-user@lucene.apache.org At: 12/02/19 14:47:06To:  
>> solr-user@lucene.apache.org
>> Subject: Re: How to control the number of grouped results [DRUPAL]
>> 
>> Where would I put this parameter ?
>> 
>> 
>> --
>> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html


--  

Christian Spitzlay
Senior Software-Entwickler

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Str. 15
D-60439 Frankfurt am Main

T: +49 69 348 739 116
christian.spitz...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Firmensitz:  Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945