Re: maxwarmingSearchers and memory leak

[SOLR4189]

Shawn, you are right.
* OS vendor and version 
CentosOS 6.5

* Java vendor and version
OpenJDK version 1.8.0_20
OpenJDK 64-bit Server VM (build 25.20-b23)

* Servlet container used to start Solr. 
Catalina(tomcat7)

* Total amount of memory in the server. 
30 GB

* Max heap size for Solr. 
8GB(JVM)

* An idea of exactly what is running on the server. 
On our servers runs solr service only and splunk forwarder

* Total index size and document count being handled by Solr (add up all 
indexes). 
60GB and 2.6 milion on one shard

* A screen shot of a process list sorted by memory usage. 
 

* A screenshot showing total system memory allocations
 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/maxwarmingSearchers-and-memory-leak-tp4321937p4322362.html
Sent from the Solr - User mailing list archive at Nabble.com.

Auto-generate unique key when adding documents from SolrJ

[OTH]

Hello all,

First of all, I am very new to Solr.

I am using Solr version 6.4.1.  I have a Solr core (non-cloud), where there
is a mandatory unique key field called "id".

I am trying to add documents to the core from Java, without having to
specify the "id" field explicitly; i.e. to have it auto-generated.

I learned that this is possible by including the following information in
the conf/solrconfig.xml file:

> 
> 
> 
> id
>   
> ...
> 
> 
> 
>   


(I did restart the server after adding the above text to the xml file.)

However, when I try to add documents from Java using SolrJ (without
specifying the "id" field), I get the following exception:

> Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://localhost:8983/solr/sales_history: Document is
> missing mandatory uniqueKey field: id


My Java code is like this:

> SolrClient solr = new HttpSolrClient.Builder(SOLR_URL).build();
> SolrInputDocument document = new SolrInputDocument();
> document.addField(..., ...);
> document.addField(..., ...);
> UpdateResponse updateResponse = solr.add(document);


The exception is thrown from the last line above.

Is there any way to add documents from Java and have the uniqueKey field be
auto-generated?


Thank you

Re: Auto-generate unique key when adding documents from SolrJ

[Alexandre Rafalovitch]

It is not enough to declare URP chain, you have to invoke it.

Either by marking it default or by adding the update.chain parameter
to the request handler (or in initParams) you use to update the
documents (usually /update). See, for example:
https://github.com/apache/lucene-solr/blob/master/solr/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml#L837

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 26 February 2017 at 10:11, OTH  wrote:
> Hello all,
>
> First of all, I am very new to Solr.
>
> I am using Solr version 6.4.1.  I have a Solr core (non-cloud), where there
> is a mandatory unique key field called "id".
>
> I am trying to add documents to the core from Java, without having to
> specify the "id" field explicitly; i.e. to have it auto-generated.
>
> I learned that this is possible by including the following information in
> the conf/solrconfig.xml file:
>
>> 
>> 
>> 
>> id
>>   
>> ...
>> 
>> 
>> 
>>   
>
>
> (I did restart the server after adding the above text to the xml file.)
>
> However, when I try to add documents from Java using SolrJ (without
> specifying the "id" field), I get the following exception:
>
>> Exception in thread "main"
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
>> from server at http://localhost:8983/solr/sales_history: Document is
>> missing mandatory uniqueKey field: id
>
>
> My Java code is like this:
>
>> SolrClient solr = new HttpSolrClient.Builder(SOLR_URL).build();
>> SolrInputDocument document = new SolrInputDocument();
>> document.addField(..., ...);
>> document.addField(..., ...);
>> UpdateResponse updateResponse = solr.add(document);
>
>
> The exception is thrown from the last line above.
>
> Is there any way to add documents from Java and have the uniqueKey field be
> auto-generated?
>
>
> Thank you

About editing managed-schema by hand

[Issei Nishigata]

Hi, All

Similar questions may have been already asked, but just in case please let
me ask you.
According to the below URL it says as "Schema modifications via the Schema
API will now be enabled by default.",
but would there be any issues if I edited with text editor instead of
Schema API?

https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6


In the answer to the past question, it seemed okay.

http://lucene.472066.n3.nabble.com/Solr-6-managed-schema-amp-version-control-td4289243.html


I was worried because managed-schema said "" when managed-schema was
automatically generated from schema.xml.
If I need to use Schema API and if I wanted to do some process that cannot
be done with Schema API(modifying unique key, etc), what should I do?


Thanks,
Issei

Re: Auto-generate unique key when adding documents from SolrJ

[OTH]

Thanks, great, it's working now!
Omer

On Sun, Feb 26, 2017 at 8:24 PM, Alexandre Rafalovitch 
wrote:

> It is not enough to declare URP chain, you have to invoke it.
>
> Either by marking it default or by adding the update.chain parameter
> to the request handler (or in initParams) you use to update the
> documents (usually /update). See, for example:
> https://github.com/apache/lucene-solr/blob/master/solr/
> server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml#L837
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 26 February 2017 at 10:11, OTH  wrote:
> > Hello all,
> >
> > First of all, I am very new to Solr.
> >
> > I am using Solr version 6.4.1.  I have a Solr core (non-cloud), where
> there
> > is a mandatory unique key field called "id".
> >
> > I am trying to add documents to the core from Java, without having to
> > specify the "id" field explicitly; i.e. to have it auto-generated.
> >
> > I learned that this is possible by including the following information in
> > the conf/solrconfig.xml file:
> >
> >> 
> >> 
> >> 
> >> id
> >>   
> >> ...
> >> 
> >> 
> >> 
> >>   
> >
> >
> > (I did restart the server after adding the above text to the xml file.)
> >
> > However, when I try to add documents from Java using SolrJ (without
> > specifying the "id" field), I get the following exception:
> >
> >> Exception in thread "main"
> >> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error
> >> from server at http://localhost:8983/solr/sales_history: Document is
> >> missing mandatory uniqueKey field: id
> >
> >
> > My Java code is like this:
> >
> >> SolrClient solr = new HttpSolrClient.Builder(SOLR_URL).build();
> >> SolrInputDocument document = new SolrInputDocument();
> >> document.addField(..., ...);
> >> document.addField(..., ...);
> >> UpdateResponse updateResponse = solr.add(document);
> >
> >
> > The exception is thrown from the last line above.
> >
> > Is there any way to add documents from Java and have the uniqueKey field
> be
> > auto-generated?
> >
> >
> > Thank you
>

Re: About editing managed-schema by hand

[Erick Erickson]

This is the sequence that gets you in trouble:
> start solr
> hand edit the schema _without_ reloading your collection or restarting all 
> your Solr instances.
> use the managed-schema API to make modifications.

In this scenario your hand-edits can be lost since the in-memory version of the
schema is written out without re-fetching it from Zookeeper.

If you only ever hand-edit your schema, you'll be fine.

If you conscientiously reload your collection (or restart all your Solr's) after
you hand-edit your schema, you'll be fine even if you use the managed schema
API calls.

But really, if you want to hand-edit your schema why not go back to using
the ClassicSchemaFactory? See:
https://cwiki.apache.org/confluence/display/solr/Schema+Factory+Definition+in+SolrConfig#SchemaFactoryDefinitioninSolrConfig-SwitchingfromManagedSchematoManuallyEditedschema.xml

Best,
Erick

On Sun, Feb 26, 2017 at 8:22 AM, Issei Nishigata  wrote:
> Hi, All
>
> Similar questions may have been already asked, but just in case please let
> me ask you.
> According to the below URL it says as "Schema modifications via the Schema
> API will now be enabled by default.",
> but would there be any issues if I edited with text editor instead of
> Schema API?
>
> https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6
>
>
> In the answer to the past question, it seemed okay.
>
> http://lucene.472066.n3.nabble.com/Solr-6-managed-schema-amp-version-control-td4289243.html
>
>
> I was worried because managed-schema said "" when managed-schema was
> automatically generated from schema.xml.
> If I need to use Schema API and if I wanted to do some process that cannot
> be done with Schema API(modifying unique key, etc), what should I do?
>
>
> Thanks,
> Issei

Add fieldType from Solr API

[OTH]

Hello,

I am new to Solr, and am using Solr v. 6.4.1.

I need to add a new "fieldType" to my schema.  My version of Solr is using
the "managed-schema" XML file, which I gather one is not supposed to modify
directly.  Is it possible to add a new fieldType using the Solr Admin via
the browser?  The "schema" page doesn't seem to provide this option, at
least from what I can tell.

Thanks

Re: Add fieldType from Solr API

[Alexandre Rafalovitch]

You can hand edit it, just make sure to reload the collection after.

Otherwise, I believe, there is API.

Not the Admin UI yet, unfortunately.

Regards,
Alex

On 26 Feb 2017 1:50 PM, "OTH"  wrote:

Hello,

I am new to Solr, and am using Solr v. 6.4.1.

I need to add a new "fieldType" to my schema.  My version of Solr is using
the "managed-schema" XML file, which I gather one is not supposed to modify
directly.  Is it possible to add a new fieldType using the Solr Admin via
the browser?  The "schema" page doesn't seem to provide this option, at
least from what I can tell.

Thanks

Re: Add fieldType from Solr API

[Kevin Risden]

As Alex said there is no Admin UI support. The API is called the Schema API:

https://cwiki.apache.org/confluence/display/solr/Schema+API

That allows you to modify the schema programatically. You will have to
reload the collection either way.

Kevin Risden

On Sun, Feb 26, 2017 at 1:33 PM, Alexandre Rafalovitch 
wrote:

> You can hand edit it, just make sure to reload the collection after.
>
> Otherwise, I believe, there is API.
>
> Not the Admin UI yet, unfortunately.
>
> Regards,
> Alex
>
> On 26 Feb 2017 1:50 PM, "OTH"  wrote:
>
> Hello,
>
> I am new to Solr, and am using Solr v. 6.4.1.
>
> I need to add a new "fieldType" to my schema.  My version of Solr is using
> the "managed-schema" XML file, which I gather one is not supposed to modify
> directly.  Is it possible to add a new fieldType using the Solr Admin via
> the browser?  The "schema" page doesn't seem to provide this option, at
> least from what I can tell.
>
> Thanks
>

solr warning - filling logs

[Satya Marivada]

Hi All,

I have configured solr with SSL and enabled http authentication. It is all
working fine on the solr admin page, indexing and querying process. One
bothering thing is that it is filling up logs every second saying no
authority, I have configured host name, port and authentication parameters
right in all config files. Not sure, where is it coming from. Any
suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud with
embedded zookeeper. Could it be some bug with solr-6.3.0 or am I missing
some configuration?

2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse exception:
java.lang.IllegalArgumentException: No Authority for
HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
at
org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
at org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io
.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

Re: solr warning - filling logs

[Dave]

You shouldn't use the embedded zookeeper with solr, it's just for development 
not anywhere near worthy of being out in production. Otherwise it looks like 
you may have a port scanner running. In any case don't use the zk that comes 
with solr 

> On Feb 26, 2017, at 6:52 PM, Satya Marivada  wrote:
> 
> Hi All,
> 
> I have configured solr with SSL and enabled http authentication. It is all
> working fine on the solr admin page, indexing and querying process. One
> bothering thing is that it is filling up logs every second saying no
> authority, I have configured host name, port and authentication parameters
> right in all config files. Not sure, where is it coming from. Any
> suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud with
> embedded zookeeper. Could it be some bug with solr-6.3.0 or am I missing
> some configuration?
> 
> 2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
> r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse exception:
> java.lang.IllegalArgumentException: No Authority for
> HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
> java.lang.IllegalArgumentException: No Authority
> at
> org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
> at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
> at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
> at
> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at org.eclipse.jetty.io
> .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)

Re: solr warning - filling logs

[Satya Marivada]

May I ask about the port scanner running? Can you please elaborate?
Sure, will try to move out to external zookeeper

On Sun, Feb 26, 2017 at 7:07 PM Dave  wrote:

> You shouldn't use the embedded zookeeper with solr, it's just for
> development not anywhere near worthy of being out in production. Otherwise
> it looks like you may have a port scanner running. In any case don't use
> the zk that comes with solr
>
> > On Feb 26, 2017, at 6:52 PM, Satya Marivada 
> wrote:
> >
> > Hi All,
> >
> > I have configured solr with SSL and enabled http authentication. It is
> all
> > working fine on the solr admin page, indexing and querying process. One
> > bothering thing is that it is filling up logs every second saying no
> > authority, I have configured host name, port and authentication
> parameters
> > right in all config files. Not sure, where is it coming from. Any
> > suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud
> with
> > embedded zookeeper. Could it be some bug with solr-6.3.0 or am I missing
> > some configuration?
> >
> > 2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
> > r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse exception:
> > java.lang.IllegalArgumentException: No Authority for
> > HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
> > java.lang.IllegalArgumentException: No Authority
> > at
> >
> org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
> > at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
> > at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
> > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > at org.eclipse.jetty.io
> > .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
>

Re: solr warning - filling logs

[Dave]

I don't know about your network setup but a port scanner sometimes can be an it 
security device that, well, scans ports looking to see if they're open. 

> On Feb 26, 2017, at 7:14 PM, Satya Marivada  wrote:
> 
> May I ask about the port scanner running? Can you please elaborate?
> Sure, will try to move out to external zookeeper
> 
>> On Sun, Feb 26, 2017 at 7:07 PM Dave  wrote:
>> 
>> You shouldn't use the embedded zookeeper with solr, it's just for
>> development not anywhere near worthy of being out in production. Otherwise
>> it looks like you may have a port scanner running. In any case don't use
>> the zk that comes with solr
>> 
>>> On Feb 26, 2017, at 6:52 PM, Satya Marivada 
>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I have configured solr with SSL and enabled http authentication. It is
>> all
>>> working fine on the solr admin page, indexing and querying process. One
>>> bothering thing is that it is filling up logs every second saying no
>>> authority, I have configured host name, port and authentication
>> parameters
>>> right in all config files. Not sure, where is it coming from. Any
>>> suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud
>> with
>>> embedded zookeeper. Could it be some bug with solr-6.3.0 or am I missing
>>> some configuration?
>>> 
>>> 2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
>>> r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse exception:
>>> java.lang.IllegalArgumentException: No Authority for
>>> HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
>>> java.lang.IllegalArgumentException: No Authority
>>> at
>>> 
>> org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
>>> at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
>>> at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
>>> at
>>> 
>> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
>>> at
>>> 
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
>>> at org.eclipse.jetty.io
>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>>> at
>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
>>> at org.eclipse.jetty.io
>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>>> at org.eclipse.jetty.io
>>> .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>>> at
>>> 
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>>> at
>>> 
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>>> at
>>> 
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>>> at
>>> 
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> at java.lang.Thread.run(Thread.java:745)
>>