Re: SolrCloud - Fails to read db config file

[Danny]

Hi,

I seem to have the same issue here.

I'm running a very simple Solr server in standalone mode, using a DIH with
the following datasource in *dataconfig.xml* :


It works fine.

Then I stop the standalone server, empty the data directory, and start a
cluster with an embedded ZooKeeper, just by changing the command line to
this one (called from the */danny* directory) :


The Solr cloud starts without problems.

Then I go in the admin interface, in the *Dataimport* page of the core, to
execute a data import. But now the *Entity* dropdown list is empty, and I
get this error in the log :


So it looks like the same problem deniz had.

I don't understand what does /configs/danny_conf/ represents for ZooKeeper ?
Is it a /real/ path, or a /virtual/ one ?

I created a symbolic link that allowed the path
/configs/danny_conf//danny/solr/core1/conf/data-config.xml to be actually
readable, but I still had the error, so I guess it's not a real path that
ZooKeeper is looking for.

In the admin interface, under *Cloud > Tree*, I can see under
/configs/danny_conf/ that all the files that are listed are actually the
files that are under this real path : /danny/solr/core1/conf/ (like
data-config.xml for example).

But I still don't understand what's going on...

Also, I didn't understand Mark Miller's tip :

Mark Miller-3 wrote
> As a workaround, try explicitly setting the directory to write the
> properties file in with the "directory" param. You should be able to set
> it to anything, as it should not be used.

Thanks for any help here :)

Danny.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/SolrCloud-Fails-to-read-db-config-file-tp4022299p4122963.html
Sent from the Solr - User mailing list archive at Nabble.com.

array of array in document response structure

[Danny]

Hello,

I'm wondering if it is possible to have arrays of arrays in the response
structure.

For example, I can use DataImportHandler to index this XML file :



into this structure :



But as you can see, in this index I lost the links between "two" elements
and "one" elements.

So I'm wondering if there's a way to store data in the index with arrays of
arrays, which would yield something like this :



or something else, as long as the structure keeps the relations between
array levels.

Thanks,
Danny.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/array-of-array-in-document-response-structure-tp4041494.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: array of array in document response structure

[Danny]

Ok :)

I'm wondering if it is possible to have arrays of arrays in the response
structure.
For example, I can use DataImportHandler to index this XML file :

http://pastebin.com/j2s3iVmk

into this structure :

http://pastebin.com/MJ7YYh0L

But as you can see, in this index I lost the links between "two" elements
and "one" elements.
So I'm wondering if there's a way to store data in the index with arrays of
arrays, which would yield something like this :

http://pastebin.com/j3eY1eVv

or something else, as long as the structure keeps the relations between
array levels.

Thanks,
Danny.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/array-of-array-in-document-response-structure-tp4041494p4041548.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: array of array in document response structure

[Danny]

Erick : you're right, allowing nested arrays would be like opening Pandora's
Box :)

Michaël : having parallel arrays and losing their relations is what I want
to avoid, actually :)

I guess I'll have to find another way.

Thanks,
Danny.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/array-of-array-in-document-response-structure-tp4041494p4041575.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr node 'Gone' status

[danny teichthal]

>From my short experience, it  indicates that the particular node lost
connection with zookeeper.
Like Binoy said, It may be because the process/host is down, but also could
be a result of a network problem.

On Tue, Jan 19, 2016 at 12:20 PM, Binoy Dalal 
wrote:

> In my experience 'Gone' indicates that that particular solr instance itself
> is down.
> To bring it back up, simply restart solr.
>
> On Tue, 19 Jan 2016, 11:16 davidphilip cherian <
> davidphilipcher...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Solr-admin cloud view page has got another new radio button indicating
> > status of node :  'Gone' status. What does that mean?  One of my
> collection
> > is in that state and it is not serving any request. How to bring that up?
> >
> --
> Regards,
> Binoy Dalal
>

SolrCloud - Strategy for recovering cluster states

[danny teichthal]

Hi,


I would like to describe a process we use for overcoming problems in
cluster state when we have networking issues. Would appreciate if anyone
can answer about what are the flaws on this solution and what is the best
practice for recovery in case of network problems involving zookeeper.
I'm working with Solr Cloud with version 5.2.1
~100 collections in a cluster of 6 machines.

This is the short procedure:
1. Bring all the cluster down.
2. Clear all data from zookeeper.
3. Upload configuration.
4. Restart the cluster.

We rely on the fact that a collection is created on core discovery process,
if it does not exist. It gives us much flexibility.
When the cluster comes up, it reads from core.properties and creates the
collections if needed.
Since we have only one configuration, the collections are automatically
linked to it and the cores inherit it from the collection.
This is a very robust procedure, that helped us overcome many problems
until we stabilized our cluster which is now pretty stable.
I know that the leader might change in such case and may lose updates, but
it is ok.


The problem is that today I want to add a new config set.
When I add it and clear zookeeper, the cores cannot be created because
there are 2 configurations. This breaks my recovery procedure.

I thought about a few options:
1. Put the config Name in core.properties - this doesn't work. (It is
supported in CoreAdminHandler, but  is discouraged according to
documentation)
2. Change recovery procedure to not delete all data from zookeeper, but
only relevant parts.
3. Change recovery procedure to delete all, but recreate and link
configurations for all collections before startup.

Option #1 is my favorite, because it is very simple, it is currently not
supported, but from looking on code it looked like it is not complex to
implement.



My questions are:
1. Is there something wrong in the recovery procedure that I described ?
2. What is the best way to fix problems in cluster state, except from
editing clusterstate.json manually? Is there an automated tool for that? We
have about 100 collections in a cluster, so editing is not really a
solution.
3.Is creating a collection via core.properties is also discouraged?



Would very appreciate any answers/ thoughts on that.


Thanks,

Re: SolrCloud - Strategy for recovering cluster states

[danny teichthal]

Hi,
Just summarizing my questions if the long mail is a little intimidating:
1. Is there a best practice/automated tool for overcoming problems in
cluster state coming from zookeeper disconnections?
2. Creating a collection via core admin is discouraged, is it true also for
core.properties discovery?

I would like to be able to specify collection.configName in the
core.properties and when starting server, the collection will be created
and linked to the config name specified.



On Mon, Feb 29, 2016 at 4:01 PM, danny teichthal 
wrote:

> Hi,
>
>
> I would like to describe a process we use for overcoming problems in
> cluster state when we have networking issues. Would appreciate if anyone
> can answer about what are the flaws on this solution and what is the best
> practice for recovery in case of network problems involving zookeeper.
> I'm working with Solr Cloud with version 5.2.1
> ~100 collections in a cluster of 6 machines.
>
> This is the short procedure:
> 1. Bring all the cluster down.
> 2. Clear all data from zookeeper.
> 3. Upload configuration.
> 4. Restart the cluster.
>
> We rely on the fact that a collection is created on core discovery
> process, if it does not exist. It gives us much flexibility.
> When the cluster comes up, it reads from core.properties and creates the
> collections if needed.
> Since we have only one configuration, the collections are automatically
> linked to it and the cores inherit it from the collection.
> This is a very robust procedure, that helped us overcome many problems
> until we stabilized our cluster which is now pretty stable.
> I know that the leader might change in such case and may lose updates, but
> it is ok.
>
>
> The problem is that today I want to add a new config set.
> When I add it and clear zookeeper, the cores cannot be created because
> there are 2 configurations. This breaks my recovery procedure.
>
> I thought about a few options:
> 1. Put the config Name in core.properties - this doesn't work. (It is
> supported in CoreAdminHandler, but  is discouraged according to
> documentation)
> 2. Change recovery procedure to not delete all data from zookeeper, but
> only relevant parts.
> 3. Change recovery procedure to delete all, but recreate and link
> configurations for all collections before startup.
>
> Option #1 is my favorite, because it is very simple, it is currently not
> supported, but from looking on code it looked like it is not complex to
> implement.
>
>
>
> My questions are:
> 1. Is there something wrong in the recovery procedure that I described ?
> 2. What is the best way to fix problems in cluster state, except from
> editing clusterstate.json manually? Is there an automated tool for that? We
> have about 100 collections in a cluster, so editing is not really a
> solution.
> 3.Is creating a collection via core.properties is also discouraged?
>
>
>
> Would very appreciate any answers/ thoughts on that.
>
>
> Thanks,
>
>
>
>
>
>

Re: SolrCloud - Strategy for recovering cluster states

[danny teichthal]

Thanks Jeff,
I understand your philosophy and it sounds correct.
Since we had many problems with zookeeper when switching to Solr Cloud. we
couldn't make it as a source of knowledge and had to relay on a more stable
source.
The issues is that when we get such an event of zookeeper, it brought our
system down, and in this case, clearing the core.properties were a life
saver.
We've managed to make it pretty stable not, but we will always need a
"dooms day" weapon.

I looked into the related JIRA and it confused me a little, and raised a
few other questions:
1. What exactly defines zookeeper as a truth?
2. What is the role of core.properties if the state is only in zookeeper?



Your tool is very interesting, I just thought about writing such a tool
myself.
>From the sources I understand that you represent each node as a path in the
git repository.
So, I guess that for restore purposes I will have to do
the opposite direction and create a node for every path entry.




On Tue, Mar 1, 2016 at 11:36 PM, Jeff Wartes  wrote:

>
> I’ve been running SolrCloud clusters in various versions for a few years
> here, and I can only think of two or three cases that the ZK-stored cluster
> state was broken in a way that I had to manually intervene by hand-editing
> the contents of ZK. I think I’ve seen Solr fixes go by for those cases,
> too. I’ve never completely wiped ZK. (Although granted, my ZK cluster has
> been pretty stable, and my collection count is smaller than yours)
>
> My philosophy is that ZK is the source of cluster configuration, not the
> collection of core.properties files on the nodes.
> Currently, cluster state is shared between ZK and core directories. I’d
> prefer, and I think Solr development is going this way, (SOLR-7269) that
> all cluster state exist and be managed via ZK, and all state be removed
> from the local disk of the cluster nodes. The fact that a node uses local
> disk based configuration to figure out what collections/replicas it has is
> something that should be fixed, in my opinion.
>
> If you’re frequently getting into bad states due to ZK issues, I’d suggest
> you file bugs against Solr for the fact that you got into the state, and
> then fix your ZK cluster.
>
> Failing that, can you just periodically back up your ZK data and restore
> it if something breaks? I wrote a little tool to watch clusterstate.json
> and write every version to a local git repo a few years ago. I was mostly
> interested because I wanted to see changes that happened pretty fast, but
> it could also serve as a backup approach. Here’s a link, although I clearly
> haven’t touched it lately. Feel free to ask if you have issues:
> https://github.com/randomstatistic/git_zk_monitor
>
>
>
>
> On 3/1/16, 12:09 PM, "danny teichthal"  wrote:
>
> >Hi,
> >Just summarizing my questions if the long mail is a little intimidating:
> >1. Is there a best practice/automated tool for overcoming problems in
> >cluster state coming from zookeeper disconnections?
> >2. Creating a collection via core admin is discouraged, is it true also
> for
> >core.properties discovery?
> >
> >I would like to be able to specify collection.configName in the
> >core.properties and when starting server, the collection will be created
> >and linked to the config name specified.
> >
> >
> >
> >On Mon, Feb 29, 2016 at 4:01 PM, danny teichthal 
> >wrote:
> >
> >> Hi,
> >>
> >>
> >> I would like to describe a process we use for overcoming problems in
> >> cluster state when we have networking issues. Would appreciate if anyone
> >> can answer about what are the flaws on this solution and what is the
> best
> >> practice for recovery in case of network problems involving zookeeper.
> >> I'm working with Solr Cloud with version 5.2.1
> >> ~100 collections in a cluster of 6 machines.
> >>
> >> This is the short procedure:
> >> 1. Bring all the cluster down.
> >> 2. Clear all data from zookeeper.
> >> 3. Upload configuration.
> >> 4. Restart the cluster.
> >>
> >> We rely on the fact that a collection is created on core discovery
> >> process, if it does not exist. It gives us much flexibility.
> >> When the cluster comes up, it reads from core.properties and creates the
> >> collections if needed.
> >> Since we have only one configuration, the collections are automatically
> >> linked to it and the cores inherit it from the collection.
> >> This is a very robust procedure, that helped us overcome many problems
> >> until we stabilized our cluster which is now pretty stable.
> >> I know that t

Re: SolrCloud - Strategy for recovering cluster states

[danny teichthal]

According to what you describe, I really don't see the need of core
discovery in Solr Cloud. It will only be used to eagerly load a core on
startup.
If I understand correctly, when ZK = truth, this eager loading can/should
be done by consulting zookeeper instead of local disk.
I agree that it is really confusing.
The best strategy that I see form is to stop relying on core.properties and
keep it all in zookeeper.


On Wed, Mar 2, 2016 at 7:54 PM, Jeff Wartes  wrote:

> Well, with the understanding that someone who isn’t involved in the
> process is describing something that isn’t built yet...
>
> I could imagine changes like:
>  - Core discovery ignores cores that aren’t present in the ZK cluster state
>  - New cores are automatically created to bring a node in line with ZK
> cluster state (addreplica, essentially)
>
> So if the clusterstate said “node XYZ has a replica of shard3 of
> collection1 and that’s all”, and you downed node XYZ and deleted the data
> directory, it’d get restored when you started the node again. And if you
> copied the core directory for shard1 of collection2 in there and restarted
> the node, it’d get ignored because the clusterstate says node XYZ doesn’t
> have that.
>
> More importantly, if you completely destroyed a node and rebuilt it from
> an image, (AWS?) that image wouldn't need any special core directories
> specific to that node. As long as the node name was the same, Solr would
> handle bringing that node back to where it was in the cluster.
>
> Back to opinions, I think mixing the cluster definition between local disk
> on the nodes and ZK clusterstate is just confusing. It should really be one
> or the other. Specifically, I think it should be local disk for
> non-SolrCloud, and ZK for SolrCloud.
>
>
>
>
>
> On 3/2/16, 12:13 AM, "danny teichthal"  wrote:
>
> >Thanks Jeff,
> >I understand your philosophy and it sounds correct.
> >Since we had many problems with zookeeper when switching to Solr Cloud. we
> >couldn't make it as a source of knowledge and had to relay on a more
> stable
> >source.
> >The issues is that when we get such an event of zookeeper, it brought our
> >system down, and in this case, clearing the core.properties were a life
> >saver.
> >We've managed to make it pretty stable not, but we will always need a
> >"dooms day" weapon.
> >
> >I looked into the related JIRA and it confused me a little, and raised a
> >few other questions:
> >1. What exactly defines zookeeper as a truth?
> >2. What is the role of core.properties if the state is only in zookeeper?
> >
> >
> >
> >Your tool is very interesting, I just thought about writing such a tool
> >myself.
> >From the sources I understand that you represent each node as a path in
> the
> >git repository.
> >So, I guess that for restore purposes I will have to do
> >the opposite direction and create a node for every path entry.
> >
> >
> >
> >
> >On Tue, Mar 1, 2016 at 11:36 PM, Jeff Wartes 
> wrote:
> >
> >>
> >> I’ve been running SolrCloud clusters in various versions for a few years
> >> here, and I can only think of two or three cases that the ZK-stored
> cluster
> >> state was broken in a way that I had to manually intervene by
> hand-editing
> >> the contents of ZK. I think I’ve seen Solr fixes go by for those cases,
> >> too. I’ve never completely wiped ZK. (Although granted, my ZK cluster
> has
> >> been pretty stable, and my collection count is smaller than yours)
> >>
> >> My philosophy is that ZK is the source of cluster configuration, not the
> >> collection of core.properties files on the nodes.
> >> Currently, cluster state is shared between ZK and core directories. I’d
> >> prefer, and I think Solr development is going this way, (SOLR-7269) that
> >> all cluster state exist and be managed via ZK, and all state be removed
> >> from the local disk of the cluster nodes. The fact that a node uses
> local
> >> disk based configuration to figure out what collections/replicas it has
> is
> >> something that should be fixed, in my opinion.
> >>
> >> If you’re frequently getting into bad states due to ZK issues, I’d
> suggest
> >> you file bugs against Solr for the fact that you got into the state, and
> >> then fix your ZK cluster.
> >>
> >> Failing that, can you just periodically back up your ZK data and restore
> >> it if something breaks? I wrote a little tool to watch clusterstate.json
> >> and write every version to a local git repo a few y

Re: Questions on SolrCloud core state, when will Solr recover a "DOWN" core to "ACTIVE" core.

[danny teichthal]

Hi Li,
If you could supply some more info from your logs would help.
We also had some similar issue. There were some bugs related to SolrCloud
that were solved on solr 4.10.4 and further on solr 5.x.
I would suggest you compare your logs with defects on 4.10.4 release notes
to see if they are the same.
Also, send relevant solr/zookeeper parts of logs to the mailing list.


On Thu, Apr 21, 2016 at 1:50 AM, Li Ding  wrote:

> Hi All,
>
> We are using SolrCloud 4.6.1.  We have observed following behaviors
> recently.  A Solr node in a Solrcloud cluster is up but some of the cores
> on the nodes are marked as down in Zookeeper.  If the cores are parts of a
> multi-sharded collection with one replica,  the queries to that collection
> will fail.  However, when this happened, if we issue queries to the core
> directly, it returns 200 and correct info.  But once Solr got into the
> state, the core will be marked down forever unless we do a restart on Solr.
>
> Has anyone seen this behavior before?  Is there any to get out of the state
> on its own?
>
> Thanks,
>
> Li
>

Cluster down for long time after zookeeper disconnection

[danny teichthal]

Hi,
We are using Solr cloud with solr 4.10.4.
On the passed week we encountered a problem where all of our servers
disconnected from zookeeper cluster.
This might be ok, the problem is that after reconnecting to zookeeper it
looks like for every collection both replicas do not have a leader and are
stuck in some kind of a deadlock for a few minutes.

>From what we understand:
One of the replicas assume it ill be the leader and at some point starting
to wait on leaderVoteWait, which is by default 3 minutes.
The other replica is stuck on this part of code for a few minutes:
 at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)
at
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:921)
at
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1521)
at
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:392)

Looks like replica 1 waits for a leader to be registered in the zookeeper,
but replica 2 is waiting for replica 1.
(org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp).

We have 100 collections distributed in 3 pairs of Solr nodes. Each
collection has one shard with 2 replicas.
As I understand from code and logs, all the collections are being
registered synchronously, which means that we have to wait 3 minutes *
number of collections for the whole cluster to come up. It could be more
than an hour!



1. We thought about lowering leaderVoteWait to solve the problem, but we
are not sure what is the risk?

2. The following thread is very similar to our case:
http://qnalist.com/questions/4812859/waitforleadertoseedownstate-when-leader-is-down.
Does anybody know if it is indeed a bug and if there's a related JIRA issue?

3. I see this on logs before the reconnection "Client session timed out,
have not heard from server in 48865ms for sessionid 0x44efbb91b5f0001,
closing socket connection and attempting reconnect", does it mean that
there was a disconnection of over 50 seconds between SOLR and zookeeper?


Thanks in advance for your kind answer

Re: Cluster down for long time after zookeeper disconnection

[danny teichthal]

Hi Alexander ,
Thanks for your reply, I looked at the release notes.
There is one bug fix - SOLR-7503
<https://issues.apache.org/jira/browse/SOLR-7503> – register cores
asynchronously.
It may reduce the registration time since it is done on parallel, but
still, 3 minutes (leaderVoteWait) is a long time to recover from a few
seconds of disconnection.

Except from that one I don't see any bug fix that addresses the same
problem.
I am able to reproduce it on 4.10.4 pretty easily, I will also try it with
5.2.1 and see if it reproduces.

Anyway, since migrating to 5.2.1 is not an option for me in the short term,
I'm left with the question if reducing leaderVoteWait may help here, and
what may be the consequences.
If i understand correctly, there might be a chance of losing updates that
were made on leader.
>From my side it is a lot worse to lose availability for 3 minutes.

I would really appreciate a feedback on this.




On Mon, Aug 10, 2015 at 6:55 PM, Alexandre Rafalovitch 
wrote:

> Did you look at release notes for Solr versions after your own?
>
> I am pretty sure some similar things were identified and/or resolved
> for 5.x. It may not help if you cannot migrate, but would at least
> give a confirmation and maybe workaround on what you are facing.
>
> Regards,
>Alex.
> 
> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> http://www.solr-start.com/
>
>
> On 10 August 2015 at 11:37, danny teichthal  wrote:
> > Hi,
> > We are using Solr cloud with solr 4.10.4.
> > On the passed week we encountered a problem where all of our servers
> > disconnected from zookeeper cluster.
> > This might be ok, the problem is that after reconnecting to zookeeper it
> > looks like for every collection both replicas do not have a leader and
> are
> > stuck in some kind of a deadlock for a few minutes.
> >
> > From what we understand:
> > One of the replicas assume it ill be the leader and at some point
> starting
> > to wait on leaderVoteWait, which is by default 3 minutes.
> > The other replica is stuck on this part of code for a few minutes:
> >  at
> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)
> > at
> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:921)
> > at
> >
> org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1521)
> > at
> >
> org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:392)
> >
> > Looks like replica 1 waits for a leader to be registered in the
> zookeeper,
> > but replica 2 is waiting for replica 1.
> >
> (org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp).
> >
> > We have 100 collections distributed in 3 pairs of Solr nodes. Each
> > collection has one shard with 2 replicas.
> > As I understand from code and logs, all the collections are being
> > registered synchronously, which means that we have to wait 3 minutes *
> > number of collections for the whole cluster to come up. It could be more
> > than an hour!
> >
> >
> >
> > 1. We thought about lowering leaderVoteWait to solve the problem, but we
> > are not sure what is the risk?
> >
> > 2. The following thread is very similar to our case:
> >
> http://qnalist.com/questions/4812859/waitforleadertoseedownstate-when-leader-is-down
> .
> > Does anybody know if it is indeed a bug and if there's a related JIRA
> issue?
> >
> > 3. I see this on logs before the reconnection "Client session timed out,
> > have not heard from server in 48865ms for sessionid 0x44efbb91b5f0001,
> > closing socket connection and attempting reconnect", does it mean that
> > there was a disconnection of over 50 seconds between SOLR and zookeeper?
> >
> >
> > Thanks in advance for your kind answer
>

Re: Cluster down for long time after zookeeper disconnection

[danny teichthal]

Erick, I assume you are referring to zkClientTimeout, it is set to 30
seconds. I also see these messages on Solr side:
 "Client session timed out, have not heard from server in 48865ms for
sessionid 0x44efbb91b5f0001, closing socket connection and attempting
reconnect".
So, I'm not sure what was the actual disconnection duration time, but it
could have been up to a minute.
We are working on finding the network issues root cause, but assuming
disconnections will always occur, are there any other options to overcome
this issues?



On Mon, Aug 10, 2015 at 11:18 PM, Erick Erickson 
wrote:

> I didn't see the zk timeout you set (just skimmed). But if your Zookeeper
> was
> down _very_ termporarily, it may suffice to up the ZK timeout. The default
> in the 10.4 time-frame (if I remember correctly) was 15 seconds which has
> proven to be too short in many circumstances.
>
> Of course if your ZK was down for minutest this wouldn't help.
>
> Best,
> Erick
>
> On Mon, Aug 10, 2015 at 1:06 PM, danny teichthal 
> wrote:
> > Hi Alexander ,
> > Thanks for your reply, I looked at the release notes.
> > There is one bug fix - SOLR-7503
> > <https://issues.apache.org/jira/browse/SOLR-7503> – register cores
> > asynchronously.
> > It may reduce the registration time since it is done on parallel, but
> > still, 3 minutes (leaderVoteWait) is a long time to recover from a few
> > seconds of disconnection.
> >
> > Except from that one I don't see any bug fix that addresses the same
> > problem.
> > I am able to reproduce it on 4.10.4 pretty easily, I will also try it
> with
> > 5.2.1 and see if it reproduces.
> >
> > Anyway, since migrating to 5.2.1 is not an option for me in the short
> term,
> > I'm left with the question if reducing leaderVoteWait may help here, and
> > what may be the consequences.
> > If i understand correctly, there might be a chance of losing updates that
> > were made on leader.
> > From my side it is a lot worse to lose availability for 3 minutes.
> >
> > I would really appreciate a feedback on this.
> >
> >
> >
> >
> > On Mon, Aug 10, 2015 at 6:55 PM, Alexandre Rafalovitch <
> arafa...@gmail.com>
> > wrote:
> >
> >> Did you look at release notes for Solr versions after your own?
> >>
> >> I am pretty sure some similar things were identified and/or resolved
> >> for 5.x. It may not help if you cannot migrate, but would at least
> >> give a confirmation and maybe workaround on what you are facing.
> >>
> >> Regards,
> >>Alex.
> >> 
> >> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 10 August 2015 at 11:37, danny teichthal 
> wrote:
> >> > Hi,
> >> > We are using Solr cloud with solr 4.10.4.
> >> > On the passed week we encountered a problem where all of our servers
> >> > disconnected from zookeeper cluster.
> >> > This might be ok, the problem is that after reconnecting to zookeeper
> it
> >> > looks like for every collection both replicas do not have a leader and
> >> are
> >> > stuck in some kind of a deadlock for a few minutes.
> >> >
> >> > From what we understand:
> >> > One of the replicas assume it ill be the leader and at some point
> >> starting
> >> > to wait on leaderVoteWait, which is by default 3 minutes.
> >> > The other replica is stuck on this part of code for a few minutes:
> >> >  at
> >> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)
> >> > at
> >> >
> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:921)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1521)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:392)
> >> >
> >> > Looks like replica 1 waits for a leader to be registered in the
> >> zookeeper,
> >> > but replica 2 is waiting for replica 1.
> >> >
> >>
> (org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp).
> >> >
> >> > We have 100 collections distributed in 3 pairs of Solr nodes. Each
> >> > collection has one shard with 2 replicas.
> >> > As I understand from code and logs, all the collections are being
> >

Re: Cluster down for long time after zookeeper disconnection

[danny teichthal]

1. Erik, thanks,  I agree that it is really serious, but I think that the 3
minutes on this case were not mandatory.
On my case it was a deadlock, which smells like some kind of bug.
One replica is waiting for other to come up, before it takes leadership,
while the other is waiting for the election results.
If I will be able to reproduce it on 5.2.1, is it legitimate to file a JIRA
issue for that?

2. Regarding session timeouts, there's something about configuration that I
don't understand.
 If zkClientTimeout is set to 30 seconds, how come see in the log that
session expired after ~50 seconds.
Maybe I have a mismatch between zookeeper and solr configuration?

3. Resuming the question of leaderVoteWait parameter, I have seen in a few
threads that it may be reduced to a minimum.
I'm not clear about the full meaning, but I understand that it is meant to
prevent lose of update on cluster startup.
Can anyone confirm/clarify that?




Links for leaderVoteWait:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201307.mbox/%3ccajt9wnhivirpn79kttcn8ekafevhhmqwkfl-+i16kbz0ogl...@mail.gmail.com%3E

http://qnalist.com/questions/4812859/waitforleadertoseedownstate-when-leader-is-down

Relevant part from My zookeeper conf:
tickTime=2000
initLimit=10
syncLimit=5



On Tue, Aug 11, 2015 at 1:06 AM, Erick Erickson 
wrote:

> Not that I know of. With ZK as the "one source of truth", dropping below
> quorum
> is Really Serious, so having to wait 3 minutes or so for action to be
> taken is the
> fallback.
>
> Best,
> Erick
>
> On Mon, Aug 10, 2015 at 1:34 PM, danny teichthal 
> wrote:
> > Erick, I assume you are referring to zkClientTimeout, it is set to 30
> > seconds. I also see these messages on Solr side:
> >  "Client session timed out, have not heard from server in 48865ms for
> > sessionid 0x44efbb91b5f0001, closing socket connection and attempting
> > reconnect".
> > So, I'm not sure what was the actual disconnection duration time, but it
> > could have been up to a minute.
> > We are working on finding the network issues root cause, but assuming
> > disconnections will always occur, are there any other options to overcome
> > this issues?
> >
> >
> >
> > On Mon, Aug 10, 2015 at 11:18 PM, Erick Erickson <
> erickerick...@gmail.com>
> > wrote:
> >
> >> I didn't see the zk timeout you set (just skimmed). But if your
> Zookeeper
> >> was
> >> down _very_ termporarily, it may suffice to up the ZK timeout. The
> default
> >> in the 10.4 time-frame (if I remember correctly) was 15 seconds which
> has
> >> proven to be too short in many circumstances.
> >>
> >> Of course if your ZK was down for minutest this wouldn't help.
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Aug 10, 2015 at 1:06 PM, danny teichthal 
> >> wrote:
> >> > Hi Alexander ,
> >> > Thanks for your reply, I looked at the release notes.
> >> > There is one bug fix - SOLR-7503
> >> > <https://issues.apache.org/jira/browse/SOLR-7503> – register cores
> >> > asynchronously.
> >> > It may reduce the registration time since it is done on parallel, but
> >> > still, 3 minutes (leaderVoteWait) is a long time to recover from a few
> >> > seconds of disconnection.
> >> >
> >> > Except from that one I don't see any bug fix that addresses the same
> >> > problem.
> >> > I am able to reproduce it on 4.10.4 pretty easily, I will also try it
> >> with
> >> > 5.2.1 and see if it reproduces.
> >> >
> >> > Anyway, since migrating to 5.2.1 is not an option for me in the short
> >> term,
> >> > I'm left with the question if reducing leaderVoteWait may help here,
> and
> >> > what may be the consequences.
> >> > If i understand correctly, there might be a chance of losing updates
> that
> >> > were made on leader.
> >> > From my side it is a lot worse to lose availability for 3 minutes.
> >> >
> >> > I would really appreciate a feedback on this.
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Aug 10, 2015 at 6:55 PM, Alexandre Rafalovitch <
> >> arafa...@gmail.com>
> >> > wrote:
> >> >
> >> >> Did you look at release notes for Solr versions after your own?
> >> >>
> >> >> I am pretty sure some similar things were identified and/or resolved
> >> >> for 5.x. It may not help if you cannot migrate

SolrCloud updates are slow on replica (DBQ?)

[danny teichthal]

Hi,


We are experiencing  some intermittent slowness on updates for one of our
collections.

We see user operations hanging on updates to SOLR via SolrJ client.

Every time in the period of the slowness we see something like this in the
log of the replica:

[org.apache.solr.update.UpdateHandler] Reordered DBQs detected.
Update=add{_version_=1504391336428568576,id=

2392581250002321} DBQs=[DBQ{version=1504391337298886656,q=level_2_id:12345}]

After  a while The DBQ is piling up and we see the list of DBQ growing.




At some point the time of updates is increase from 300 ms to 20 seconds and
then on the leader log I see read timeout exception and it initiates
recovery on the replica.

At that point all updates start to be very slow – from 20 seconds to 60
seconds. Especially updates with deletByQuery.

We are not sure if the DBQ is the cause or symptom. But, what does not make
sense to me is that the slowness is only on the replica side.

We suspect that the fact that the updates become slow on the replica cause
a timeout on the leader side and cause the recovery.


Would really appreciate any help on this.


Thanks,








Some info:

DBQ are sent as a separate update request from the add requests.


We currently use SolrCloud 4.9.0.

We have ~140 collections on  4 nodes – 1,2,3,4.

Each collection has a single shard with a leader and another replica.

~70 collections are on node 1 and 2 as leader and replica and the other
collections are on 3 and 4.



On each node there’s about 65GB of index with 25,000,000 documents.



This is our update handler, autoSoftCommit is set to 2 seconds, but there
may be manual soft commits coming from user operations from time to time:







1

12

true





   1000

   2000

Re: SolrCloud updates are slow on replica (DBQ?)

[danny teichthal]

Thanks Erik,
Actually, only lately we started to use autoSoftCommit lately because of
the performance warning, it was after reading the first link you provided.
Our application does frequent updates from batch and online requests.
Until now we issued softCommit after each user transaction finished. We
were able to reduce the vast majority of the manual commits and left a few
cases where the commit was essential for the screens not to fail.
Due to business requirements, 2 seconds are the maximum we can do for now.
But if you say that a few more seconds will make a difference we will try
to increase it.

As for GC, we continuously check the GC logs and can say for sure that it
is not the problem on our case.
Regarding cache - we don't use auto warm at all. For my small understanding
the cache is not that big, please correct me if I'm wrong.








2 more quetions, just for understanding:
1. What is the reason behind removing the maxdocs?  If I set no limit,
couldn't it explode the transaction log  in case of heavy indexing?
2. Do you think that the DBQ is causing a problem or is just indicating on
it, is there a problem with many DBQs?

We will probably start by removing the maxDocs and openSearcher=true.
The link about indexing performance also looks very relevant - I will read
it thoroughly.

Thanks again,





On Sun, Jun 21, 2015 at 6:29 PM, Erick Erickson 
wrote:

> The very first thing I would do is straighten out your commit strategy,
> they are _very_ aggressive. I'd guess you're also seeing warnings in
> the logs about "too many on deck searchers" or something like, or
> you've upped your max warming searchers in solrconfig.xml.
>
> Soft commits aren't free. They're less expensive than hard
> commits (openSearcher=true), but they're not free. Here's a long
> writeup on this:
>
> https://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> What I'd do:
> 1> remove maxDocs entirely
> 2> set openSearcher=false for your autoCommit
> 3> remove maxDocs from your autoSoftCommit
> 4> lengthen the soft commit as much as you can stand.
> 5> if you must have very short soft commits, consider
> turning off (or at least down) your caches in solrconfig.xml
> 6> stop issuing any kind of commits from the client. This is
>an anti-pattern except in very unusual circumstances and
>in your setup you see all the docs 2 seconds later anyway
>   so it is doing you no good and (maybe) active harm.
>
> If the problem persists, try looking at your garbage collection,
> you may well be hitting long GC pauses.
>
> Also note that there was a bottleneck in Solr prior to 5.2
> when replicas were present, see:
> http://lucidworks.com/blog/indexing-performance-solr-5-2-now-twice-fast/
>
> Best,
> Erick
>
> On Sun, Jun 21, 2015 at 7:14 AM, danny teichthal 
> wrote:
> > Hi,
> >
> >
> > We are experiencing  some intermittent slowness on updates for one of our
> > collections.
> >
> > We see user operations hanging on updates to SOLR via SolrJ client.
> >
> > Every time in the period of the slowness we see something like this in
> the
> > log of the replica:
> >
> > [org.apache.solr.update.UpdateHandler] Reordered DBQs detected.
> > Update=add{_version_=1504391336428568576,id=
> >
> > 2392581250002321}
> DBQs=[DBQ{version=1504391337298886656,q=level_2_id:12345}]
> >
> > After  a while The DBQ is piling up and we see the list of DBQ growing.
> >
> >
> >
> >
> > At some point the time of updates is increase from 300 ms to 20 seconds
> and
> > then on the leader log I see read timeout exception and it initiates
> > recovery on the replica.
> >
> > At that point all updates start to be very slow – from 20 seconds to 60
> > seconds. Especially updates with deletByQuery.
> >
> > We are not sure if the DBQ is the cause or symptom. But, what does not
> make
> > sense to me is that the slowness is only on the replica side.
> >
> > We suspect that the fact that the updates become slow on the replica
> cause
> > a timeout on the leader side and cause the recovery.
> >
> >
> > Would really appreciate any help on this.
> >
> >
> > Thanks,
> >
> >
> >
> >
> >
> >
> >
> >
> > Some info:
> >
> > DBQ are sent as a separate update request from the add requests.
> >
> >
> > We currently use SolrCloud 4.9.0.
> >
> > We have ~140 collections on  4 nodes – 1,2,3,4.
> >
> > Each collection has a single shard with a leader and another replica.
> >
> >

Re: SOLR war for SOLR 6

[danny teichthal]

If you are running on tomcat you will probably have a deployment problem.
On version 5.2.1 it worked fine for me, I manually packaged solr.war on
build time.
But, when trying to upgrade to Solr 5.5.1, I had problems with incompatible
servlet-api of Solr's jetty version and my tomcat servlert-api.
Solr code explicitly use some new methods that existed in the Jetty, but
not in my tomcat.
For me it was a no-go, from all the reasons that Shawn stated.




On Sat, Jun 18, 2016 at 12:26 AM, Shawn Heisey  wrote:

> On 6/16/2016 1:20 AM, Bharath Kumar wrote:
> > I was trying to generate a solr war out of the solr 6 source, but even
> > after i create the war, i was not able to get it deployed correctly on
> > jboss. Wanted to know if anyone was able to successfully generate solr
> > war and deploy it on tomcat or jboss? Really appreciate your help on
> > this.
>
> FYI: If you do this, you're running an unsupported configuration.
> You're on your own for both getting it working AND any problems that are
> related to the deployment rather than Solr itself.
>
> You actually don't need to create a war.  Just run "ant clean server" in
> the solr directory of the source code and then install the exploded
> webapp (found in server/solr-webapp/webapp) into the container.  There
> should be instructions available for how to install an exploded webapp
> into tomcat or jboss.  As already stated, you are on your own for
> finding and following those instructions, and if Solr doesn't deploy,
> you will need to talk to somebody who knows the container for help.
> Once they are sure you have the config for the container right, they may
> refer you back here ... but because it's an unsupported config, the
> amount of support we can offer is minimal.
>
> https://wiki.apache.org/solr/WhyNoWar
>
> If you want the admin UI to work when you install into a user-supplied
> container, then you must set the context path for the app to "/solr".
> The admin UI in 6.x will not work if you use another path, and that is
> not considered a bug, because the only supported container has the path
> hardcoded to /solr.
>
> Thanks,
> Shawn
>
>

SOLR-7036 - Faster method for group.facet - new patch for trunk

[danny teichthal]

Hi,
SOLR-7036 introduced a new faster method for group.facet, which uses
UnInvertedField.
It was patched for version 4.x.
Over the last week, my colleague uploaded a new patch that work against the
trunk.

We would really appreciate if anyone could take a look at it and give us
some feedback about it.
Full details and performance tests results were also added to the JIRA
issue.

We are willing to work at it and if possible backport it to an older branch.

Link:
https://issues.apache.org/jira/browse/SOLR-7036


Thanks in advance,

Re: Solr DeleteByQuery vs DeleteById

[danny teichthal]

Hi Bharath,
I'm no expert, but we had some major problems because of deleteByQuery ( in
short DBQ).
We ended up replacing all of our DBQ to delete by ids.

My suggestion is that if you don't realy need it - don't use it.
Especially in your case, since you already know the population of ids, it
is redundant to query for it.

I don't know how CDCR works, but we have a replication factor of 2 on our
SolrCloud cluster.
Since Solr 5.x , DBQ were stuck for a long while on the replicas, blocking
all updates.
It appears that on the replica side, there's an overhead of reordering and
executing the same DBQ over and over again, for consistency reasons.
It ends up buffering many delete by queries and blocks all updates.
In addition there's another defect on related slowness on DBQ - LUCENE-7049





On Tue, Aug 9, 2016 at 7:14 AM, Bharath Kumar 
wrote:

> Hi All,
>
> We are using SOLR 6.1 and i wanted to know which is better to use -
> deleteById or deleteByQuery?
>
> We have a program which deletes 10 documents every 5 minutes from the
> SOLR and we do it in a batch of 200 to delete those documents. For that we
> now use deleteById(List ids, 1) to delete.
> I wanted to know if we change it to deleteByQuery(query, 1) where the
> query is like this - (id:1 OR id:2 OR id:3 OR id:4). Will this have a
> performance impact?
>
> We use SOLR cloud with 3 SOLR nodes in the cluster and also we have a
> similar setup on the target site and we use Cross Data Center Replication
> to replicate from main site.
>
> Can you please let me know if using deleteByQuery will have any impact? I
> see it opens real time searcher on all the nodes in cluster.
>
> --
> Thanks & Regards,
> Bharath MV Kumar
>
> "Life is short, enjoy every moment of it"
>

Re: SOLR-7036 - a new Faster method for facet grouping

[danny teichthal]

Hi,
A reminder, in case anyone that is interested in performance of grouped
facet missed the older mail.
There's a new patch for improving grouped facet performance that works with
latest branch.

It uses the UIF method from JSON API.
Can be a first step for adding support for grouped facets on JSON API.

Please take a look at https://issues.apache.org/jira/browse/SOLR-7036
Comments and votes are welcome.





On Wed, Jul 27, 2016 at 11:31 AM, danny teichthal 
wrote:

> Hi,
> SOLR-7036 introduced a new faster method for group.facet, which uses
> UnInvertedField.
> It was patched for version 4.x.
> Over the last week, my colleague uploaded a new patch that work against
> the trunk.
>
> We would really appreciate if anyone could take a look at it and give us
> some feedback about it.
> Full details and performance tests results were also added to the JIRA
> issue.
>
> We are willing to work at it and if possible backport it to an older
> branch.
>
> Link:
> https://issues.apache.org/jira/browse/SOLR-7036
>
>
> Thanks in advance,
>

Re: Solr 6.1.0, zookeeper 3.4.8, Solrj and SolrCloud

[danny teichthal]

Hi,
Not sure if it is related, but could be - I see that you do this =
CloudSolrClient
solrClient = new
CloudSolrClient.Builder().withZkHost(zkHosts).build();
Are you creating a new client on each update?
If yes, pay attention that the Solr Client should be a singleton.

Regarding session timeout, what value did you set - zkClientTimeout?
The parameter - maxSessionTimeout, controls this time out on zookeeper side.
zkClientTimeout - controls your client timeout.


Session expiry can also be affected from:
1. Garbage collection on Solr node/Zookeeper.
2. Slow IO on disk.
3. Network latency.

You should check these metrics on your system at the time you got this
expiry to see if it might be related.
If your zkClientTimeout  is set to a small value in addition to one of the
factors above - you could get many of these exceptions.






On Thu, Aug 18, 2016 at 6:51 PM, Narayana B  wrote:

> Hi SolrTeam,
>
> I see session exipre and my solr index fails.
>
> please help me here, my infra details are shared below
>
> I have total 3 compute
> nodes[pcam-stg-app-02,pcam-stg-app-03,pcam-stg-app-04]
>
> 1) 3 nodes are running with zoo1, zoo2, zoo3 instances
>
> /apps/scm-core/zookeeper/zkData/zkData1/myid  value 1
> /apps/scm-core/zookeeper/zkData/zkData2/myid  value 2
> /apps/scm-core/zookeeper/zkData/zkData3/myid  value 3
>
> zoo1.cfg my setup
>
> tickTime=2000
> initLimit=5
> syncLimit=2
> dataDir=/apps/scm-core/zookeeper/zkData/zkData1
> clientPort=2181
> server.1=pcam-stg-app-01:2888:3888
> server.2=pcam-stg-app-02:2888:3888
> server.3=pcam-stg-app-03:2888:3888
> server.4=pcam-stg-app-04:2888:3888
> dataLogDir=/apps/scm-core/zookeeper/zkLogData/zkLogData1
> # Default 64M, changed to 128M, represented in KiloBytes
> preAllocSize=131072
> # Default : 10
> snapCount=100
> globalOutstandingLimit=1000
> maxClientCnxns=100
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=23
> minSessionTimeout=4
> maxSessionTimeout=30
>
> [zk: pcam-stg-app-02:2181(CONNECTED) 0] ls /
> [zookeeper, solr]
> [zk: pcam-stg-app-02:2181(CONNECTED) 1] ls /solr
> [configs, overseer, aliases.json, live_nodes, collections, overseer_elect,
> security.json, clusterstate.json]
>
>
>
> 2) 2 nodes are running solrcloud
> pcam-stg-app-03: solr port 8983, solr port 8984
> pcam-stg-app-04: solr port 8983, solr port 8984
>
>
> Config upload to zookeeper
>
> server/scripts/cloud-scripts/zkcli.sh -zkhost
> pcam-stg-app-02:2181,pcam-stg-app-03:2181,pcam-stg-app-04:2181/solr \
> -cmd upconfig -confname scdata -confdir
> /apps/scm-core/solr/solr-6.1.0/server/solr/configsets/data_
> driven_schema_configs/conf
>
> Collection creation url:
>
> http://pcam-stg-app-03:8983/solr/admin/collections?action=
> CREATE&name=scdata_test&numShards=2&replicationFactor=
> 2&maxShardsPerNode=2&createNodeSet=pcam-stg-app-03:
> 8983_solr,pcam-stg-app-03:8984_solr,pcam-stg-app-04:
> 8983_solr,pcam-stg-app-04:8984_solr&collection.configName=scdata
>
> solrj client
>
>
> String zkHosts =
> "pcam-stg-app-02:2181,pcam-stg-app-03:2181,pcam-stg-app-04:2181/solr";
> CloudSolrClient solrClient = new
> CloudSolrClient.Builder().withZkHost(zkHosts).build();
> solrClient.setDefaultCollection("scdata_test");
> solrClient.setParallelUpdates(true);
>
> List cpnSpendSavingsList = new ArrayList<>();
> i have done data setter to cpnSpendSavingsList
>
> solrClient.addBeans(cpnSpendSavingsList);
> solrClient.commit();
>
>
>
>
> SessionExpire Error for the collections
>
> Why this SessionExpire error comes when i start bulk insert/update to solr
>
>
> org.apache.solr.common.SolrException: Could not load collection from ZK:
> scdata_test
> at
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:1047)
> at
> org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(
> ZkStateReader.java:610)
> at
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(
> ClusterState.java:211)
> at
> org.apache.solr.common.cloud.ClusterState.hasCollection(
> ClusterState.java:113)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(
> CloudSolrClient.java:1239)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:961)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:934)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.
> java:106)
> at
> org.apache.solr.client.solrj.SolrClient.addBeans(SolrClient.java:357)
> at
> org.apache.solr.client.solrj.SolrClient.addBeans(SolrClient.java:329)
> at
> com.cisco.pcam.spark.stream.HiveDataProcessStream.main(
> HiveDataProcessStream.java:165)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
> 

Changing definition of id field

[danny teichthal]

Hi,
I have an "id" field that is defined on schema.xml with type long.
For some use cases the id that is indexed exceeds Max long limitation.
I thought about solving it by changing the id to type string.

For my surprise, by only changing the definition on schema.xml and
restarting Solr, I was able to read documents and update without the need
to re-index.

My questions are:
1. Is there another way to overcome the problem except from changing the id
to string?
2. Are there any downsides by changing id to string? I know that the
schema.xml example comes with id as string by default.
3. I want to avoid full indexing, is it valid just to change the field
type? Are there any known issues with this?There's a comment on the example
schema about not changing the type, mentioning document routing.



This is how the long and string types are defined:



I'm using Solr Cloud on version 5.2.1.
Multiple collections, every collection has only one shard.

Thanks,

Re: Changing definition of id field

[danny teichthal]

Thanks Shawn,
I understand that changing id to to string is not an option.

I have a limitation that our Solr cluster  is "live" during full indexing.
We have many updates and the index is incrementally updated.
There's no way to index everything on a side index and replace.
So, I'm trying to find a solution where users can still work not depending
on my full indexing.
We have a periodic indexing job that runs once in a few weeks.

I thought about another option:
1. Add a new field - "id_str".
2. Let periodic full indexing run.
3. Somewhere later during full change the uniqeKey from my "id" to "id_str".

May this work? Assuming that all ids are unique?



On Thu, Mar 9, 2017 at 5:14 PM, Shawn Heisey  wrote:

> On 3/9/2017 4:20 AM, danny teichthal wrote:
> > I have an "id" field that is defined on schema.xml with type long.
> > For some use cases the id that is indexed exceeds Max long limitation.
> > I thought about solving it by changing the id to type string.
> >
> > For my surprise, by only changing the definition on schema.xml and
> > restarting Solr, I was able to read documents and update without the need
> > to re-index.
>
> In general, changing the class used for a field won't work without a
> reindex.  You'll probably find that queries don't work to find the old
> documents, even if Solr is able to correctly show the field in results.
> I am very surprised that *anything* works with that change and an
> existing index.
>
> If "id" is your uniqueKey and you are relying on Solr's ability to
> replace existing documents using that field, you'll likely discover that
> reindexing existing documents will result in duplicates instead of
> deleting the old one.
>
> Thanks,
> Shawn
>
>

Question about autoAddReplicas

[Tseng, Danny]

Hi,

I create a collection of 2 shards with 1 replication factor and enable 
autoAddReplicas. Then I kill shard2 with 'kill -9' . The overseer asked the 
other solr node to create a new solr core and point to the dataDir of shard2. 
Unfortunately, the new core failed to come up because of pre-existing write 
lock. This is the new solr cluster state after fail over. Notice that shard2 
doesn't have dataDir assigned. Am I missing something?

[cid:image001.png@01D2A99B.712BB300]

RE: Question about autoAddReplicas

[Tseng, Danny]

g.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
   at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
   at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   at org.eclipse.jetty.server.Server.handle(Server.java:518)
   at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir 
'hdfs://psvrlxcdh5mmdev1.somewhere.com:8020/Test/LDM/psvrlxbdecdh1Cluster/solr/collection1/core_node2/data/index/'
 of core 'collection1_shard2_replica1' is already locked. The most likely cause 
is another Solr server (or another solr core in this server) also configured to 
use this directory; other possible causes may be specific to lockType: hdfs
   at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:658)
   at org.apache.solr.core.SolrCore.(SolrCore.java:850)
   ... 36 more


From: Tseng, Danny [mailto:dts...@informatica.com]
Sent: Thursday, March 30, 2017 9:35 PM
To: solr-user@lucene.apache.org
Subject: Question about autoAddReplicas

Hi,

I create a collection of 2 shards with 1 replication factor and enable 
autoAddReplicas. Then I kill shard2 with 'kill -9' . The overseer asked the 
other solr node to create a new solr core and point to the dataDir of shard2. 
Unfortunately, the new core failed to come up because of pre-existing write 
lock. This is the new solr cluster state after fail over. Notice that shard2 
doesn't have dataDir assigned. Am I missing something?

[cid:image001.png@01D2A99B.712BB300]

Nested documents, block join - re-indexing a single document upon update

[danny teichthal]

 Hi All,


 To make things short, I would like to use block joins, but to be able to
index each document on the block separately.

Is it possible?



In more details:



We have some nested parent-child structure where:

1.   Parent may have a single level of children

2.   Parent and child documents may be updated independently.

3.   We may want to search for parent by child info and vise versa.



At first we thought of managing the parent and child in different
documents, de-normalizing child data at parent level and parent data on
child level.

After reading *Mikhail*blog
http://blog.griddynamics.com/2013/09/solr-block-join-support.html, we
thought of using the block join for this purpose.



But, I got into a wall when trying to update a single child document.

For me, it's ok if SOLR will internally index the whole block, I just don't
want to fetch the whole hierarchy from DB for update.



I was trying to achieve this using atomic updates - since all the fields
must be stored anyway - if I send an atomic update on one of the children
with the _*root*_ field then there's no need to send the whole hierarchy.

But, when I try this method, I see that the child document is indeed
updated, but it's order is changed to be after the parent.



This is what I did:

1.   Change the root field to be stored - 

2.   Put attached docs on example\exampledocs.

3.   Run post.jar on parent-child.xml

4.   Run post.jar on update-child-atomic.xml.

5.   Now -
http://localhost:8983/solr/collection1/select?q=%7B!parent+which%3D%27type_s%3Aparent%27%7D%2BCOLOR_s%3ARed+%2BSIZE_s%3AXL%0A&wt=json&indent=true,
returns parent 10 as expected.

6.   But,
http://localhost:8983/solr/collection1/select?q={!parent+which%3D'type_s%3Aparent'}%2BCOLOR_s%3AGreen+%2BSIZE_s%3AXL%0A&wt=json&indent=true-
returns nothing.

7.   When searching *:* on Admin,   record with id=12 was updated with
'Green', but it is returned below the parent record.

8.



Thanks in advance.



In case the attachments does not work:

1st file to post:





  *:*

  



  10

  parent

  Nike

  

11

Red

XL

  

  

12

Blue

XL

  



  

  









2nd file:





  

  

12

Green

XL

10

  



  

  



  *:*
  

  10
  parent
  Nike
  
11
Red
XL
  
  
12
Blue
XL
  

  
  
  
  
  
12
Green
XL
		10
  

  
  

Re: Nested documents, block join - re-indexing a single document upon update

[danny teichthal]

Thanks Jack,
I understand that updating a single document on a block is currently not
supported.
But,  atomic update to a single document does not have to be in conflict
with block joins.

If I got it right from the documentation:
Currently, If a document is atomically  updated, SOLR finds the stored
document, and re index it, changing only the fields that were specified as
update="".

It looked intuitive that if we specify the _root_ while using atomic
update, SOLR will find the whole block by _root_, update the changed
document, and re-index the whole block.

Of course I cannot estimate the feasibility and effort of implementing it,
but it looks like a nice enhancement.



On Sun, Mar 16, 2014 at 4:09 PM, Jack Krupansky wrote:

> You stumbled upon the whole point of block join - that the documents are
> and must be managed as a block and not individually.
>
> -- Jack Krupansky
>
> From: danny teichthal
> Sent: Sunday, March 16, 2014 6:47 AM
> To: solr-user@lucene.apache.org
> Subject: Nested documents, block join - re-indexing a single document upon
> update
>
>
>
>
> Hi All,
>
>
>
>
> To make things short, I would like to use block joins, but to be able to
> index each document on the block separately.
>
> Is it possible?
>
>
>
> In more details:
>
>
>
> We have some nested parent-child structure where:
>
> 1.   Parent may have a single level of children
>
> 2.   Parent and child documents may be updated independently.
>
> 3.   We may want to search for parent by child info and vise versa.
>
>
>
> At first we thought of managing the parent and child in different
> documents, de-normalizing child data at parent level and parent data on
> child level.
>
> After reading Mikhail blog
> http://blog.griddynamics.com/2013/09/solr-block-join-support.html, we
> thought of using the block join for this purpose.
>
>
>
> But, I got into a wall when trying to update a single child document.
>
> For me, it's ok if SOLR will internally index the whole block, I just
> don't want to fetch the whole hierarchy from DB for update.
>
>
>
> I was trying to achieve this using atomic updates - since all the fields
> must be stored anyway - if I send an atomic update on one of the children
> with the _root_ field then there's no need to send the whole hierarchy.
>
> But, when I try this method, I see that the child document is indeed
> updated, but it's order is changed to be after the parent.
>
>
>
> This is what I did:
>
> 1.   Change the root field to be stored -  type="string" indexed="true" stored="true"/>
>
> 2.   Put attached docs on example\exampledocs.
>
> 3.   Run post.jar on parent-child.xml
>
> 4.   Run post.jar on update-child-atomic.xml.
>
> 5.   Now -
> http://localhost:8983/solr/collection1/select?q=%7B!parent+which%3D%27type_s%3Aparent%27%7D%2BCOLOR_s%3ARed+%2BSIZE_s%3AXL%0A&wt=json&indent=true,
> returns parent 10 as expected.
>
> 6.   But,
> http://localhost:8983/solr/collection1/select?q={!parent+which%3D'type_s%3Aparent'}%2BCOLOR_s%3AGreen+%2BSIZE_s%3AXL%0A&wt=json&indent=true-
>  returns nothing.
>
> 7.   When searching *:* on Admin,   record with id=12 was updated with
> 'Green', but it is returned below the parent record.
>
> 8.
>
>
>
> Thanks in advance.
>
>
>
> In case the attachments does not work:
>
> 1st file to post:
>
>
>
> 
>
>   *:*
>
>   
>
> 
>
>   10
>
>   parent
>
>   Nike
>
>   
>
> 11
>
> Red
>
> XL
>
>   
>
>   
>
> 12
>
> Blue
>
> XL
>
>   
>
> 
>
>   
>
>   
>
> 
>
>
>
>
>
>
>
> 2nd file:
>
>
>
> 
>
>   
>
>   
>
> 12
>
> Green
>
> XL
>
> 10
>
>   
>
> 
>
>   
>
>   
>
> 
>
>

Re: Nested documents, block join - re-indexing a single document upon update

[danny teichthal]

Thanks,
Indeed, the subject line was misleading.
Then I will file a new improvement request for "block atomic update
support".




On Tue, Mar 18, 2014 at 2:08 PM, Jack Krupansky wrote:

> That's a reasonable request and worth a Jira, but different from what you
> have specified in your subject line: "re-indexing a single document" - the
> entire block needs to be re-indexed.
>
> I suppose people might want a "block atomic update" - where multiple child
> documents as well as the parent document can be updated with one rewrite of
> the entire block, and maybe some way to delete individual child documents
> as well.
>
> -- Jack Krupansky
>
> -Original Message- From: danny teichthal
> Sent: Tuesday, March 18, 2014 3:58 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Nested documents, block join - re-indexing a single document
> upon update
>
>
> Thanks Jack,
> I understand that updating a single document on a block is currently not
> supported.
> But,  atomic update to a single document does not have to be in conflict
> with block joins.
>
> If I got it right from the documentation:
> Currently, If a document is atomically  updated, SOLR finds the stored
> document, and re index it, changing only the fields that were specified as
> update="".
>
> It looked intuitive that if we specify the _root_ while using atomic
> update, SOLR will find the whole block by _root_, update the changed
> document, and re-index the whole block.
>
> Of course I cannot estimate the feasibility and effort of implementing it,
> but it looks like a nice enhancement.
>
>
>
> On Sun, Mar 16, 2014 at 4:09 PM, Jack Krupansky 
> wrote:
>
>  You stumbled upon the whole point of block join - that the documents are
>> and must be managed as a block and not individually.
>>
>> -- Jack Krupansky
>>
>> From: danny teichthal
>> Sent: Sunday, March 16, 2014 6:47 AM
>> To: solr-user@lucene.apache.org
>> Subject: Nested documents, block join - re-indexing a single document upon
>> update
>>
>>
>>
>>
>> Hi All,
>>
>>
>>
>>
>> To make things short, I would like to use block joins, but to be able to
>> index each document on the block separately.
>>
>> Is it possible?
>>
>>
>>
>> In more details:
>>
>>
>>
>> We have some nested parent-child structure where:
>>
>> 1.   Parent may have a single level of children
>>
>> 2.   Parent and child documents may be updated independently.
>>
>> 3.   We may want to search for parent by child info and vise versa.
>>
>>
>>
>> At first we thought of managing the parent and child in different
>> documents, de-normalizing child data at parent level and parent data on
>> child level.
>>
>> After reading Mikhail blog
>> http://blog.griddynamics.com/2013/09/solr-block-join-support.html, we
>> thought of using the block join for this purpose.
>>
>>
>>
>> But, I got into a wall when trying to update a single child document.
>>
>> For me, it's ok if SOLR will internally index the whole block, I just
>> don't want to fetch the whole hierarchy from DB for update.
>>
>>
>>
>> I was trying to achieve this using atomic updates - since all the fields
>> must be stored anyway - if I send an atomic update on one of the children
>> with the _root_ field then there's no need to send the whole hierarchy.
>>
>> But, when I try this method, I see that the child document is indeed
>> updated, but it's order is changed to be after the parent.
>>
>>
>>
>> This is what I did:
>>
>> 1.   Change the root field to be stored - > type="string" indexed="true" stored="true"/>
>>
>> 2.   Put attached docs on example\exampledocs.
>>
>> 3.   Run post.jar on parent-child.xml
>>
>> 4.   Run post.jar on update-child-atomic.xml.
>>
>> 5.   Now -
>> http://localhost:8983/solr/collection1/select?q=%7B!
>> parent+which%3D%27type_s%3Aparent%27%7D%2BCOLOR_s%
>> 3ARed+%2BSIZE_s%3AXL%0A&wt=json&indent=true,
>> returns parent 10 as expected.
>>
>> 6.   But,
>> http://localhost:8983/solr/collection1/select?q={!parent+
>> which%3D'type_s%3Aparent'}%2BCOLOR_s%3AGreen+%2BSIZE_s%
>> 3AXL%0A&wt=json&indent=true-<http://localhost:8983/solr/collection1/select?q=%7B!parent+which%3D'type_s%3Aparent'%7D%2BCOLOR_s%3AGreen+%2BSIZE_s%3AXL%0A&wt=json&indent=true->returns
>>  nothing.
>>
>>
>> 7.   When searching *:* on Admin,   record with id=12 was updated with
>> 'Green', but it is returned below the parent record.
>>
>> 8.
>>
>>
>>
>> Thanks in advance.
>>
>>
>>
>> In case the attachments does not work:
>>
>> 1st file to post:
>>
>>
>>
>> 
>>
>>   *:*
>>
>>   
>>
>> 
>>
>>   10
>>
>>   parent
>>
>>   Nike
>>
>>   
>>
>> 11
>>
>> Red
>>
>> XL
>>
>>   
>>
>>   
>>
>> 12
>>
>> Blue
>>
>> XL
>>
>>   
>>
>> 
>>
>>   
>>
>>   
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> 2nd file:
>>
>>
>>
>> 
>>
>>   
>>
>>   
>>
>> 12
>>
>> Green
>>
>> XL
>>
>> 10
>>
>>   
>>
>> 
>>
>>   
>>
>>   
>>
>> 
>>
>>
>>
>

Solr performance: multiValued filed vs separate fields

[danny teichthal]

I wonder about performance difference of 2 indexing options: 1- multivalued
field 2- separate fields

The case is as follows: Each document has 100 “properties”: prop1..prop100.
The values are strings and there is no relation between different
properties. I would like to search by exact match on several properties by
known values (like ids). For example: search for all docs having
prop1=”blue” and prop6=”high”

I can choose to build the indexes in 1 of 2 ways: 1- the trivial way – 100
separate fields, 1 for each property, multiValued=false. the values are
just property values. 2- 1 field (named “properties”) multiValued=true. The
field will have 100 values: value1=”prop1:blue”.. value6=”high” etc

Is it correct to say that option1 will have much better performance in
searching? How about indexing performance?

Different update handlers for auto commit configuration

[danny teichthal]

Hi,
On our system we currently initiate a soft commit to SOLR after each
business transaction that initiate an update. Hard commits are automatic
each 2 minutes.
We want to limit the explicit commit and move to autoSoftCommit.

Because of business restrictions:
Online request should be available for searching after 2 seconds.
Update from batch jobs can be available after 10 seconds. (maybe more,
currently on discussion).
There are some transactions that must be available immediately.

Question
I thought about creating 3 different update handlers, each containing a
different autoSoftCommit configuration. Is this an acceptable solution, are
there any downsides in using multiple update handlers?

Thanks,

Re: Different update handlers for auto commit configuration

[danny teichthal]

Thanks for the reply Erik,
I thought that the auto commit is per update handler because they are
configured within the update handler tag.
>From a little debugging  on SOLR code and it looked as if the Commit
Tracker will schedule a soft commit on update if it gets commitwithin
value, and if not it will use the autosofcommit timeUpperBound. So,
according to my understanding, the autosoftcommit should be equivalent to
the commit within.
Anyway, it looks like the commitWithin is more flexible, so I'll try it.

Regarding  the requirement, the problem is that we are suppose to be "Near
real time".
Since we are already in production it's harder to change the requirements.

You are right about the caches and warming, the cache is not being used
that much and we cannot do warming at all.
 But, the trigger for the change is that on load time the commit frequency
increases and we get the performance warnings.
 We may have a few commits in the same second.
So, as a first step if we set it to 2 we will have a benefit.
In addition, on our big cores we have an average of commit per second on 24
hours, so still there's some benefit.



On Mon, Dec 1, 2014 at 12:42 AM, Erick Erickson 
wrote:

> Uhhhm, the soft/hard commit settings are global, not
> configured in each update handler.
>
> How are updates being done? Because if you're using SolrJ,
> you can just use the server.add(doclist, commitwithin) and it'll
> just be handled automatically.
>
> True, putting a 2 second commitwithin on an online update
> would pick up some batch updates that happened to come in, but
> that's probably OK.
>
> I'd also be sure that the 2 second requirement is real. Soft commits
> aren't as expensive as hard commits with openSearcher=true, but
> they aren't free either. At that fast a commit rate you probably won't
> get much benefit out of the top-level caches, and you'll be warming an
> awful lot.
>
> FWIW,
> Erick
>
> On Sun, Nov 30, 2014 at 12:32 PM, danny teichthal 
> wrote:
> > Hi,
> > On our system we currently initiate a soft commit to SOLR after each
> > business transaction that initiate an update. Hard commits are automatic
> > each 2 minutes.
> > We want to limit the explicit commit and move to autoSoftCommit.
> >
> > Because of business restrictions:
> > Online request should be available for searching after 2 seconds.
> > Update from batch jobs can be available after 10 seconds. (maybe more,
> > currently on discussion).
> > There are some transactions that must be available immediately.
> >
> > Question
> > I thought about creating 3 different update handlers, each containing a
> > different autoSoftCommit configuration. Is this an acceptable solution,
> are
> > there any downsides in using multiple update handlers?
> >
> > Thanks,
>

Re: Different update handlers for auto commit configuration

[danny teichthal]

Thanks for the clarification, I indeed mixed it with UpdateRequestHandler.

On Mon, Dec 1, 2014 at 11:24 PM, Chris Hostetter 
wrote:

>
> : I thought that the auto commit is per update handler because they are
> : configured within the update handler tag.
>
>  is not the same thing as a  that does
> updates.
>
> there can be many "Update request handlers" configured, but there is only
> ever one  in a SolrCore.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

CopyField exclude patterns

[danny teichthal]

Hi,
Is there a way to make some patterns to be excluded on the source of a
copyField?

We are using globs to copy all our text fields to some target field.
It looks something like this:


I would like a subset of the fields starting with "prefix_" to be excluded
and not copied to destination. (e.g. all fields with "prefix_abc_*" ).
Is there a way to do it on SOLR?

I couldn't find anything saying that it exists.

Thanks

Re: CopyField exclude patterns

[danny teichthal]

Alexander and Jack Thanks for the reply.
Looking at both, I think that the CloneFieldUpdateProcessor can do what I
need without having to implement a custom one.
By the way, Is there a performance penalty by update processor comparing to
copy Field?



On Mon, Feb 2, 2015 at 4:29 PM, Alexandre Rafalovitch 
wrote:

> Not on copyField,
>
> You can use UpdateRequestProcessor instead (
>
> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.html
> ).
>
> This allows to specify both inclusion and exclusion patterns.
>
> Regards,
>Alex.
> 
> Sign up for my Solr resources newsletter at http://www.solr-start.com/
>
>
> On 2 February 2015 at 02:53, danny teichthal  wrote:
> > Hi,
> > Is there a way to make some patterns to be excluded on the source of a
> > copyField?
> >
> > We are using globs to copy all our text fields to some target field.
> > It looks something like this:
> > 
> >
> > I would like a subset of the fields starting with "prefix_" to be
> excluded
> > and not copied to destination. (e.g. all fields with "prefix_abc_*" ).
> > Is there a way to do it on SOLR?
> >
> > I couldn't find anything saying that it exists.
> >
> > Thanks
>

Lengthy description is converted to hash symbols

[Danny Watari]

Hi, I have a field that is defined to be of type "text_en".  Occasionally, I
notice that lengthy strings are converted to hash symbols.  Here is a
snippet of my field type:


  


  
  


  




Here is an example of the field's value:
###


Any ideas why this might be happening?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Lengthy-description-is-converted-to-hash-symbols-tp4053338.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Lengthy description is converted to hash symbols

[Danny Watari]

Yes... the  is what I see in the admin console when I perform a
search for the document.  Currently, I am using solrj and the addBean()
method to update the core.  Whats strange is in our QA env, the document
indexed correctly.  But in prod, I see hash symbols and thus any user search
against that field fails to find the document.  Btw, I see no errors in the
logs!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Lengthy-description-is-converted-to-hash-symbols-tp4053338p4053505.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Lengthy description is converted to hash symbols

[Danny Watari]

I looked at the text via the admin analysis tool.  The text appeared to be
ok!  Unfortunately, the description is client data... so I can't post it
here, but I do not see any issues when running the analysis tool.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Lengthy-description-is-converted-to-hash-symbols-tp4053338p4053526.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Lengthy description is converted to hash symbols

[Danny Watari]

Here is a query that should return 2 documents... but it only returns 1.

/solr/m7779912/select?indent=on&version=2.2&q=description%3Agateway&fq=&start=0&rows=10&fl=description&qt=&wt=&explainOther=&hl.fl=

Oddly enough, the description of the two documents are exactly the same. 
Except one is indexed correctly and the other contains the hash symbols.

Btw, when the core was created, it was built from scratch via a pojo's and
the addBeans() method.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Lengthy-description-is-converted-to-hash-symbols-tp4053338p4053544.html
Sent from the Solr - User mailing list archive at Nabble.com.

Streaming .zip file using ContentStreamBase

[Danny Dvinov]

Hello,

Wondering if anyone could point me to the right way of streaming a .zip file: 
my goal is to stream a zipped version of the index. I zip up the index files I 
get from calling IndexCommit#getFileNames, and then attempt to stream using a 
custom handler with the following in handleRequestBody:

File targetZip = new File({pathToMyZipFile}));
ModifiableSolrParams params = new ModifiableSolrParams(req.getParams());
params.set(CommonParams.WT, "raw");
req.setParams(params);

ContentStreamBase content = new ContentStreamBase.FileStream(targetZip);
content.setContentType("application/zip");
rsp.add(RawResponseWriter.CONTENT, content);

I'm able to open the zip file (targetZip) and see the contents. However, I'm 
having trouble getting there on the receiving end, where I try reading the file 
by doing essentially the same as below:

BufferedReader reader = null;
BufferedWriter writer = null;
try {
InputStream is = content.getStream();
reader = new BufferedReader(new InputStreamReader(is, 
ContentStreamBase.DEFAULT_CHARSET));
File temp = File.createTempFile("myFile", ".zip");
writer = new BufferedWriter(new OutputStreamWriter(new 
FileOutputStream(temp), ContentStreamBase.DEFAULT_CHARSET));
String currentLine;
while((currentLine = reader.readLine()) != null) {
writer.write(currentLine);
writer.newLine();
}

writer.close();
} catch (Exception x) {
x.printStackTrace();
} finally {
IOUtils.closeQuietly(reader);
IOUtils.closeQuietly(writer);
}
Resulting "temp" file ends up being corrupted: "extra bytes at beginning or 
within zipfile " and "start of central directory not found; zipfile corrupt" 
error messages when opening the file. If I try reading it using ZipInputStream, 
I do see expected ZipEntry in it (in my case, the segments_# file, but I'm 
seeing the same issue when streaming a .zip containing a simple ".txt" file as 
well), but it has a size 0. Hoping my issue has something to do with the 
contentType/encoding, but haven't been able to pinpoint the exact problem 
(using "application/octet-stream" or "application/x-zip-compressed" doesn't 
seem to change things). Please let me know if you have ideas.

Thanks!

Issue streaming .zip file using ContentStreamBase

[Danny Dvinov]

Hello,

Wondering if anyone could point me to the right way of streaming a .zip file: 
my goal is to stream a zipped version of the index. I zip up the index files I 
get from calling IndexCommit#getFileNames, and then attempt to stream using a 
custom handler with the following in handleRequestBody:

File targetZip = new File({pathToMyZipFile}));
ModifiableSolrParams params = new ModifiableSolrParams(req.getParams());
params.set(CommonParams.WT, "raw");
req.setParams(params);

ContentStreamBase content = new ContentStreamBase.FileStream(targetZip);
content.setContentType("application/zip");
rsp.add(RawResponseWriter.CONTENT, content);

I'm able to open the zip file (targetZip) and see the contents. However, I'm 
having trouble getting there on the receiving end, where I try reading the file 
by doing essentially the same as below:

BufferedReader reader = null;
BufferedWriter writer = null;
try {
InputStream is = content.getStream();
reader = new BufferedReader(new InputStreamReader(is, 
ContentStreamBase.DEFAULT_CHARSET));
File temp = File.createTempFile("myFile", ".zip");
writer = new BufferedWriter(new OutputStreamWriter(new 
FileOutputStream(temp), ContentStreamBase.DEFAULT_CHARSET));
String currentLine;
while((currentLine = reader.readLine()) != null) {
writer.write(currentLine);
writer.newLine();
}

writer.close();
} catch (Exception x) {
x.printStackTrace();
} finally {
IOUtils.closeQuietly(reader);
IOUtils.closeQuietly(writer);
}

Resulting "temp" file ends up being corrupted. If I try reading it using 
ZipInputStream, I do see expected ZipEntry in it (in my case, the segments_# 
file), but it has a size 0. Hoping my issue has something to do with the 
contentType/encoding, but haven't been able to pinpoint the exact problem. 
Please let me know if you have ideas.

Thanks!

SOLR zookeeper connection timeout during startup is hardcoded to 10000ms

[Danny Shih]

Hi,

During startup in cloud mode, the SOLR zookeeper connection timeout appears to 
be hardcoded to 1000ms:
https://github.com/apache/lucene-solr/blob/5eab1c3c688a0d8db650c657567f197fb3dcf181/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ZkClientClusterStateProvider.java#L45

And it is not configurable via zkClientTimeout (solr.xml) or SOLR_WAIT_FOR_ZK 
(solr.in.sh).

Is there a way to configure this, and if not, should I open a bug?

Thanks,
Danny

Solr8 improvements to SolrCloud leader election

[Danny Shih]

Are there any significant (or not so significant) changes?  I have browsed the 
release notes and searched JIRA, but the latest news seems to be in 7.3 (where 
the old Leader-In-Recovery logic was replaced).

Context:
We are currently running Solr 7.4 in production.  In the past year, we’ve seen 
two cases where, during a rolling restart, one of the collections inexplicably 
ends up without a leader.  We have ways to mitigate – shutdown all instances 
then start up just one to have it be leader for everything, and we just learned 
about the FORCELEADER API.  However, when this condition has struck, reads from 
that collection do not work, so there’s customer impact.

We are planning an upgrade to Solr8.  The timing of that could be influenced by 
whether this upgrade could improve things in this particular area.

Thanks for any info!