date:20201019

[GitHub] [lucene-solr] jtibshirani removed a comment on pull request #1930: LUCENE-9322: add VectorValues to new Lucene90 codec

2020-10-19 Thread GitBox



jtibshirani removed a comment on pull request #1930:
URL: https://github.com/apache/lucene-solr/pull/1930#issuecomment-711759394


   @msokolov I did a belated review and had some high-level thoughts. Hopefully 
post-commit reviews are still helpful in giving ideas for follow-ups.
   
   > It seems messy to have the ScoringFunction implementation in the main 
VectorValues interface API file. 
   
   It seems like we’re using `ScoreFunction` for two separate use cases:
   1. Configuring the ANN indexing + search strategy (to be introduced later). 
For example, HNSW needs to know up-front what similarity measure to use when 
building the graph.
   2. Providing convenience methods for users that want to incorporate vectors 
into scoring, outside of ANN.
   
   We could think about separating out these cases. Maybe `ScoreFunction` could 
stick closely to case 1 and avoid exposing score implementations. This fits 
well with its role of specifying the ANN index strategy, which might encompass 
other ANN approaches in the future like `KMEANS_EUCLIDEAN`. And for use case 2, 
we could introduce static utility methods like `dotProduct(float[] a, float[] 
b)` ? The ANN implementations would be free to make use of these methods if 
they prove helpful. If we go with this design, perhaps `ScoreFunction` would be 
renamed to something like `SearchStrategy`.
   
   Separately, I wonder if we need to expose a random access interface on the 
public `VectorValues`. When using vectors for scoring, iterator access seems 
sufficient. I’m having trouble thinking of a use for random access outside of 
ANN, and some ANN implementations don’t even need it -- for example most 
k-means and LSH approaches only need forward iteration. Would it make sense to 
keep the `VectorValues` surface area smaller and only provide iterator access? 
Random access could still be supported at a lower level, in a place that's 
available to HNSW. I’m not sure exactly how this would look, perhaps it'd be 
clearer once we see how HNSW will be integrated.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene-solr] jtibshirani commented on pull request #1930: LUCENE-9322: add VectorValues to new Lucene90 codec

2020-10-19 Thread GitBox



jtibshirani commented on pull request #1930:
URL: https://github.com/apache/lucene-solr/pull/1930#issuecomment-711759394


   @msokolov I did a belated review and had some high-level thoughts. Hopefully 
post-commit reviews are still helpful in giving ideas for follow-ups.
   
   > It seems messy to have the ScoringFunction implementation in the main 
VectorValues interface API file. 
   
   It seems like we’re using `ScoreFunction` for two separate use cases:
   1. Configuring the ANN indexing + search strategy (to be introduced later). 
For example, HNSW needs to know up-front what similarity measure to use when 
building the graph.
   2. Providing convenience methods for users that want to incorporate vectors 
into scoring, outside of ANN.
   
   We could think about separating out these cases. Maybe `ScoreFunction` could 
stick closely to case 1 and avoid exposing score implementations. This fits 
well with its role of specifying the ANN index strategy, which might encompass 
other ANN approaches in the future like `KMEANS_EUCLIDEAN`. And for use case 2, 
we could introduce static utility methods like `dotProduct(float[] a, float[] 
b)` ? The ANN implementations would be free to make use of these methods if 
they prove helpful. If we go with this design, perhaps `ScoreFunction` would be 
renamed to something like `SearchStrategy`.
   
   Separately, I wonder if we need to expose a random access interface on the 
public `VectorValues`. When using vectors for scoring, iterator access seems 
sufficient. I’m having trouble thinking of a use for random access outside of 
ANN, and some ANN implementations don’t even need it -- for example most 
k-means and LSH approaches only need forward iteration. Would it make sense to 
keep the `VectorValues` surface area smaller and only provide iterator access? 
Random access could still be supported at a lower level, in a place that's 
available to HNSW. I’m not sure exactly how this would look, perhaps it'd be 
clearer once we see how HNSW will be integrated.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API

2020-10-19 Thread Julie Tibshirani (Jira)



[ 
https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216503#comment-17216503
 ] 

Julie Tibshirani commented on LUCENE-9322:
--

Thank @msokolov for the great PR!! I left some belated thoughts on the PR 
regarding ScoreFunction.

> Discussing a unified vectors format API
> ---
>
> Key: LUCENE-9322
> URL: https://issues.apache.org/jira/browse/LUCENE-9322
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Julie Tibshirani
>Priority: Major
> Fix For: master (9.0)
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Two different approximate nearest neighbor approaches are currently being 
> developed, one based on HNSW (LUCENE-9004) and another based on coarse 
> quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to 
> handle vectors. In LUCENE-9136 we discussed the possibility of a unified API 
> that could support both approaches. The two ANN strategies give different 
> trade-offs in terms of speed, memory, and complexity, and it’s likely that 
> we’ll want to support both. Vector search is also an active research area, 
> and it would be great to be able to prototype and incorporate new approaches 
> without introducing more formats.
> To me it seems like a good time to begin discussing a unified API. The 
> prototype for coarse quantization 
> ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit 
> soon (this depends on everyone's feedback of course). The approach is simple 
> and shows solid search performance, as seen 
> [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326].
>  I think this API discussion is an important step in moving that 
> implementation forward.
> The goals of the API would be
> # Support for storing and retrieving individual float vectors.
> # Support for approximate nearest neighbor search -- given a query vector, 
> return the indexed vectors that are closest to it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Assigned] (SOLR-14944) solr metrics should remove "spins" references

2020-10-19 Thread Andrzej Bialecki (Jira)



 [ 
https://issues.apache.org/jira/browse/SOLR-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki reassigned SOLR-14944:
---

Assignee: Andrzej Bialecki

> solr metrics should remove "spins" references
> -
>
> Key: SOLR-14944
> URL: https://issues.apache.org/jira/browse/SOLR-14944
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (9.0)
>Reporter: Robert Muir
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: master (9.0)
>
>
> The lucene internal IOUtils.spins stuff was exposed in various ways here, in 
> order to not break stuff in LUCENE-9576 I simply wired these apis to 
> {{false}}, but they should probably be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (SOLR-14944) solr metrics should remove "spins" references

2020-10-19 Thread Andrzej Bialecki (Jira)



[ 
https://issues.apache.org/jira/browse/SOLR-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216524#comment-17216524
 ] 

Andrzej Bialecki commented on SOLR-14944:
-

+1 to remove this in 9.0. I'll also add a deprecation notice in 8x.

> solr metrics should remove "spins" references
> -
>
> Key: SOLR-14944
> URL: https://issues.apache.org/jira/browse/SOLR-14944
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (9.0)
>Reporter: Robert Muir
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: master (9.0)
>
>
> The lucene internal IOUtils.spins stuff was exposed in various ways here, in 
> order to not break stuff in LUCENE-9576 I simply wired these apis to 
> {{false}}, but they should probably be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async

2020-10-19 Thread Atri Sharma (Jira)



[ 
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216596#comment-17216596
 ] 

Atri Sharma commented on SOLR-14354:


Where are we with this issue? This is marked as a blocker for 8.7 – is that 
still the case?

> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
>  Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png, 
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n 
> equals to number of shards) to an executor. So each request will correspond 
> to a thread, after sending a request that thread basically do nothing just 
> waiting for response from other side. That thread will be swapped out and CPU 
> will try to handle another thread (this is called context switch, CPU will 
> save the context of the current thread and switch to another one). When some 
> data (not all) come back, that thread will be called to parsing these data, 
> then it will wait until more data come back. So there will be lots of context 
> switching in CPU. That is quite inefficient on using threads.Basically we 
> want less threads and most of them must busy all the time, because threads 
> are not free as well as context switching. That is the main idea behind 
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path";)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without 
> any block. Then when the client received the header from other side, it will 
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not 
> all response) from the data it will call {{onContent(buffer)}} listeners. 
> When everything finished it will call {{onComplete}} listeners. One main 
> thing that will must notice here is all listeners should finish quick, if the 
> listener block, all further data of that request won’t be handled until the 
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
>  Jetty HttpClient already provides several listeners, one of them is 
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
>   // Obtain the input stream on the response content
>   try (InputStream input = listener.getInputStream()) {
> // Read the response content
>   }
> } {code}
> In this case, there will be 2 thread
>  * one thread trying to read the response content from InputStream
>  * one thread (this is a short-live task) feeding content to above 
> InputStream whenever some byte[] is available. Note that if this thread 
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into 
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
>   executor.submit(() ->
> try (is) {
>   // Read the content from InputStream
> }
>   )
> }) {code}
>  The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time 
> since sending req is a very quick operation. With this operation, handling 
> threads won’t be spin up until first bytes are sent back. Notice that in this 
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it 
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
>   public void onComplete(Result result) {
> try {
>   byte

[jira] [Commented] (SOLR-14067) Deprecate StatelessScriptUpdateProcessor and disabled by default

2020-10-19 Thread Atri Sharma (Jira)



[ 
https://issues.apache.org/jira/browse/SOLR-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216598#comment-17216598
 ] 

Atri Sharma commented on SOLR-14067:


Is this a 8.7 blocker?

> Deprecate StatelessScriptUpdateProcessor and disabled by default
> 
>
> Key: SOLR-14067
> URL: https://issues.apache.org/jira/browse/SOLR-14067
> Project: Solr
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Priority: Blocker
> Fix For: 8.7
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We should eliminate all scripting capabilities within Solr. Let us start with 
> the StatelessScriptUpdateProcessor deprecation/removal.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (SOLR-13973) Deprecate Tika

2020-10-19 Thread Atri Sharma (Jira)



[ 
https://issues.apache.org/jira/browse/SOLR-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216597#comment-17216597
 ] 

Atri Sharma commented on SOLR-13973:


Do we want to treat this as. a blocker for 8.7?

> Deprecate Tika
> --
>
> Key: SOLR-13973
> URL: https://issues.apache.org/jira/browse/SOLR-13973
> Project: Solr
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Priority: Blocker
> Fix For: 8.7
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Solr's primary responsibility should be to focus on search and scalability. 
> Having to deal with the problems (CVEs) of Velocity, Tika etc. can slow us 
> down. I propose that we deprecate it going forward.
> Tika can be run outside Solr. Going forward, if someone wants to use these, 
> it should be possible to bring them into third party packages and installed 
> via package manager.
> Plan is to just to throw warnings in logs and add deprecation notes in 
> reference guide for now. Removal can be done in 9.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[jira] [Commented] (SOLR-11611) Starting Solr using solr.cmd fails under Windows, when the path contains a parenthesis character

2020-10-19 Thread Atri Sharma (Jira)



[ 
https://issues.apache.org/jira/browse/SOLR-11611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216599#comment-17216599
 ] 

Atri Sharma commented on SOLR-11611:


Whats the status of this? Is this a 8.7 blocker?

> Starting Solr using solr.cmd fails under Windows, when the path contains a 
> parenthesis character
> 
>
> Key: SOLR-11611
> URL: https://issues.apache.org/jira/browse/SOLR-11611
> Project: Solr
>  Issue Type: Bug
>  Components: scripts and tools, SolrCLI
>Affects Versions: 7.1, 7.4
> Environment: Microsoft Windows [Version 10.0.15063]
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
>Reporter: Jakob Furrer
>Priority: Blocker
> Fix For: 8.7
>
> Attachments: SOLR-11611.patch, SOLR-11611.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Starting Solr using solr.cli fails in Windows, when the path contains spaces.
> Use the following example to reproduce the error:
> {quote}C:\>c:
> C:\>cd "C:\Program Files (x86)\Company Name\ProductName Solr\bin"
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>dir
>  Volume in Laufwerk C: hat keine Bezeichnung.
>  Volumeseriennummer: 8207-3B8B
>  Verzeichnis von C:\Program Files (x86)\Company Name\ProductName Solr\bin
> 06.11.2017  15:52  .
> 06.11.2017  15:52  ..
> 06.11.2017  15:39  init.d
> 03.11.2017  17:32 8 209 post
> 03.11.2017  17:3275 963 solr
> 06.11.2017  14:2469 407 solr.cmd
>3 Datei(en),153 579 Bytes
>3 Verzeichnis(se), 51 191 619 584 Bytes frei
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>solr.cmd start
> *"\Company" kann syntaktisch an dieser Stelle nicht verarbeitet werden.*
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

[GitHub] [lucene-solr] shalinmangar opened a new pull request #2004: SOLR-14942: Reduce leader election time on node shutdown

2020-10-19 Thread GitBox

shalinmangar opened a new pull request #2004:
URL: https://github.com/apache/lucene-solr/pull/2004

This PR is based on the work of @CaoManhDat

# Description

The shutdown process waits for all replicas/cores to be closed before
removing the election node of the leader. This can take some time due to index
flush or merge activities on the leader cores and delays new leaders from being
elected. This leads to long gap of time during which indexing requests fail.

# Solution

This PR introduces a Phaser that registers and deregisters all update
requests. The shutdown process pauses indexing requests, waits for all
in-flight indexing requests to complete, removes election nodes (thus
triggering leader election) and then closes all replicas.

# Tests

Since this PR makes changes to Solr's shutdown process, it is difficult to
write a unit/integration tests inside Solr itself. This was tested by creating
a test harness outside of Solr. The steps were:
- Create a Collection with 1 shard, 3 replicas
- Do heavy indexing when all replicas are ACTIVE
- Shut down the leader node
- Measure the leaderless time (time during which there is no live/active
leader)

Before the patch | After the patch
-- | --
Shutting down time: 100ms Leader election: 46s | Shutting down time: 12s
Leader election: 3s
Shutting down time: 37s Leader election: 3s | Shutting down time: 31s Leader
election: 3s
Shutting down time: 200ms Leader election: 17s | Shutting down time: 100ms
Leader election: 3s
Shutting down time: 200ms Leader election: 27s | Shutting down time: 4s
Leader election: 3s

Since the patch introduces waiting for all updates to finish before giving
up the leadership so the shut down time gets increased on average, but also it
makes leader election more stable and improves consistency.

# Checklist

Please review the following and check all that apply:

- [X] I have reviewed the guidelines for [How to
Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms
to the standards described there to the best of my ability.
- [X] I have created a Jira issue and added the issue ID to my pull request
title.
- [ ] I have given Solr maintainers
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
to contribute to my PR branch. (optional but recommended)
- [X] I have developed this patch against the `master` branch.
- [X] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Ref
Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide)
(for Solr changes only).

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

1 2 >

1 - 100 of 124 matches

Mail list logo