[jira] [Commented] (SOLR-14963) Child "rows" param should apply per level
[
https://issues.apache.org/jira/browse/SOLR-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224061#comment-17224061
]
Bar Rotstein commented on SOLR-14963:
-
Perhaps having the limit applied to the immediate children only be in addition
to the default limit being _unlimited_ will give better flexibility, since
using the _nest_path_ field one can limit the which child docs will be returned?
This way one can limit the number of immediate children that will be returned
and control whether deeper children will be appended.
> Child "rows" param should apply per level
> -
>
> Key: SOLR-14963
> URL: https://issues.apache.org/jira/browse/SOLR-14963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> The {{[child rows=10]}} doc transformer "rows" param _should_ apply per
> parent, and it's documented this way: "The maximum number of child documents
> to be returned per parent document.". However, it is instead implemented as
> an overall limit as the child documents are processed in a depth-first order
> way. The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14963) Child "rows" param should apply per level
[
https://issues.apache.org/jira/browse/SOLR-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224121#comment-17224121
]
David Smiley commented on SOLR-14963:
-
Bar, I'm not sure if you are proposing something different than the discussion
above, or merely agreeing based on the fact that childFilter param adds
additional controls as well.
So two changes:
* change default of "limit" to unlimited
* change implementation of "limit" to match its documentation
> Child "rows" param should apply per level
> -
>
> Key: SOLR-14963
> URL: https://issues.apache.org/jira/browse/SOLR-14963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> The {{[child rows=10]}} doc transformer "rows" param _should_ apply per
> parent, and it's documented this way: "The maximum number of child documents
> to be returned per parent document.". However, it is instead implemented as
> an overall limit as the child documents are processed in a depth-first order
> way. The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14963) Child "rows" param should apply per level
[
https://issues.apache.org/jira/browse/SOLR-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224194#comment-17224194
]
Bar Rotstein commented on SOLR-14963:
-
Yes,
I do believe the default "limit" should be unlimited.
I was thinking about approaching the implementation of "limit" to only limit
the immediate child documents, and return all descendants of these immediate
children (if any) that match the childFilter.
Do you think a better approach would be if limit applied on a per level basis?
I was just thinking, usually if you do not wish to get descendants lower than a
certain level, you would probably use a different childFilter, so the
transformer might as well return all children of the immediate children that
made it through the limit and match childFilter.
WDYT, [~dsmiley] [~arafalov]?
> Child "rows" param should apply per level
> -
>
> Key: SOLR-14963
> URL: https://issues.apache.org/jira/browse/SOLR-14963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> The {{[child rows=10]}} doc transformer "rows" param _should_ apply per
> parent, and it's documented this way: "The maximum number of child documents
> to be returned per parent document.". However, it is instead implemented as
> an overall limit as the child documents are processed in a depth-first order
> way. The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14963) Child "rows" param should apply per level
[
https://issues.apache.org/jira/browse/SOLR-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224194#comment-17224194
]
Bar Rotstein edited comment on SOLR-14963 at 10/31/20, 9:50 PM:
Yes,
I do believe the default "limit" should be unlimited.
I was thinking about approaching the implementation of "limit" to only limit
the immediate child documents, and return all descendants of these immediate
children (if any) that match the childFilter.
Do you think a better approach would be if limit applied on a per level basis?
I was just thinking, usually if you do not wish to get descendants lower than a
certain level, you would probably use a different childFilter, so the
transformer might as well return all children of the immediate children that
made it through the limit and match childFilter.
I even made a rough implementation of this PR:
https://github.com/apache/lucene-solr/pull/2043
WDYT, [~dsmiley] [~arafalov]?
was (Author: brot):
Yes,
I do believe the default "limit" should be unlimited.
I was thinking about approaching the implementation of "limit" to only limit
the immediate child documents, and return all descendants of these immediate
children (if any) that match the childFilter.
Do you think a better approach would be if limit applied on a per level basis?
I was just thinking, usually if you do not wish to get descendants lower than a
certain level, you would probably use a different childFilter, so the
transformer might as well return all children of the immediate children that
made it through the limit and match childFilter.
WDYT, [~dsmiley] [~arafalov]?
> Child "rows" param should apply per level
> -
>
> Key: SOLR-14963
> URL: https://issues.apache.org/jira/browse/SOLR-14963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> The {{[child rows=10]}} doc transformer "rows" param _should_ apply per
> parent, and it's documented this way: "The maximum number of child documents
> to be returned per parent document.". However, it is instead implemented as
> an overall limit as the child documents are processed in a depth-first order
> way. The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14963) Child "rows" param should apply per level
[
https://issues.apache.org/jira/browse/SOLR-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224194#comment-17224194
]
Bar Rotstein edited comment on SOLR-14963 at 10/31/20, 11:28 PM:
-
Yes,
I do believe the default "limit" should be unlimited.
I was thinking about approaching the implementation of "limit" to only limit
the immediate child documents, and return all descendants of these immediate
children (if any) that match the childFilter.
Do you think a better approach would be if limit applied on a per level basis?
I was just thinking, usually if you do not wish to get descendants lower than a
certain level, you would probably use a different childFilter, so the
transformer might as well return all children of the immediate children that
made it through the limit and match childFilter.
I even made a rough implementation of this in this PR:
[https://github.com/apache/lucene-solr/pull/2043]
WDYT, [~dsmiley] [~arafalov]?
was (Author: brot):
Yes,
I do believe the default "limit" should be unlimited.
I was thinking about approaching the implementation of "limit" to only limit
the immediate child documents, and return all descendants of these immediate
children (if any) that match the childFilter.
Do you think a better approach would be if limit applied on a per level basis?
I was just thinking, usually if you do not wish to get descendants lower than a
certain level, you would probably use a different childFilter, so the
transformer might as well return all children of the immediate children that
made it through the limit and match childFilter.
I even made a rough implementation of this PR:
https://github.com/apache/lucene-solr/pull/2043
WDYT, [~dsmiley] [~arafalov]?
> Child "rows" param should apply per level
> -
>
> Key: SOLR-14963
> URL: https://issues.apache.org/jira/browse/SOLR-14963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> The {{[child rows=10]}} doc transformer "rows" param _should_ apply per
> parent, and it's documented this way: "The maximum number of child documents
> to be returned per parent document.". However, it is instead implemented as
> an overall limit as the child documents are processed in a depth-first order
> way. The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14969) Prevent creating multiple cores with the same name which leads to instabilities (race condition)
[
https://issues.apache.org/jira/browse/SOLR-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224208#comment-17224208
]
ASF subversion and git services commented on SOLR-14969:
Commit b27c8b90213cd388a9aa79deec7be753d10c9bb9 in lucene-solr's branch
refs/heads/branch_8x from Erick Erickson
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b27c8b9 ]
SOLR-14969: Prevent creating multiple cores with the same name which leads to
instabilities (race condition)
> Prevent creating multiple cores with the same name which leads to
> instabilities (race condition)
>
>
> Key: SOLR-14969
> URL: https://issues.apache.org/jira/browse/SOLR-14969
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: multicore
>Affects Versions: 8.6, 8.6.3
>Reporter: Andreas Hubold
>Assignee: Erick Erickson
>Priority: Major
> Attachments: CmCoreAdminHandler.java
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> CoreContainer#create does not correctly handle concurrent requests to create
> the same core. There's a race condition (see also existing TODO comment in
> the code), and CoreContainer#createFromDescriptor may be called subsequently
> for the same core name.
> The _second call_ then fails to create an IndexWriter, and exception handling
> causes an inconsistent CoreContainer state.
> {noformat}
> 2020-10-27 00:29:25.350 ERROR (qtp2029754983-24) [ ]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
> CREATEing SolrCore 'blueprint_acgqqafsogyc_comments': Unable to create core
> [blueprint_acgqqafsogyc_comments] Caused by: Lock held by this virtual
> machine: /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1312)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:95)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
> ...
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [blueprint_acgqqafsogyc_comments]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1408)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1273)
> ... 47 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.(SolrCore.java:1071)
> at org.apache.solr.core.SolrCore.(SolrCore.java:906)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1387)
> ... 48 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2184)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2308)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1130)
> at org.apache.solr.core.SolrCore.(SolrCore.java:1012)
> ... 50 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
> this virtual machine:
> /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at
> org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139)
> at
> org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
> at
> org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
> at
> org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:105)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:785)
> at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:126)
> at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
> at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:261)
> at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:135)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2145)
> {noformat}
> CoreContainer#createFromDescriptor removes the CoreDescriptor when handling
> this exception. The SolrCore created for the first successful call is still
> registered in SolrCores.cores, but now there's no corresponding
> CoreDescriptor for that name anymore.
> This inconsistency leads to subsequent NullPointerExceptions, for example
> when using CoreAdmin STATUS with the core name:
> CoreAdminOperation#getCoreStatus first gets the non-null SolrCore
> (cores.getCore
[jira] [Resolved] (SOLR-14969) Prevent creating multiple cores with the same name which leads to instabilities (race condition)
[
https://issues.apache.org/jira/browse/SOLR-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson resolved SOLR-14969.
---
Fix Version/s: 8.8
Resolution: Fixed
> Prevent creating multiple cores with the same name which leads to
> instabilities (race condition)
>
>
> Key: SOLR-14969
> URL: https://issues.apache.org/jira/browse/SOLR-14969
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: multicore
>Affects Versions: 8.6, 8.6.3
>Reporter: Andreas Hubold
>Assignee: Erick Erickson
>Priority: Major
> Fix For: 8.8
>
> Attachments: CmCoreAdminHandler.java
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> CoreContainer#create does not correctly handle concurrent requests to create
> the same core. There's a race condition (see also existing TODO comment in
> the code), and CoreContainer#createFromDescriptor may be called subsequently
> for the same core name.
> The _second call_ then fails to create an IndexWriter, and exception handling
> causes an inconsistent CoreContainer state.
> {noformat}
> 2020-10-27 00:29:25.350 ERROR (qtp2029754983-24) [ ]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
> CREATEing SolrCore 'blueprint_acgqqafsogyc_comments': Unable to create core
> [blueprint_acgqqafsogyc_comments] Caused by: Lock held by this virtual
> machine: /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1312)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:95)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
> ...
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [blueprint_acgqqafsogyc_comments]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1408)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1273)
> ... 47 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.(SolrCore.java:1071)
> at org.apache.solr.core.SolrCore.(SolrCore.java:906)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1387)
> ... 48 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2184)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2308)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1130)
> at org.apache.solr.core.SolrCore.(SolrCore.java:1012)
> ... 50 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
> this virtual machine:
> /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at
> org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139)
> at
> org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
> at
> org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
> at
> org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:105)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:785)
> at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:126)
> at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
> at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:261)
> at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:135)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2145)
> {noformat}
> CoreContainer#createFromDescriptor removes the CoreDescriptor when handling
> this exception. The SolrCore created for the first successful call is still
> registered in SolrCores.cores, but now there's no corresponding
> CoreDescriptor for that name anymore.
> This inconsistency leads to subsequent NullPointerExceptions, for example
> when using CoreAdmin STATUS with the core name:
> CoreAdminOperation#getCoreStatus first gets the non-null SolrCore
> (cores.getCore(cname)) but core.getInstancePath() throws an NPE, because the
> CoreDescriptor is not registered anymore:
> {noformat}
> 2020-10-27 00:29:25.353 INFO (qtp2029754983-19) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={core=blueprint_acgqqafsogyc_comments&action=STATUS&index
[jira] [Commented] (SOLR-14969) Prevent creating multiple cores with the same name which leads to instabilities (race condition)
[
https://issues.apache.org/jira/browse/SOLR-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224209#comment-17224209
]
ASF subversion and git services commented on SOLR-14969:
Commit 2c49c4a27d99316c9d962a50171f728de58bfa2a in lucene-solr's branch
refs/heads/master from Erick Erickson
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2c49c4a ]
SOLR-14969: Prevent creating multiple cores with the same name which leads to
instabilities (race condition)
> Prevent creating multiple cores with the same name which leads to
> instabilities (race condition)
>
>
> Key: SOLR-14969
> URL: https://issues.apache.org/jira/browse/SOLR-14969
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: multicore
>Affects Versions: 8.6, 8.6.3
>Reporter: Andreas Hubold
>Assignee: Erick Erickson
>Priority: Major
> Attachments: CmCoreAdminHandler.java
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> CoreContainer#create does not correctly handle concurrent requests to create
> the same core. There's a race condition (see also existing TODO comment in
> the code), and CoreContainer#createFromDescriptor may be called subsequently
> for the same core name.
> The _second call_ then fails to create an IndexWriter, and exception handling
> causes an inconsistent CoreContainer state.
> {noformat}
> 2020-10-27 00:29:25.350 ERROR (qtp2029754983-24) [ ]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
> CREATEing SolrCore 'blueprint_acgqqafsogyc_comments': Unable to create core
> [blueprint_acgqqafsogyc_comments] Caused by: Lock held by this virtual
> machine: /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1312)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:95)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
> ...
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [blueprint_acgqqafsogyc_comments]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1408)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1273)
> ... 47 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.(SolrCore.java:1071)
> at org.apache.solr.core.SolrCore.(SolrCore.java:906)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1387)
> ... 48 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2184)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2308)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1130)
> at org.apache.solr.core.SolrCore.(SolrCore.java:1012)
> ... 50 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by
> this virtual machine:
> /var/solr/data/blueprint_acgqqafsogyc_comments/data/index/write.lock
> at
> org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139)
> at
> org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
> at
> org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
> at
> org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:105)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:785)
> at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:126)
> at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
> at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:261)
> at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:135)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2145)
> {noformat}
> CoreContainer#createFromDescriptor removes the CoreDescriptor when handling
> this exception. The SolrCore created for the first successful call is still
> registered in SolrCores.cores, but now there's no corresponding
> CoreDescriptor for that name anymore.
> This inconsistency leads to subsequent NullPointerExceptions, for example
> when using CoreAdmin STATUS with the core name:
> CoreAdminOperation#getCoreStatus first gets the non-null SolrCore
> (cores.getCore(cn
[jira] [Commented] (LUCENE-9583) How should we expose VectorValues.RandomAccess?
[
https://issues.apache.org/jira/browse/LUCENE-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224226#comment-17224226
]
Tomoko Uchida commented on LUCENE-9583:
---
bq. I wonder if the graph approach really needs random access. For each node we
need to access the list of neighbors so this pattern doesn't require random
access because the list is already sorted by doc ids. So instead of adding
another interface I wonder what do you think of adding a reset method in
VectorValues ? For each node, the pattern to access would be to reset the
iterator first and then move it to the first neighbor. We can make optimization
internally to provide fast reset so that we don't need two implementations for
the first two approaches that we foresee ?
I would prefer this approach, encouraging forward only iteration and calling
reset() when needed, even if it could look a bit non-intuitive to implement
graph based aknn search.
I feel exposing public APIs for "random" access pattern needs more careful
decision and we should start from conservative ways (we already discussed about
that several times and couldn't reach an agreement so far, according to my
understanding).
> How should we expose VectorValues.RandomAccess?
> ---
>
> Key: LUCENE-9583
> URL: https://issues.apache.org/jira/browse/LUCENE-9583
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael Sokolov
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the newly-added {{VectorValues}} API, we have a {{RandomAccess}}
> sub-interface. [~jtibshirani] pointed out this is not needed by some
> vector-indexing strategies which can operate solely using a forward-iterator
> (it is needed by HNSW), and so in the interest of simplifying the public API
> we should not expose this internal detail (which by the way surfaces internal
> ordinals that are somewhat uninteresting outside the random access API).
> I looked into how to move this inside the HNSW-specific code and remembered
> that we do also currently make use of the RA API when merging vector fields
> over sorted indexes. Without it, we would need to load all vectors into RAM
> while flushing/merging, as we currently do in
> {{BinaryDocValuesWriter.BinaryDVs}}. I wonder if it's worth paying this cost
> for the simpler API.
> Another thing I noticed while reviewing this is that I moved the KNN
> {{search(float[] target, int topK, int fanout)}} method from {{VectorValues}}
> to {{VectorValues.RandomAccess}}. This I think we could move back, and
> handle the HNSW requirements for search elsewhere. I wonder if that would
> alleviate the major concern here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
