[
https://issues.apache.org/jira/browse/SOLR-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-14595:
--------------------------------------
Attachment: SOLR-14595.patch
Status: Open (was: Open)
Ok, here's what seems to be happening:
* there is a "parent" facet using a sort that can cause documents to "move
down" after refinement (ex: "count asc" or "sum(field) asc") and a
"refine:true" child facet that sorts on "index asc" (which means it _may_ be
processed by "method:stream" if requested)
* during phase #1, the "parent" facet has some bucket "P_X" returned by both
shards
** depending on the shard, the list of buckets returned for the "child" facet
under parent bucket "P_X" can be very different, with little/no overlap (ie:
imagine all "odd" values<25 on shard1 and all "even" values<25 on shard2)
*** if we use "method:stream" then there is *no implicit (or user specified
explicit) overrequest considered* so fewer total buckets are returned by each
shard _and there is a *higher* chance the shard will return "more:true"_
**** ie: "limit:10" with odd/even<25 shards will return
1,3,5,7,9,11,13,15,17,19,*more:true* and 2,4,6,8,10,12,14,16,18,20,*more:true*
from our odd/even shards
*** if we use any other "method" then there is some overrequest done at the
shard level (either implicitly or via an explicit param) so we get more buckets
returnd by each shard _and there is a *lower* chance the shard will return
"more:true"_
**** ie: limit:10 will cause overrequest=5 and our odd/even<25 shards will
return 1,3,5,7,9,11,13,15,17,19,*21,23,more:false* and
2,4,6,8,10,12,14,16,18,20,*22,24,more:false* respectively
* during phase #2 "P_X" may not be a "competitive" parent bucket, and it may
not be considered for refinement
** but after refinement, other buckets may "move down", and "P_X" may "move
up" in the sorted list making it a (complete) bucket to return
** when considering some "child" facet bucket "C_Y", to determine if it is
"complete", the only consideration is if we there are any shards that did not
return bucket "C_Y" (under "P_X") but indicated that they have "more:true"
*** if phase#1 used "method:stream" and got "more:true" then a bucket "C_Y"
not returned by all shards will not be considered complete _even though the
"index asc" garuntees there's no chance it exists on any shard that didn't
return it_
*** *BUT* if phase#1 used some other "method" then it may have gotten
"more:false" from more shards then "method:stream" would have because of the
overrequest, and more buckets will be considered "complete"
...i've updated the attached patch to include the smallest possible test of
this discrepency can think of in TestJsonFacetRefinement
----
Depending on how you look at it, this is either:
* a FacetFieldProcessByTermEnum bug for not supporting/implementing
overrequest (either using the implicit hueristic or checking for an explicit
param) so we don't get consistent behavior between various processors during
"phase #1"
** "Fixing" this would be fairly straight forward, but would cause "wasted
work" (and network traffic) in the per-shard requests for the "common case"
usage of FacetFieldProcessByTermEnum
* a FacetRequestSortedMerger.isBucketComplete() bug, because it only considers
wether "shardHasMoreBuckets" but doesn't consider the sort order and wether or
not "having more buckets" has anything to do with if/why a particulr bucket
might be returned.
** making this method sophisticated enough to consider wether a bucket _might_
be returned by a shard that 'hasMoreBuckets" given what we know about the facet
sort would be pretty hard
This is such an esoteric edge case thta isn't likely to come up in any "real
world" usage of "method:stream" ... so I'm not sure how much effort it's really
worth to try and "fix" this (particularly since it doesn't cause any "incorrect
calculation" type bugs, it only causes fewer buckets to be returned) in a way
that won't hurt performance in the "typical" usage.
I'm going to think on this a little more, but i suspect that for now i'll just
modify TestCloudJSONFacetSKGEquiv to work around this (either by never testing
"index asc" combined with "refine:true", or perhaps only ever testing "index
asc" with "overrequest:0" ... not sure yet)
> json.facet subfacet 'sort:"index asc", refine:true' can return diff results
> using method:enum
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-14595
> URL: https://issues.apache.org/jira/browse/SOLR-14595
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Facet Module
> Reporter: Chris M. Hostetter
> Assignee: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-14595.patch, SOLR-14595.patch
>
>
> jenkins found a failing seed for TestCloudJSONFacetSKGEquiv that has nothing
> to do with SKG -- it shows that using {{method:enum}} can sometimes return
> different set of buckets then {{method:smart}} when computing a facet that
> uses {{"sort":"index asc", "refine":true}} _as a subfacet_ of some other
> facet.
> (In all the cases i've been able to trigger with more targetted testing, the
> "parent facet" needs to use a sort option that cause buckets to "sort worse"
> when more data is known about them -- ie: "count asc" or SKG -- but i haven't
> determined if that's actaully neccessary to trigger the fialure)
> original jenkins failure...
> {noformat}
> master jenkins (@ 541fc984e90) ...
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestCloudJSONFacetSKGEquiv -Dtests.method=testRandom
> -Dtests.seed=356C5A0B17DE491 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=en-KN -Dtests.timezone=Asia/Ho_Chi_Minh
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> [junit4] FAILURE 1.05s | TestCloudJSONFacetSKGEquiv.testRandom <<<
> [junit4] > Throwable #1: java.lang.AssertionError:
> rows=0&q=(field_7_multi_sds:19+OR+field_11_multi_sdsS:61+OR+field_8_multi_sdsS:45+OR+field_10_multi_sds:21+OR+field_2_multi_sdsS:28+OR+field_8_multi_sdsS:33+OR+field_10_multi_sds:54+OR+field_12_multi_ss:41)&fore=(field_5_multi_sdsS:48+OR+field_7_multi_sds:24+OR+field_13_multi_sds:61+OR+field_10_multi_sds:32+OR+field_9_multi_ss:45+OR+field_10_multi_sds:16+OR+field_11_multi_sdsS:28+OR+field_2_multi_sdsS:33+OR+field_8_multi_sdsS:43+OR+field_7_multi_sds:9)&back=(field_2_multi_sdsS:5+OR+field_9_multi_ss:16+OR+field_0_multi_ss:40+OR+field_0_multi_ss:16+OR+field_10_multi_sds:34+OR+field_10_multi_sds:58+OR+field_9_multi_ss:15+OR+field_1_multi_sds:44+OR+field_13_multi_sds:51+OR+field_10_multi_sds:21)&json.facet={"facet_1":{"method":"${method_val:smart}","limit":12,"sort":"count+asc","refine":true,"type":"terms","field":"field_12_multi_ss","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"},"facet_2":{"method":"${method_val:smart}","limit":1,"overrequest":38,"prefix":"2","sort":"index+asc","refine":true,"type":"terms","field":"field_3_multi_ss","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"},"facet_3":{"method":"${method_val:smart}","overrequest":0,"perSeg":false,"sort":"skg+desc","refine":true,"type":"terms","field":"field_8_multi_idsS","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"}}}}}}}}&_stateVer_=org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection:4
> ===> Mismatch: .facet_1.buckets[8][facet_2].buckets.length:1!=0 using
> method_val=enum
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
