jaepil opened a new pull request, #15827: URL: https://github.com/apache/lucene/pull/15827
## Summary Add `BayesianBM25Similarity` and `LogOddsConjunctionQuery` to Lucene, enabling probabilistic hybrid search that combines BM25 text retrieval with KNN vector search (or any other probability-producing scorers) through a principled Bayesian framework. Reference implementation: [cognica-io/bayesian-bm25](https://github.com/cognica-io/bayesian-bm25) **This PR adds 5 new files and modifies zero existing files.** ## Motivation Standard BM25 scores are unbounded and incomparable across queries. KNN cosine similarities live in [0, 1]. Combining these heterogeneous signals typically requires ad-hoc normalization (e.g., RRF, linear interpolation). This PR provides a theoretically grounded alternative: 1. **BayesianBM25Similarity** transforms BM25 scores into calibrated probabilities in (0, 1) 2. **LogOddsConjunctionQuery** combines multiple probability signals via log-odds, resolving the shrinkage problem of naive probabilistic AND Based on "Bayesian BM25" and "From Bayesian Inference to Neural Computation". ## Algorithm ### BayesianBM25Similarity Extends `Similarity`, following the same structure as `BM25Similarity` (same `LENGTH_TABLE[256]`, `idf()`, `avgFieldLength()`, `scorer()` pattern). ``` bm25 = standard BM25 score (identical formula) likelihood = sigmoid(alpha * (bm25 - beta)) tfPrior = clamp(base + range * min(1, freq / saturation), 0.1, 0.9) posterior = L * p / (L * p + (1 - L) * (1 - p)) ``` - **alpha/beta**: Auto-estimated from IDF weight when not specified (`alpha = 2/weight`, `beta = weight * 0.5`), or user-provided for manual calibration - **tf-only prior**: Uses term frequency alone, not document length ratio (see design decisions below) ### LogOddsConjunctionQuery Extends `Query`, following `DisjunctionMaxQuery` pattern. Inner `LogOddsConjunctionScorer` extends `DisjunctionScorer`. ``` for each matching sub-scorer: gatedLogit = softplus(logit(clamp(subScore))) logitSum = sum of gatedLogit values meanLogit = logitSum / n (n = total clauses, not just matching) scaledLogit = meanLogit * n^alpha (default alpha = 0.5, sqrt(n) scaling) score = sigmoid(scaledLogit) ``` Non-matching sub-scorers contribute 0 (neutral evidence). The `alpha` parameter controls confidence scaling: `alpha=0.5` (default) implements the sqrt(n) scaling law where confidence grows as the square root of agreeing signals. ## Design Decisions for Lucene ### tf-only prior (no norm_prior) The reference implementation uses a composite prior combining `tf_prior` and `norm_prior`. The `norm_prior` peaks at `doc_len_ratio = 0.5`, which means a medium-length document scores higher than a shorter one -- violating `SimScorer`'s monotonicity contract ("score must not increase when unsigned norm increases"). We use tf-only prior to preserve this contract. ### Softplus gating The reference `log_odds_conjunction()` applies no gating by default (`gating="none"`), with ReLU/Swish/GELU as options. In a Lucene disjunction, a non-matching sub-scorer contributes `logit(0.5) = 0` (neutral). Without gating, a matching sub-scorer with probability < 0.5 contributes a negative logit -- meaning a document is penalized for *containing* a query term. This conflates "absence of evidence" with "evidence of absence." - **ReLU** (`max(0, logit)`) fixes this but collapses all weak matches to exactly 0, losing ordering among them - **Swish/GELU** suppress negative logits but still produce negative values - **Softplus** (`log(1 + exp(logit))`) is always positive, preserving the ordering among weak matches while guaranteeing that any match contributes more than no match Softplus is hardcoded rather than configurable because the distinction between "no match" and "weak match" is structural in Lucene's disjunction model, not a tuning parameter. ### Double-precision posterior The posterior formula `L*p / (L*p + (1-L)*(1-p))` involves float multiplications whose rounding errors can invert the ordering between adjacent freq values (e.g., `score(0.99999994) > score(1.0)`), violating `SimScorer`'s freq monotonicity contract. The Bayesian transform computes in double precision internally, casting to float only at the final result. ### Default codec for KNN tests Vector tests use `TestUtil.getDefaultCodec()` to avoid randomized codecs that may produce unreliable HNSW graphs, matching the pattern in Lucene's `BaseKnnVectorQueryTestCase`. ## Integration with BooleanQuery `LogOddsConjunctionQuery` produces scores in (0, 1) and follows standard Lucene `Query` semantics, so it composes naturally with `BooleanQuery`: | Pattern | Lucene Expression | Semantics | |---------|-------------------|-----------| | Hybrid AND filter | `hybrid MUST, text FILTER` | Only text-matching docs scored by hybrid | | Hybrid OR hybrid | `hybrid1 SHOULD, hybrid2 SHOULD` | Match either hybrid, both boost score | | Hybrid NOT vector | `text MUST, knn MUST_NOT` | Exclude semantically similar docs from text results | | Multi-signal fusion | `LogOddsConjunction(text, knn1, knn2)` | Three-way text + vector + vector combination | ## New Files | File | Description | |------|-------------| | `BayesianBM25Similarity.java` | Similarity that transforms BM25 into calibrated probabilities | | `LogOddsConjunctionQuery.java` | Query combining probability scores via log-odds conjunction | | `LogOddsConjunctionScorer.java` | DisjunctionScorer with softplus-gated log-odds scoring | | `TestBayesianBM25Similarity.java` | Tests for BayesianBM25 | | `TestLogOddsConjunctionQuery.java` | Tests for LogOddsConjunction + hybrid search | ## Test Coverage ### BayesianBM25Similarity (12 tests) - **BaseSimilarityTestCase.testRandomScoring**: Randomized monotonicity, score/explanation consistency, numerical stability across random parameters, corpora, term stats, and norms - **BaseSimilarityTestCase.testBulkScore**: BulkSimScorer produces identical results to SimScorer - Parameter validation (illegal k1, b, alpha, beta) - Sigmoid numerical stability (large positive/negative inputs, zero) - Score range (0, 1) across all freq/norm combinations - tf-prior monotonicity in freq ### LogOddsConjunctionQuery (28 tests) **Core query behavior:** - Basic fusion, single/empty clause rewrite, alpha validation - equals/hashCode, toString, QueryUtils.check - Explanation correctness (CheckHits.verifyExplanation) - WAND max score correctness (CheckHits.checkTopScores) - Randomized top-docs validation **Hybrid search (BayesianBM25 + text):** - Multi-field hybrid (title + body) - More signals increase score (softplus anti-shrinkage property) - Full explanation pipeline (LogOdds -> BayesianBM25 posterior -> likelihood + tfPrior) - Randomized multi-field hybrid with WAND validation - BooleanQuery-wrapped hybrid (per-term multi-field LogOdds combined via BooleanQuery) **Vector + text hybrid:** - Text + vector combination: dual-signal doc ranks above single-signal - Both text-only and vector-only signals contribute to results - Multiple vector fields combined via LogOdds - Three-way hybrid: text + vector1 + vector2 **Boolean logic combinations:** - MUST (AND): text filter + hybrid scoring - SHOULD (OR): two independent hybrids combined - MUST_NOT (NOT): exclude vector-similar docs from text results - Vector AND vector: MUST conjunction of two KNN queries - Vector NOT: KNN MUST_NOT exclusion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
