jaepil opened a new pull request, #15827:
URL: https://github.com/apache/lucene/pull/15827

   ## Summary
   
   Add `BayesianBM25Similarity` and `LogOddsConjunctionQuery` to Lucene, 
enabling probabilistic hybrid search that combines BM25 text retrieval with KNN 
vector search (or any other probability-producing scorers) through a principled 
Bayesian framework.
   
   Reference implementation: 
[cognica-io/bayesian-bm25](https://github.com/cognica-io/bayesian-bm25)
   
   **This PR adds 5 new files and modifies zero existing files.**
   
   ## Motivation
   
   Standard BM25 scores are unbounded and incomparable across queries. KNN 
cosine similarities live in [0, 1]. Combining these heterogeneous signals 
typically requires ad-hoc normalization (e.g., RRF, linear interpolation). This 
PR provides a theoretically grounded alternative:
   
   1. **BayesianBM25Similarity** transforms BM25 scores into calibrated 
probabilities in (0, 1)
   2. **LogOddsConjunctionQuery** combines multiple probability signals via 
log-odds, resolving the shrinkage problem of naive probabilistic AND
   
   Based on "Bayesian BM25" and "From Bayesian Inference to Neural Computation".
   
   ## Algorithm
   
   ### BayesianBM25Similarity
   
   Extends `Similarity`, following the same structure as `BM25Similarity` (same 
`LENGTH_TABLE[256]`, `idf()`, `avgFieldLength()`, `scorer()` pattern).
   
   ```
   bm25       = standard BM25 score (identical formula)
   likelihood = sigmoid(alpha * (bm25 - beta))
   tfPrior    = clamp(base + range * min(1, freq / saturation), 0.1, 0.9)
   posterior  = L * p / (L * p + (1 - L) * (1 - p))
   ```
   
   - **alpha/beta**: Auto-estimated from IDF weight when not specified (`alpha 
= 2/weight`, `beta = weight * 0.5`), or user-provided for manual calibration
   - **tf-only prior**: Uses term frequency alone, not document length ratio 
(see design decisions below)
   
   ### LogOddsConjunctionQuery
   
   Extends `Query`, following `DisjunctionMaxQuery` pattern. Inner 
`LogOddsConjunctionScorer` extends `DisjunctionScorer`.
   
   ```
   for each matching sub-scorer:
       gatedLogit = softplus(logit(clamp(subScore)))
   logitSum    = sum of gatedLogit values
   meanLogit   = logitSum / n          (n = total clauses, not just matching)
   scaledLogit = meanLogit * n^alpha   (default alpha = 0.5, sqrt(n) scaling)
   score       = sigmoid(scaledLogit)
   ```
   
   Non-matching sub-scorers contribute 0 (neutral evidence). The `alpha` 
parameter controls confidence scaling: `alpha=0.5` (default) implements the 
sqrt(n) scaling law where confidence grows as the square root of agreeing 
signals.
   
   ## Design Decisions for Lucene
   
   ### tf-only prior (no norm_prior)
   
   The reference implementation uses a composite prior combining `tf_prior` and 
`norm_prior`. The `norm_prior` peaks at `doc_len_ratio = 0.5`, which means a 
medium-length document scores higher than a shorter one -- violating 
`SimScorer`'s monotonicity contract ("score must not increase when unsigned 
norm increases"). We use tf-only prior to preserve this contract.
   
   ### Softplus gating
   
   The reference `log_odds_conjunction()` applies no gating by default 
(`gating="none"`), with ReLU/Swish/GELU as options. In a Lucene disjunction, a 
non-matching sub-scorer contributes `logit(0.5) = 0` (neutral). Without gating, 
a matching sub-scorer with probability < 0.5 contributes a negative logit -- 
meaning a document is penalized for *containing* a query term. This conflates 
"absence of evidence" with "evidence of absence."
   
   - **ReLU** (`max(0, logit)`) fixes this but collapses all weak matches to 
exactly 0, losing ordering among them
   - **Swish/GELU** suppress negative logits but still produce negative values
   - **Softplus** (`log(1 + exp(logit))`) is always positive, preserving the 
ordering among weak matches while guaranteeing that any match contributes more 
than no match
   
   Softplus is hardcoded rather than configurable because the distinction 
between "no match" and "weak match" is structural in Lucene's disjunction 
model, not a tuning parameter.
   
   ### Double-precision posterior
   
   The posterior formula `L*p / (L*p + (1-L)*(1-p))` involves float 
multiplications whose rounding errors can invert the ordering between adjacent 
freq values (e.g., `score(0.99999994) > score(1.0)`), violating `SimScorer`'s 
freq monotonicity contract. The Bayesian transform computes in double precision 
internally, casting to float only at the final result.
   
   ### Default codec for KNN tests
   
   Vector tests use `TestUtil.getDefaultCodec()` to avoid randomized codecs 
that may produce unreliable HNSW graphs, matching the pattern in Lucene's 
`BaseKnnVectorQueryTestCase`.
   
   ## Integration with BooleanQuery
   
   `LogOddsConjunctionQuery` produces scores in (0, 1) and follows standard 
Lucene `Query` semantics, so it composes naturally with `BooleanQuery`:
   
   | Pattern | Lucene Expression | Semantics |
   |---------|-------------------|-----------|
   | Hybrid AND filter | `hybrid MUST, text FILTER` | Only text-matching docs 
scored by hybrid |
   | Hybrid OR hybrid | `hybrid1 SHOULD, hybrid2 SHOULD` | Match either hybrid, 
both boost score |
   | Hybrid NOT vector | `text MUST, knn MUST_NOT` | Exclude semantically 
similar docs from text results |
   | Multi-signal fusion | `LogOddsConjunction(text, knn1, knn2)` | Three-way 
text + vector + vector combination |
   
   ## New Files
   
   | File | Description |
   |------|-------------|
   | `BayesianBM25Similarity.java` | Similarity that transforms BM25 into 
calibrated probabilities |
   | `LogOddsConjunctionQuery.java` | Query combining probability scores via 
log-odds conjunction |
   | `LogOddsConjunctionScorer.java` | DisjunctionScorer with softplus-gated 
log-odds scoring |
   | `TestBayesianBM25Similarity.java` | Tests for BayesianBM25 |
   | `TestLogOddsConjunctionQuery.java` | Tests for LogOddsConjunction + hybrid 
search |
   
   ## Test Coverage
   
   ### BayesianBM25Similarity (12 tests)
   
   - **BaseSimilarityTestCase.testRandomScoring**: Randomized monotonicity, 
score/explanation consistency, numerical stability across random parameters, 
corpora, term stats, and norms
   - **BaseSimilarityTestCase.testBulkScore**: BulkSimScorer produces identical 
results to SimScorer
   - Parameter validation (illegal k1, b, alpha, beta)
   - Sigmoid numerical stability (large positive/negative inputs, zero)
   - Score range (0, 1) across all freq/norm combinations
   - tf-prior monotonicity in freq
   
   ### LogOddsConjunctionQuery (28 tests)
   
   **Core query behavior:**
   - Basic fusion, single/empty clause rewrite, alpha validation
   - equals/hashCode, toString, QueryUtils.check
   - Explanation correctness (CheckHits.verifyExplanation)
   - WAND max score correctness (CheckHits.checkTopScores)
   - Randomized top-docs validation
   
   **Hybrid search (BayesianBM25 + text):**
   - Multi-field hybrid (title + body)
   - More signals increase score (softplus anti-shrinkage property)
   - Full explanation pipeline (LogOdds -> BayesianBM25 posterior -> likelihood 
+ tfPrior)
   - Randomized multi-field hybrid with WAND validation
   - BooleanQuery-wrapped hybrid (per-term multi-field LogOdds combined via 
BooleanQuery)
   
   **Vector + text hybrid:**
   - Text + vector combination: dual-signal doc ranks above single-signal
   - Both text-only and vector-only signals contribute to results
   - Multiple vector fields combined via LogOdds
   - Three-way hybrid: text + vector1 + vector2
   
   **Boolean logic combinations:**
   - MUST (AND): text filter + hybrid scoring
   - SHOULD (OR): two independent hybrids combined
   - MUST_NOT (NOT): exclude vector-similar docs from text results
   - Vector AND vector: MUST conjunction of two KNN queries
   - Vector NOT: KNN MUST_NOT exclusion


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to