This is an automated email from the ASF dual-hosted git repository.
eldenmoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 8e9c74edbba [doc](inverted-index) Document the norms index property
and its VARIANT default (#4146)
8e9c74edbba is described below
commit 8e9c74edbba5e27630e26394c83f0112a678299e
Author: lihangyu <[email protected]>
AuthorDate: Thu Oct 15 15:17:13 2026 +0800
[doc](inverted-index) Document the norms index property and its VARIANT
default (#4146)
Document the `norms` index property, storage cost, scoring limitations,
and the `inverted_index_skip_norms_for_variant` BE setting.
Update English and Chinese dev and 4.x docs; mark these settings as
available since 4.1.0 in 4.x.
---------
Co-authored-by: Claude Opus 5 <[email protected]>
---
.../sql-data-types/semi-structured/VARIANT.md | 14 +++++++++++++-
docs/table-design/index/inverted-index/scoring.md | 12 ++++++++++++
.../sql-data-types/semi-structured/VARIANT.md | 14 +++++++++++++-
.../current/table-design/index/inverted-index/scoring.md | 12 ++++++++++++
.../sql-data-types/semi-structured/VARIANT.md | 14 ++++++++++++++
.../table-design/index/inverted-index/scoring.md | 14 ++++++++++++++
.../sql-data-types/semi-structured/VARIANT.md | 14 ++++++++++++++
.../table-design/index/inverted-index/scoring.md | 14 ++++++++++++++
8 files changed, 106 insertions(+), 2 deletions(-)
diff --git
a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
index c2c531fbcf6..b6a4ffa5425 100644
--- a/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
+++ b/docs/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
@@ -670,7 +670,7 @@ SELECT * FROM tbl WHERE v['id_2'] MATCH 'Apache';
You can specify index properties for certain VARIANT subpaths, and even
configure both tokenized and non-tokenized inverted indexes for the same path.
Path-specific indexes require the path type to be declared via Schema Template.
```sql
--- Common properties: field_pattern (target path), analyzer, parser,
support_phrase, etc.
+-- Common properties: field_pattern (target path), analyzer, parser,
support_phrase, norms, etc.
CREATE TABLE IF NOT EXISTS tbl (
k BIGINT,
v VARIANT<'content' : STRING>,
@@ -697,6 +697,18 @@ SELECT * FROM tbl WHERE v['pattern_1'] MATCH 'Doris';
SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris';
```
+Tokenized indexes on VARIANT paths write BM25 norms by default, costing one
byte per row per path, including rows with no value for that path. For paths
used only for filtering, disable norms with `"norms" = "false"`. Without
`field_pattern`, this property applies to all subpaths covered by the index:
+
+```sql
+CREATE TABLE IF NOT EXISTS tbl (
+ k BIGINT,
+ v VARIANT<'body_*' : STRING>,
+ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english",
"field_pattern" = "body_*", "norms" = "false")
+);
+```
+
+Disabling norms does not affect `MATCH` filtering, but `score()` returns an
error when it reads a segment without norms. The BE config
`inverted_index_skip_norms_for_variant` disables norms for all VARIANT paths.
See [Record Length and the norms
Property](../../../../table-design/index/inverted-index/scoring.md#record-length-and-the-norms-property)
for details.
+
### When indexes don’t work
1. Type changes cause index loss: if a subpath changes to an incompatible type
(e.g., INT → JSONB), the index is lost. Fix by pinning types and indexes via
Schema Template.
diff --git a/docs/table-design/index/inverted-index/scoring.md
b/docs/table-design/index/inverted-index/scoring.md
index a74fae19b66..5c3fbad264e 100644
--- a/docs/table-design/index/inverted-index/scoring.md
+++ b/docs/table-design/index/inverted-index/scoring.md
@@ -111,6 +111,18 @@ Where:
When a query contains multiple terms, **the final score is the sum of the
scores of each term**.
+### Record Length and the `norms` Property
+
+`|d|` comes from norms, which cost one byte per row per indexed field,
including rows with no value for that field. Tokenized indexes write norms by
default (`"norms" = "true"`); non-tokenized indexes do not. For indexes used
only for filtering, set `"norms" = "false"` to save space:
+
+```sql
+INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english",
"norms" = "false")
+```
+
+VARIANT subpaths inherit the index's `norms` property, costing `rows × paths`
bytes. Setting the BE config `inverted_index_skip_norms_for_variant` (default
`false`, changeable at runtime) to `true` disables norms for all VARIANT path
indexes, overriding the index property.
+
+Disabling norms does not affect `MATCH_*` filtering, but `score()` returns an
error if it reads any segment without norms. These settings affect newly
written segments; existing segments remain unchanged until compaction rewrites
them. Upgrade all BEs to a version supporting these settings before changing
them.
+
## Interpreting the Results
Understanding the scoring results helps you use relevance ranking more
accurately:
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
index 1838e4cf5f6..d60acd8c678 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
@@ -670,7 +670,7 @@ SELECT * FROM tbl WHERE v['id_2'] MATCH 'Apache';
可为 VARIANT 的部分子列单独指定索引属性，甚至在同一路径上同时配置“分词与不分词”的两种倒排索引。指定路径索引需要在 Schema Template
中声明该路径的类型。
```sql
--- 常用属性：field_pattern（目标子路径）、analyzer、parser、support_phrase 等
+-- 常用属性：field_pattern（目标子路径）、analyzer、parser、support_phrase、norms 等
CREATE TABLE IF NOT EXISTS tbl (
k BIGINT,
v VARIANT<'content' : STRING>,
@@ -697,6 +697,18 @@ SELECT * FROM tbl WHERE v['pattern_1'] MATCH 'Doris';
SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris';
```
+VARIANT 路径上的分词索引默认写入 BM25 norms，每行每路径占 1 字节，包括该路径没有值的行。仅用于过滤的路径可通过 `"norms" =
"false"` 关闭；不带 `field_pattern` 时，该属性作用于索引覆盖的所有子路径：
+
+```sql
+CREATE TABLE IF NOT EXISTS tbl (
+ k BIGINT,
+ v VARIANT<'body_*' : STRING>,
+ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english",
"field_pattern" = "body_*", "norms" = "false")
+);
+```
+
+关闭 norms 不影响 `MATCH` 过滤，但 `score()` 读到缺少 norms 的段时会报错。BE 配置项
`inverted_index_skip_norms_for_variant` 可统一关闭 VARIANT 路径的 norms，详见[记录长度与 norms
属性](../../../../table-design/index/inverted-index/scoring.md#记录长度与-norms-属性)。
+
### 索引失效问题 {#when-indexes-dont-work}
1. 类型变更导致索引丢失：子列类型发生不兼容变更（如 INT→JSONB）会丢失索引。可通过 Schema Template 固定类型与索引。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md
index 7d0dbcefeed..488275fe6ef 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/scoring.md
@@ -111,6 +111,18 @@ avgdl = total_terms / total_rows
当查询包含多个词项时，**最终得分为各词项得分之和**。
+### 记录长度与 `norms` 属性
+
+`|d|` 来自 norms，每个索引字段每行占 1 字节，包括该字段没有值的行。分词索引默认写入 norms（`"norms" =
"true"`），非分词索引不写。仅用于过滤的索引可设置 `"norms" = "false"` 节省空间：
+
+```sql
+INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english",
"norms" = "false")
+```
+
+VARIANT 子路径继承索引的 `norms` 属性，空间开销为 `行数 × 路径数` 字节。BE 配置项
`inverted_index_skip_norms_for_variant`（默认 `false`，可动态修改）设为 `true` 时，所有 VARIANT
路径索引都不写 norms，优先于索引属性。
+
+关闭 norms 不影响 `MATCH_*` 过滤，但 `score()` 读到任一缺少 norms 的段时会报错。该属性和配置项只影响新写入的段；已有段在
compaction 重写前保持不变。修改前须将所有 BE 升级到支持它们的版本。
+
## 结果解读
理解打分结果有助于更准确地使用相关性排序：
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
index e842c7182d3..4ead77a8fef 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
@@ -305,6 +305,20 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris';
注意：2.1.7+ 仅支持 InvertedIndex V2 属性（文件更少、写入 IOPS 更低，适配存算分离）。2.1.8+ 不再支持离线 Build
Index 构建。
+自 Doris 4.1.0 起，可通过以下索引属性或 BE 配置项关闭 norms。
+
+VARIANT 路径上的分词索引默认写入 BM25 norms，每行每路径占 1 字节，包括该路径没有值的行。仅用于过滤的路径可通过 `"norms" =
"false"` 关闭；不带 `field_pattern` 时，该属性作用于索引覆盖的所有子路径：
+
+```sql
+CREATE TABLE IF NOT EXISTS tbl (
+ k BIGINT,
+ v VARIANT<'body_*' : STRING>,
+ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english",
"field_pattern" = "body_*", "norms" = "false")
+);
+```
+
+关闭 norms 不影响 `MATCH` 过滤，但 `score()` 读到缺少 norms 的段时会报错。BE 配置项
`inverted_index_skip_norms_for_variant` 可统一关闭 VARIANT 路径的 norms，详见[记录长度与 norms
属性](../../../../table-design/index/inverted-index/scoring.md#记录长度与-norms-属性)。
+
### 索引失效问题
1. 类型变更导致索引丢失：子列类型发生不兼容变更（如 INT→JSONB）会丢失索引。可通过 Schema Template 固定类型与索引。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/scoring.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/scoring.md
index 7d0dbcefeed..8002ea61262 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/scoring.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/scoring.md
@@ -111,6 +111,20 @@ avgdl = total_terms / total_rows
当查询包含多个词项时，**最终得分为各词项得分之和**。
+### 记录长度与 `norms` 属性
+
+`norms` 索引属性和 `inverted_index_skip_norms_for_variant` BE 配置项自 Doris 4.1.0 起支持。
+
+`|d|` 来自 norms，每个索引字段每行占 1 字节，包括该字段没有值的行。分词索引默认写入 norms（`"norms" =
"true"`），非分词索引不写。仅用于过滤的索引可设置 `"norms" = "false"` 节省空间：
+
+```sql
+INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english",
"norms" = "false")
+```
+
+VARIANT 子路径继承索引的 `norms` 属性，空间开销为 `行数 × 路径数` 字节。BE 配置项
`inverted_index_skip_norms_for_variant`（默认 `false`，可动态修改）设为 `true` 时，所有 VARIANT
路径索引都不写 norms，优先于索引属性。
+
+关闭 norms 不影响 `MATCH_*` 过滤，但 `score()` 读到任一缺少 norms 的段时会报错。该属性和配置项只影响新写入的段；已有段在
compaction 重写前保持不变。修改前须将所有 BE 升级到支持它们的版本。
+
## 结果解读
理解打分结果有助于更准确地使用相关性排序：
diff --git
a/versioned_docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
b/versioned_docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
index 9e4f4765293..55f58cdab4c 100644
---
a/versioned_docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
+++
b/versioned_docs/version-4.x/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md
@@ -305,6 +305,20 @@ SELECT * FROM tbl WHERE v['pattern_1'] = 'Doris';
Note: 2.1.7+ supports only InvertedIndex V2 properties (fewer files, lower
write IOPS; suitable for disaggregated storage/compute). 2.1.8+ removes offline
Build Index.
+Since Doris 4.1.0, norms can be disabled with the index property or BE config
below.
+
+Tokenized indexes on VARIANT paths write BM25 norms by default, costing one
byte per row per path, including rows with no value for that path. For paths
used only for filtering, disable norms with `"norms" = "false"`. Without
`field_pattern`, this property applies to all subpaths covered by the index:
+
+```sql
+CREATE TABLE IF NOT EXISTS tbl (
+ k BIGINT,
+ v VARIANT<'body_*' : STRING>,
+ INDEX idx_body(v) USING INVERTED PROPERTIES("parser" = "english",
"field_pattern" = "body_*", "norms" = "false")
+);
+```
+
+Disabling norms does not affect `MATCH` filtering, but `score()` returns an
error when it reads a segment without norms. The BE config
`inverted_index_skip_norms_for_variant` disables norms for all VARIANT paths.
See [Record Length and the norms
Property](../../../../table-design/index/inverted-index/scoring.md#record-length-and-the-norms-property)
for details.
+
### When indexes don’t work
1. Type changes cause index loss: if a subpath changes to an incompatible type
(e.g., INT → JSONB), the index is lost. Fix by pinning types and indexes via
Schema Template.
diff --git
a/versioned_docs/version-4.x/table-design/index/inverted-index/scoring.md
b/versioned_docs/version-4.x/table-design/index/inverted-index/scoring.md
index a74fae19b66..c631e539408 100644
--- a/versioned_docs/version-4.x/table-design/index/inverted-index/scoring.md
+++ b/versioned_docs/version-4.x/table-design/index/inverted-index/scoring.md
@@ -111,6 +111,20 @@ Where:
When a query contains multiple terms, **the final score is the sum of the
scores of each term**.
+### Record Length and the `norms` Property
+
+The `norms` index property and the `inverted_index_skip_norms_for_variant` BE
config are supported since Doris 4.1.0.
+
+`|d|` comes from norms, which cost one byte per row per indexed field,
including rows with no value for that field. Tokenized indexes write norms by
default (`"norms" = "true"`); non-tokenized indexes do not. For indexes used
only for filtering, set `"norms" = "false"` to save space:
+
+```sql
+INDEX idx_content(content) USING INVERTED PROPERTIES("parser" = "english",
"norms" = "false")
+```
+
+VARIANT subpaths inherit the index's `norms` property, costing `rows × paths`
bytes. Setting the BE config `inverted_index_skip_norms_for_variant` (default
`false`, changeable at runtime) to `true` disables norms for all VARIANT path
indexes, overriding the index property.
+
+Disabling norms does not affect `MATCH_*` filtering, but `score()` returns an
error if it reads any segment without norms. These settings affect newly
written segments; existing segments remain unchanged until compaction rewrites
them. Upgrade all BEs to a version supporting these settings before changing
them.
+
## Interpreting the Results
Understanding the scoring results helps you use relevance ranking more
accurately:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
