This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new 04fb7a8e661 Add file_cache_statistics system table doc (#1021) 04fb7a8e661 is described below commit 04fb7a8e661eb5ed4eba30b1e80d5a6681e1597d Author: Mingyu Chen <morning...@163.com> AuthorDate: Sun Aug 25 07:58:10 2024 +0800 Add file_cache_statistics system table doc (#1021) --- docs/admin-manual/log-management/fe-log.md | 6 +- .../system-tables/file_cache_statistics.md | 110 ++++++++++++++++ .../admin-manual/system-tables/table_properties.md | 140 ++++++++++++--------- .../current/admin-manual/log-management/fe-log.md | 6 +- .../system-tables/file_cache_statistics.md | 113 +++++++++++++++++ .../admin-manual/system-tables/table_properties.md | 111 ++++++++++++++++ .../lakehouse/datalake-building/iceberg-build.md | 6 +- .../version-2.1.json | 4 + .../admin-manual/log-management/fe-log.md | 2 +- .../system-tables/file_cache_statistics.md | 113 +++++++++++++++++ .../admin-manual/system-tables/table_properties.md | 111 ++++++++++++++++ .../admin-manual/system-tables/tables.md | 64 ++++++++++ .../admin-manual/log-management/fe-log.md | 6 +- .../system-tables/file_cache_statistics.md | 113 +++++++++++++++++ .../admin-manual/system-tables/table_properties.md | 111 ++++++++++++++++ sidebars.json | 4 +- .../admin-manual/log-management/fe-log.md | 3 +- .../system-tables/file_cache_statistics.md | 110 ++++++++++++++++ .../admin-manual/system-tables/table_options.md | 69 ++++++++++ .../admin-manual/system-tables/table_properties.md | 110 ++++++++++++++++ .../admin-manual/system-tables/tables.md | 64 ++++++++++ .../admin-manual/log-management/fe-log.md | 6 +- .../system-tables/file_cache_statistics.md | 110 ++++++++++++++++ .../admin-manual/system-tables/table_options.md | 69 ++++++++++ .../admin-manual/system-tables/table_properties.md | 110 ++++++++++++++++ versioned_sidebars/version-2.1-sidebars.json | 9 ++ versioned_sidebars/version-3.0-sidebars.json | 4 +- 27 files changed, 1613 insertions(+), 71 deletions(-) diff --git a/docs/admin-manual/log-management/fe-log.md b/docs/admin-manual/log-management/fe-log.md index bb5807b96b0..943de526503 100644 --- a/docs/admin-manual/log-management/fe-log.md +++ b/docs/admin-manual/log-management/fe-log.md @@ -79,7 +79,11 @@ The following configuration items are configured in the `fe.conf` file. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | Module types in `fe.audit.log`. Default includes slow query, query, load, stream load. "Query" includes all DDL, DML, SQL operations. "Slow query" refers to operations that exceed the `qe_slow_log_ms` threshold. "Load" refers to Broker Load. "Stream load" refers to stream load operations. | | `qe_slow_log_ms` | 5000 | | When the execution time of DDL, DML, SQL statements exceeds this threshold, it will be separately recorded in the `slow_query` module of `fe.audit.log`. Default is 5000 ms. | | `audit_log_enable_compress` | false | true, false | Whether to enable compression for historical `fe.audit.log` logs. Default is off. When enabled, historical audit logs will be archived using gzip compression. | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | Output mode of FE logs. `NORMAL` is the default output mode, with synchronous output and location information. `BRIEF` mode is synchronous output without location information. `ASYNC` mode is asynchronous output without location information, with performance increasing in that order. | +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE log output mode, where `NORMAL` is the default output mode, log output is synchronous and includes location information. `ASYNC` is the default log output is asynchronous and includes location information. `BRIEF` mode is log output asynchronously but does not include location information. The performance of the three log output modes increases in sequence. | + +> Note +> +> Starting from version 3.0.2, the default value of `sys_log_mode` configuration is changed to `AYSNC`. ## Enable DEBUG Log diff --git a/docs/admin-manual/system-tables/file_cache_statistics.md b/docs/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..616084ae01d --- /dev/null +++ b/docs/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,110 @@ +--- +{ + "title": "file_cache_statistics", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Used to view the metric information related to data cache on each BE node. The metric information is sourced from the monitoring metrics related to BE's data cache. + +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +## Belongs to Database + +`information_schema` + +## Table Information + +| Column Name | Type | Description | +|---|---|---| +| BE_ID | BIGINT | BE node ID | +| BE_IP | VARCHAR(256) | BE node IP | +| CACHE_PATH | VARCHAR(256) | BE node cache path | +| METRIC_NAME | VARCHAR(256) | Metric name | +| METRIC_VALUE | DOUBLE | Metric value | + +> Different Doris version may have different metrics + +### 2.1.x Metrics + +> Only important metrics are listed. + +- `normal_queue_curr_elements` + + Number of File Blocks currently in the cache. + +- `normal_queue_max_elements` + + Maximum number of File Blocks allowed in the cache. + +- `normal_queue_curr_size` + + Current cache size. + +- `normal_queue_max_size` + + Maximum cache size allowed. + +- `hits_ratio` + + Overall cache hit ratio since BE startup. + +- `hits_ratio_5m` + + Cache hit ratio in the last 5 minutes. + +- `hits_ratio_1h` + + Cache hit ratio in the last 1 hour. + +### 3.0.x Metrics + +TODO + +## Examples + +1. Query all cache metrics + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. Query cache hit ratio and sort by hit ratio + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` diff --git a/docs/admin-manual/system-tables/table_properties.md b/docs/admin-manual/system-tables/table_properties.md index 402b6890e74..182a9fe3d6e 100644 --- a/docs/admin-manual/system-tables/table_properties.md +++ b/docs/admin-manual/system-tables/table_properties.md @@ -24,69 +24,87 @@ specific language governing permissions and limitations under the License. --> -## Overview +### Overview -`table_properties` is a built-in system table of doris, which is stored under the information_schema database. -You can view the current table properties information of each table through the `table_properties` system table. +Used to view the properties of tables (including internal and external tables). -## Database +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +### Database `information_schema` -## Table Information - -+----------------+-------------+------+-------+---------+-------+ -| Field | Type | Null | Key | Default | Extra | -+----------------+-------------+------+-------+---------+-------+ -| TABLE_CATALOG | varchar(64) | Yes | false | NULL | | -| TABLE_SCHEMA | varchar(64) | Yes | false | NULL | | -| TABLE_NAME | varchar(64) | Yes | false | NULL | | -| PROPERTY_NAME | text | Yes | false | NULL | | -| PROPERTY_VALUE | text | Yes | false | NULL | | -+----------------+-------------+------+-------+---------+-------+ - - -### Example -mysql> select * from information_schema.table_properties where table_name="rangetable" and table_schema="test_table_properties_db"; -+---------------+--------------------------+------------+------------------------------------------------+-------------------------+ -| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | -+---------------+--------------------------+------------+------------------------------------------------+-------------------------+ -| internal | test_table_properties_db | rangetable | min_load_replica_num | -1 | -| internal | test_table_properties_db | rangetable | data_sort.col_num | 6 | -| internal | test_table_properties_db | rangetable | group_commit_interval_ms | 10000 | -| internal | test_table_properties_db | rangetable | data_sort.sort_type | LEXICAL | -| internal | test_table_properties_db | rangetable | is_being_synced | false | -| internal | test_table_properties_db | rangetable | binlog.enable | false | -| internal | test_table_properties_db | rangetable | enable_mow_light_delete | false | -| internal | test_table_properties_db | rangetable | binlog.ttl_seconds | 86400 | -| internal | test_table_properties_db | rangetable | inverted_index_storage_format | V2 | -| internal | test_table_properties_db | rangetable | time_series_compaction_empty_rowsets_threshold | 5 | -| internal | test_table_properties_db | rangetable | default.replication_allocation | tag.location.default: 1 | -| internal | test_table_properties_db | rangetable | time_series_compaction_level_threshold | 1 | -| internal | test_table_properties_db | rangetable | time_series_compaction_time_threshold_seconds | 3600 | -| internal | test_table_properties_db | rangetable | storage_format | V2 | -| internal | test_table_properties_db | rangetable | store_row_column | false | -| internal | test_table_properties_db | rangetable | light_schema_change | true | -| internal | test_table_properties_db | rangetable | enable_unique_key_merge_on_write | false | -| internal | test_table_properties_db | rangetable | in_memory | false | -| internal | test_table_properties_db | rangetable | file_cache_ttl_seconds | 0 | -| internal | test_table_properties_db | rangetable | group_commit_data_bytes | 134217728 | -| internal | test_table_properties_db | rangetable | compaction_policy | size_based | -| internal | test_table_properties_db | rangetable | _auto_bucket | false | -| internal | test_table_properties_db | rangetable | binlog.max_history_nums | 9223372036854775807 | -| internal | test_table_properties_db | rangetable | time_series_compaction_file_count_threshold | 2000 | -| internal | test_table_properties_db | rangetable | skip_write_index_on_load | false | -| internal | test_table_properties_db | rangetable | disable_auto_compaction | false | -| internal | test_table_properties_db | rangetable | row_store_page_size | 16384 | -| internal | test_table_properties_db | rangetable | time_series_compaction_goal_size_mbytes | 1024 | -| internal | test_table_properties_db | rangetable | storage_medium | HDD | -| internal | test_table_properties_db | rangetable | enable_single_replica_compaction | false | -| internal | test_table_properties_db | rangetable | compression | LZ4F | -| internal | test_table_properties_db | rangetable | binlog.max_bytes | 9223372036854775807 | -+---------------+--------------------------+------------+------------------------------------------------+-------------------------+ - - - -### KeyWords - - table_options, information_schema +### Table Information + +| Column Name | Type | Description | +|-----------------|--------------|---------------------| +| TABLE_CATALOG | VARCHAR(64) | Catalog to which the table belongs | +| TABLE_SCHEMA | VARCHAR(64) | Database to which the table belongs | +| TABLE_NAME | VARCHAR(64) | Name of the table | +| PROPERTY_NAME | STRING | Name of the property | +| PROPERTY_VALUE | STRING | Value of the property | + +:::tip +For specific details on table properties, refer to the **Create Table** documentation. +::: + +### Examples + +1. Query all table properties + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. Query the default replication number + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/log-management/fe-log.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/log-management/fe-log.md index e740e4e4235..d98408aba7d 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/log-management/fe-log.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/log-management/fe-log.md @@ -79,7 +79,11 @@ under the License. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | `fe.audit.log` 中的模块类型。默认包括慢查询、查询、导入、stream load。其中“查询”只所有 DDL、DML、SQL 操作。“慢查询”指这些操作执行时间超过 `qe_slow_log_ms` 阈值的操作。“导入”指 Broker Load。“stream load”指 stream load 操作。 | | `qe_slow_log_ms` | 5000 | | 当 DDL、DML、SQL 语句的执行时间超过这个阈值后,会在 `fe.audit.log` 的 `slow_query` 模块中单独记录。默认 5000 ms | | `audit_log_enable_compress` | false | true, false | 是否开启历史 `fe.audit.log` 日志压缩。默认关闭。开启后,历史审计日志会使用 gzip 压缩归档 | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`BRIEF` 模式是日志同步输出但不包含位置信息。`ASYNC` 模式是日志异步输出且不包含位置信息,三种日志输出模式的性能依次递增 | +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`ASYNC` 默认是日志异步输出且包含位置信息。 `BRIEF` 模式是日志异步输出但不包含位置信息。三种日志输出模式的性能依次递增 | + +> 注: +> +> 1. 从 3.0.2 版本开始,`sys_log_mode` 配置默认改为 `AYSNC`。 ## 开启 DEBUG 日志 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/file_cache_statistics.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..7e4e30bc269 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,113 @@ +--- +{ + "title": "file_cache_statistics", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看各个 BE 节点 [数据缓存](../../lakehouse/filecache.md) 相关的指标信息。 + +指标信息来源于 BE 的数据缓存相关监控指标。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| BE_ID | BIGINT | BE 节点 ID | +| BE_IP | VARCHAR(256) | BE 节点 IP | +| CACHE_PATH | VARCHAR(256) | BE 节点缓存路径 | +| METRIC_NAME | VARCHAR(256) | 指标名称 | +| METRIC_VALUE | DOUBLE | 指标值 | + +> 不同 Doris 版本可能包含不同的指标信息。 + +### 2.1.x 版本指标说明 + +> 仅列举重要指标。 + +- `normal_queue_curr_elements` + + 当前缓存中 File Block 的个数。 + +- `normal_queue_max_elements` + + 缓存允许的 File Block 最大个数。 + +- `normal_queue_curr_size` + + 当前缓存大小 + +- `normal_queue_max_size` + + 缓存允许的最大大小 + +- `hits_ratio` + + 自 BE 启动后的缓存总命中率。 + +- `hits_ratio_5m` + + 最近 5 分钟的缓存命中率。 + +- `hits_ratio_1h` + + 最近 1 小时的缓存命中率。 + +### 3.0.x 版本指标说明 + +TODO + +## 示例 + +1. 查询所有缓存指标 + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. 查询缓存命中率,并按命中率排序 + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/table_properties.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/table_properties.md new file mode 100644 index 00000000000..44d2c853d14 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/system-tables/table_properties.md @@ -0,0 +1,111 @@ +--- +{ + "title": "table_properties", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看表(包括内表和外表)的属性信息。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| TABLE_CATALOG | VARCHAR(64 )| 表所属 Catalog | +| TABLE_SCHEMA | VARCHAR(64) | 表所属 Database | +| TABLE_NAME | VARCHAR(64) | 表名 | +| PROPERTY_NAME | STRING | 属性名称 | +| PROPERTY_VALUE | STRING | 属性值 | + +:::tip +关于表属性的具体减少,请参阅**建表**相关文档。 +::: + +## 示例 + +1. 查询所有表属性 + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. 查询默认副本数 + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/datalake-building/iceberg-build.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/datalake-building/iceberg-build.md index a35acff5e5f..79a352575f1 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/datalake-building/iceberg-build.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/datalake-building/iceberg-build.md @@ -36,6 +36,7 @@ under the License. 使用前,请先设置: <br /> set global enable_nereids_planner = true; +<br /> set global enable_fallback_to_original_planner = false; <br /> 从老版本升级上来的集群,这些参数可能有变化。 @@ -60,7 +61,10 @@ set global enable_fallback_to_original_planner = false; 上面主要演示了如何在 Apache Doris 中创建 HMS Iceberg Catalog。Apache Doris 目前支持多种类型的 Iceberg Catalog。更多配置,请参阅 [Iceberg Catalog](../datalake-analytics/iceberg.md) - 注意,如果需要通过 Apache Doris 通过 HMS Catalog 创建 Iceberg 表或写入数据,需要在 Catalog 属性中显式增加 `fs.defaultFS` 属性以及 `warehouse` 属性。如果创建 Catalog 仅用于查询,则这两个参数可以省略。 + 注意: + + - 如果需要通过 Apache Doris 通过 HMS Catalog 创建 Iceberg 表或写入数据,需要在 Catalog 属性中显式增加 `fs.defaultFS` 属性以及 `warehouse` 属性。如果创建 Catalog 仅用于查询,则这两个参数可以省略。 + - Hive Catalog 可以查询 Iceberg 表,但是不能在 Hive Catalog 中创建 Iceberg 表。 - 删除 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json index 897067e01d2..02a4c1a7065 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json @@ -199,6 +199,10 @@ "message": "配置管理", "description": "The label for category Config in sidebar docs" }, + "sidebar.docs.category.System Tables": { + "message": "系统表", + "description": "The label for category System Table in sidebar docs" + }, "sidebar.docs.category.FE OPEN API": { "message": "FE OPEN API", "description": "The label for category FE OPEN API in sidebar docs" diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/log-management/fe-log.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/log-management/fe-log.md index e740e4e4235..f3302e9e7c2 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/log-management/fe-log.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/log-management/fe-log.md @@ -79,7 +79,7 @@ under the License. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | `fe.audit.log` 中的模块类型。默认包括慢查询、查询、导入、stream load。其中“查询”只所有 DDL、DML、SQL 操作。“慢查询”指这些操作执行时间超过 `qe_slow_log_ms` 阈值的操作。“导入”指 Broker Load。“stream load”指 stream load 操作。 | | `qe_slow_log_ms` | 5000 | | 当 DDL、DML、SQL 语句的执行时间超过这个阈值后,会在 `fe.audit.log` 的 `slow_query` 模块中单独记录。默认 5000 ms | | `audit_log_enable_compress` | false | true, false | 是否开启历史 `fe.audit.log` 日志压缩。默认关闭。开启后,历史审计日志会使用 gzip 压缩归档 | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`BRIEF` 模式是日志同步输出但不包含位置信息。`ASYNC` 模式是日志异步输出且不包含位置信息,三种日志输出模式的性能依次递增 | +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`ASYNC` 默认是日志异步输出且包含位置信息。 `BRIEF` 模式是日志异步输出但不包含位置信息。三种日志输出模式的性能依次递增。| ## 开启 DEBUG 日志 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..7e4e30bc269 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,113 @@ +--- +{ + "title": "file_cache_statistics", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看各个 BE 节点 [数据缓存](../../lakehouse/filecache.md) 相关的指标信息。 + +指标信息来源于 BE 的数据缓存相关监控指标。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| BE_ID | BIGINT | BE 节点 ID | +| BE_IP | VARCHAR(256) | BE 节点 IP | +| CACHE_PATH | VARCHAR(256) | BE 节点缓存路径 | +| METRIC_NAME | VARCHAR(256) | 指标名称 | +| METRIC_VALUE | DOUBLE | 指标值 | + +> 不同 Doris 版本可能包含不同的指标信息。 + +### 2.1.x 版本指标说明 + +> 仅列举重要指标。 + +- `normal_queue_curr_elements` + + 当前缓存中 File Block 的个数。 + +- `normal_queue_max_elements` + + 缓存允许的 File Block 最大个数。 + +- `normal_queue_curr_size` + + 当前缓存大小 + +- `normal_queue_max_size` + + 缓存允许的最大大小 + +- `hits_ratio` + + 自 BE 启动后的缓存总命中率。 + +- `hits_ratio_5m` + + 最近 5 分钟的缓存命中率。 + +- `hits_ratio_1h` + + 最近 1 小时的缓存命中率。 + +### 3.0.x 版本指标说明 + +TODO + +## 示例 + +1. 查询所有缓存指标 + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. 查询缓存命中率,并按命中率排序 + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/table_properties.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/table_properties.md new file mode 100644 index 00000000000..44d2c853d14 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/table_properties.md @@ -0,0 +1,111 @@ +--- +{ + "title": "table_properties", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看表(包括内表和外表)的属性信息。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| TABLE_CATALOG | VARCHAR(64 )| 表所属 Catalog | +| TABLE_SCHEMA | VARCHAR(64) | 表所属 Database | +| TABLE_NAME | VARCHAR(64) | 表名 | +| PROPERTY_NAME | STRING | 属性名称 | +| PROPERTY_VALUE | STRING | 属性值 | + +:::tip +关于表属性的具体减少,请参阅**建表**相关文档。 +::: + +## 示例 + +1. 查询所有表属性 + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. 查询默认副本数 + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/tables.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/tables.md new file mode 100644 index 00000000000..c057272227c --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/system-tables/tables.md @@ -0,0 +1,64 @@ +--- +{ + "title": "tables", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +存储当前 Catalog 下所有的表信息。 + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| TABLE_CATALOG | VARCHAR(512) | 所属 Catalog | +| TABLE_SCHEMA | VARCHAR(64) | 所属 Database | +| TABLE_NAME | VARCHAR(64) | 表名称 | +| TABLE_TYPE | VARCHAR(64) | 表类型,包括:SYSTEM VIEW、VIEW、BASE TABLE | +| ENGINE | VARCHAR(64) | 表引擎类型 | +| VERSION | BIGINT | 无效值 | +| ROW_FORMAT | VARCHAR(10) | 无效值 | +| TABLE_ROWS | BIGINT | 表预估行数 | +| AVG_ROW_LENGTH | BIGINT | 表平均行大小 | +| DATA_LENGTH | BIGINT | 表预估大小 | +| MAX_DATA_LENGTH | BIGINT | 无效值 | +| INDEX_LENGTH | BIGINT | 无效值 | +| DATA_FREE | BIGINT | 无效值 | +| AUTO_INCREMENT | BIGINT | 无效值 | +| CREATE_TIME | DATETIME | 表创建时间 | +| UPDATE_TIME | DATETIME | 表更新时间 | +| CHECK_TIME | DATETIME | 无效值 | +| TABLE_COLLATION | VARCHAR(32) | 固定值:utf-8 | +| CHECKSUM | BIGINT | 无效值 | +| CREATE_OPTIONS | VARCHAR(255) | 无效值 | +| TABLE_COMMENT | VARCHAR(2048) | 表注释 | + +## 示例 + +无 + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/log-management/fe-log.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/log-management/fe-log.md index e740e4e4235..d98408aba7d 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/log-management/fe-log.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/log-management/fe-log.md @@ -79,7 +79,11 @@ under the License. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | `fe.audit.log` 中的模块类型。默认包括慢查询、查询、导入、stream load。其中“查询”只所有 DDL、DML、SQL 操作。“慢查询”指这些操作执行时间超过 `qe_slow_log_ms` 阈值的操作。“导入”指 Broker Load。“stream load”指 stream load 操作。 | | `qe_slow_log_ms` | 5000 | | 当 DDL、DML、SQL 语句的执行时间超过这个阈值后,会在 `fe.audit.log` 的 `slow_query` 模块中单独记录。默认 5000 ms | | `audit_log_enable_compress` | false | true, false | 是否开启历史 `fe.audit.log` 日志压缩。默认关闭。开启后,历史审计日志会使用 gzip 压缩归档 | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`BRIEF` 模式是日志同步输出但不包含位置信息。`ASYNC` 模式是日志异步输出且不包含位置信息,三种日志输出模式的性能依次递增 | +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE 日志的输出模式,其中 `NORMAL` 为默认的输出模式,日志同步输出且包含位置信息。`ASYNC` 默认是日志异步输出且包含位置信息。 `BRIEF` 模式是日志异步输出但不包含位置信息。三种日志输出模式的性能依次递增 | + +> 注: +> +> 1. 从 3.0.2 版本开始,`sys_log_mode` 配置默认改为 `AYSNC`。 ## 开启 DEBUG 日志 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..7e4e30bc269 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,113 @@ +--- +{ + "title": "file_cache_statistics", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看各个 BE 节点 [数据缓存](../../lakehouse/filecache.md) 相关的指标信息。 + +指标信息来源于 BE 的数据缓存相关监控指标。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| BE_ID | BIGINT | BE 节点 ID | +| BE_IP | VARCHAR(256) | BE 节点 IP | +| CACHE_PATH | VARCHAR(256) | BE 节点缓存路径 | +| METRIC_NAME | VARCHAR(256) | 指标名称 | +| METRIC_VALUE | DOUBLE | 指标值 | + +> 不同 Doris 版本可能包含不同的指标信息。 + +### 2.1.x 版本指标说明 + +> 仅列举重要指标。 + +- `normal_queue_curr_elements` + + 当前缓存中 File Block 的个数。 + +- `normal_queue_max_elements` + + 缓存允许的 File Block 最大个数。 + +- `normal_queue_curr_size` + + 当前缓存大小 + +- `normal_queue_max_size` + + 缓存允许的最大大小 + +- `hits_ratio` + + 自 BE 启动后的缓存总命中率。 + +- `hits_ratio_5m` + + 最近 5 分钟的缓存命中率。 + +- `hits_ratio_1h` + + 最近 1 小时的缓存命中率。 + +### 3.0.x 版本指标说明 + +TODO + +## 示例 + +1. 查询所有缓存指标 + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. 查询缓存命中率,并按命中率排序 + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/table_properties.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/table_properties.md new file mode 100644 index 00000000000..44d2c853d14 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/system-tables/table_properties.md @@ -0,0 +1,111 @@ +--- +{ + "title": "table_properties", + "language": "zh-CN" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 概述 + +用于查看表(包括内表和外表)的属性信息。 + +:::tip +该系统表自 2.1.6 和 3.0.2 版本支持。 +::: + +## 所属数据库 + +`information_schema` + +## 表信息 + +| 列名 | 类型 | 说明 | +|---|---|---| +| TABLE_CATALOG | VARCHAR(64 )| 表所属 Catalog | +| TABLE_SCHEMA | VARCHAR(64) | 表所属 Database | +| TABLE_NAME | VARCHAR(64) | 表名 | +| PROPERTY_NAME | STRING | 属性名称 | +| PROPERTY_VALUE | STRING | 属性值 | + +:::tip +关于表属性的具体减少,请参阅**建表**相关文档。 +::: + +## 示例 + +1. 查询所有表属性 + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. 查询默认副本数 + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` + diff --git a/sidebars.json b/sidebars.json index 1f4128c6dab..f1a9f8821e5 100644 --- a/sidebars.json +++ b/sidebars.json @@ -500,7 +500,9 @@ "type": "category", "label": "System Tables", "items": [ - "admin-manual/system-tables/tables" + "admin-manual/system-tables/file_cache_statistics", + "admin-manual/system-tables/tables", + "admin-manual/system-tables/table_properties" ] }, "admin-manual/audit-plugin", diff --git a/versioned_docs/version-2.1/admin-manual/log-management/fe-log.md b/versioned_docs/version-2.1/admin-manual/log-management/fe-log.md index bb5807b96b0..8671e7cfbc9 100644 --- a/versioned_docs/version-2.1/admin-manual/log-management/fe-log.md +++ b/versioned_docs/version-2.1/admin-manual/log-management/fe-log.md @@ -79,8 +79,7 @@ The following configuration items are configured in the `fe.conf` file. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | Module types in `fe.audit.log`. Default includes slow query, query, load, stream load. "Query" includes all DDL, DML, SQL operations. "Slow query" refers to operations that exceed the `qe_slow_log_ms` threshold. "Load" refers to Broker Load. "Stream load" refers to stream load operations. | | `qe_slow_log_ms` | 5000 | | When the execution time of DDL, DML, SQL statements exceeds this threshold, it will be separately recorded in the `slow_query` module of `fe.audit.log`. Default is 5000 ms. | | `audit_log_enable_compress` | false | true, false | Whether to enable compression for historical `fe.audit.log` logs. Default is off. When enabled, historical audit logs will be archived using gzip compression. | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | Output mode of FE logs. `NORMAL` is the default output mode, with synchronous output and location information. `BRIEF` mode is synchronous output without location information. `ASYNC` mode is asynchronous output without location information, with performance increasing in that order. | - +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE log output mode, where `NORMAL` is the default output mode, log output is synchronous and includes location information. `ASYNC` is the default log output is asynchronous and includes location information. `BRIEF` mode is log output asynchronously but does not include location information. The performance of the three log output modes increases in sequence. | ## Enable DEBUG Log diff --git a/versioned_docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md b/versioned_docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..616084ae01d --- /dev/null +++ b/versioned_docs/version-2.1/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,110 @@ +--- +{ + "title": "file_cache_statistics", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Used to view the metric information related to data cache on each BE node. The metric information is sourced from the monitoring metrics related to BE's data cache. + +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +## Belongs to Database + +`information_schema` + +## Table Information + +| Column Name | Type | Description | +|---|---|---| +| BE_ID | BIGINT | BE node ID | +| BE_IP | VARCHAR(256) | BE node IP | +| CACHE_PATH | VARCHAR(256) | BE node cache path | +| METRIC_NAME | VARCHAR(256) | Metric name | +| METRIC_VALUE | DOUBLE | Metric value | + +> Different Doris version may have different metrics + +### 2.1.x Metrics + +> Only important metrics are listed. + +- `normal_queue_curr_elements` + + Number of File Blocks currently in the cache. + +- `normal_queue_max_elements` + + Maximum number of File Blocks allowed in the cache. + +- `normal_queue_curr_size` + + Current cache size. + +- `normal_queue_max_size` + + Maximum cache size allowed. + +- `hits_ratio` + + Overall cache hit ratio since BE startup. + +- `hits_ratio_5m` + + Cache hit ratio in the last 5 minutes. + +- `hits_ratio_1h` + + Cache hit ratio in the last 1 hour. + +### 3.0.x Metrics + +TODO + +## Examples + +1. Query all cache metrics + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. Query cache hit ratio and sort by hit ratio + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` diff --git a/versioned_docs/version-2.1/admin-manual/system-tables/table_options.md b/versioned_docs/version-2.1/admin-manual/system-tables/table_options.md new file mode 100644 index 00000000000..34958b077ae --- /dev/null +++ b/versioned_docs/version-2.1/admin-manual/system-tables/table_options.md @@ -0,0 +1,69 @@ +--- +{ + "title": "table_options", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +`table_options` is a built-in system table of doris, which is stored under the information_schema database. You can view the current table options information of each table through the `table_options` system table. + +## Database + +`information_schema` + +## Table Information + ++-----------------+-------------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++-----------------+-------------+------+-------+---------+-------+ +| TABLE_CATALOG | VARCHAR(64) | Yes | false | NULL | | +| TABLE_SCHEMA | VARCHAR(64) | Yes | false | NULL | | +| TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +| TABLE_MODEL | TEXT | Yes | false | NULL | | +| TABLE_MODEL_KEY | TEXT | Yes | false | NULL | | +| DISTRIBUTE_KEY | TEXT | Yes | false | NULL | | +| DISTRIBUTE_TYPE | TEXT | Yes | false | NULL | | +| BUCKETS_NUM | INT | Yes | false | NULL | | +| PARTITION_NUM | INT | Yes | false | NULL | | ++-----------------+-------------+------+-------+---------+-------+ + + +### Example +mysql> select * from information_schema.table_options where table_schema="test_table_options_db"; ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ +| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_MODEL | TABLE_MODEL_KEY | DISTRIBUTE_KEY | DISTRIBUTE_TYPE | BUCKETS_NUM | PARTITION_NUM | ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ +| internal | test_table_options_db | randomtable | DUP | user_id,date,timestamp | RANDOM | RANDOM | 16 | 1 | +| internal | test_table_options_db | test_row_column_page_size1 | DUP | aaa | aaa | HASH | 1 | 1 | +| internal | test_table_options_db | aggregate_table | AGG | user_id,date,city,age,sex | user_id | HASH | 1 | 1 | +| internal | test_table_options_db | rangetable | AGG | user_id,date,timestamp,city,age,sex | user_id | HASH | 8 | 3 | +| internal | test_table_options_db | unique_table | UNI | user_id,username | user_id | HASH | 1 | 1 | +| internal | test_table_options_db | listtable | AGG | user_id,date,timestamp,city,age,sex | user_id | HASH | 16 | 3 | +| internal | test_table_options_db | duplicate_table | DUP | timestamp,type,error_code | type | HASH | 1 | 1 | ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ + + +### KeyWords + + table_options, information_schema diff --git a/versioned_docs/version-2.1/admin-manual/system-tables/table_properties.md b/versioned_docs/version-2.1/admin-manual/system-tables/table_properties.md new file mode 100644 index 00000000000..182a9fe3d6e --- /dev/null +++ b/versioned_docs/version-2.1/admin-manual/system-tables/table_properties.md @@ -0,0 +1,110 @@ +--- +{ + "title": "table_properties", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +### Overview + +Used to view the properties of tables (including internal and external tables). + +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +### Database + +`information_schema` + +### Table Information + +| Column Name | Type | Description | +|-----------------|--------------|---------------------| +| TABLE_CATALOG | VARCHAR(64) | Catalog to which the table belongs | +| TABLE_SCHEMA | VARCHAR(64) | Database to which the table belongs | +| TABLE_NAME | VARCHAR(64) | Name of the table | +| PROPERTY_NAME | STRING | Name of the property | +| PROPERTY_VALUE | STRING | Value of the property | + +:::tip +For specific details on table properties, refer to the **Create Table** documentation. +::: + +### Examples + +1. Query all table properties + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. Query the default replication number + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` diff --git a/versioned_docs/version-2.1/admin-manual/system-tables/tables.md b/versioned_docs/version-2.1/admin-manual/system-tables/tables.md new file mode 100644 index 00000000000..7c1c8c22482 --- /dev/null +++ b/versioned_docs/version-2.1/admin-manual/system-tables/tables.md @@ -0,0 +1,64 @@ +--- +{ + "title": "tables", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Stores all table information under the current catalog. + +## Database + +`information_schema` + +## Table Information + +| Column | Type | Description | +|---|---|---| +| TABLE_CATALOG | VARCHAR(512) | Catalog | +| TABLE_SCHEMA | VARCHAR(64) | Database | +| TABLE_NAME | VARCHAR(64) | Table | +| TABLE_TYPE | VARCHAR(64) | Table type: SYSTEM VIEW, VIEW, BASE TABLE | +| ENGINE | VARCHAR(64) | Table engine type | +| VERSION | BIGINT | Invalid value | +| ROW_FORMAT | VARCHAR(10) | Invalid value | +| TABLE_ROWS | BIGINT | Estimated row count | +| AVG_ROW_LENGTH | BIGINT | Average row size | +| DATA_LENGTH | BIGINT | Estimated table size | +| MAX_DATA_LENGTH | BIGINT | Invalid value | +| INDEX_LENGTH | BIGINT | Invalid value | +| DATA_FREE | BIGINT | Invalid value | +| AUTO_INCREMENT | BIGINT | Invalid value | +| CREATE_TIME | DATETIME | Create time | +| UPDATE_TIME | DATETIME | Update time | +| CHECK_TIME | DATETIME | Invalid value | +| TABLE_COLLATION | VARCHAR(32) | Fix value: utf-8 | +| CHECKSUM | BIGINT | Invalid value | +| CREATE_OPTIONS | VARCHAR(255) | Invalid value | +| TABLE_COMMENT | VARCHAR(2048) | Comment | + +## Example + +None + diff --git a/versioned_docs/version-3.0/admin-manual/log-management/fe-log.md b/versioned_docs/version-3.0/admin-manual/log-management/fe-log.md index bb5807b96b0..943de526503 100644 --- a/versioned_docs/version-3.0/admin-manual/log-management/fe-log.md +++ b/versioned_docs/version-3.0/admin-manual/log-management/fe-log.md @@ -79,7 +79,11 @@ The following configuration items are configured in the `fe.conf` file. | `audit_log_modules` | `{"slow_query", "query", "load", "stream_load"}` | | Module types in `fe.audit.log`. Default includes slow query, query, load, stream load. "Query" includes all DDL, DML, SQL operations. "Slow query" refers to operations that exceed the `qe_slow_log_ms` threshold. "Load" refers to Broker Load. "Stream load" refers to stream load operations. | | `qe_slow_log_ms` | 5000 | | When the execution time of DDL, DML, SQL statements exceeds this threshold, it will be separately recorded in the `slow_query` module of `fe.audit.log`. Default is 5000 ms. | | `audit_log_enable_compress` | false | true, false | Whether to enable compression for historical `fe.audit.log` logs. Default is off. When enabled, historical audit logs will be archived using gzip compression. | -| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | Output mode of FE logs. `NORMAL` is the default output mode, with synchronous output and location information. `BRIEF` mode is synchronous output without location information. `ASYNC` mode is asynchronous output without location information, with performance increasing in that order. | +| `sys_log_mode` | `NORMAL` | `NORMAL`, `BRIEF`, `ASYNC` | FE log output mode, where `NORMAL` is the default output mode, log output is synchronous and includes location information. `ASYNC` is the default log output is asynchronous and includes location information. `BRIEF` mode is log output asynchronously but does not include location information. The performance of the three log output modes increases in sequence. | + +> Note +> +> Starting from version 3.0.2, the default value of `sys_log_mode` configuration is changed to `AYSNC`. ## Enable DEBUG Log diff --git a/versioned_docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md b/versioned_docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md new file mode 100644 index 00000000000..616084ae01d --- /dev/null +++ b/versioned_docs/version-3.0/admin-manual/system-tables/file_cache_statistics.md @@ -0,0 +1,110 @@ +--- +{ + "title": "file_cache_statistics", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Used to view the metric information related to data cache on each BE node. The metric information is sourced from the monitoring metrics related to BE's data cache. + +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +## Belongs to Database + +`information_schema` + +## Table Information + +| Column Name | Type | Description | +|---|---|---| +| BE_ID | BIGINT | BE node ID | +| BE_IP | VARCHAR(256) | BE node IP | +| CACHE_PATH | VARCHAR(256) | BE node cache path | +| METRIC_NAME | VARCHAR(256) | Metric name | +| METRIC_VALUE | DOUBLE | Metric value | + +> Different Doris version may have different metrics + +### 2.1.x Metrics + +> Only important metrics are listed. + +- `normal_queue_curr_elements` + + Number of File Blocks currently in the cache. + +- `normal_queue_max_elements` + + Maximum number of File Blocks allowed in the cache. + +- `normal_queue_curr_size` + + Current cache size. + +- `normal_queue_max_size` + + Maximum cache size allowed. + +- `hits_ratio` + + Overall cache hit ratio since BE startup. + +- `hits_ratio_5m` + + Cache hit ratio in the last 5 minutes. + +- `hits_ratio_1h` + + Cache hit ratio in the last 1 hour. + +### 3.0.x Metrics + +TODO + +## Examples + +1. Query all cache metrics + + ``` + mysql> select * from information_schema.file_cache_statistics; + +-------+---------------+----------------------------+----------------------------+--------------------+ + | BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE | + +-------+---------------+----------------------------+----------------------------+--------------------+ + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_elements | 1392 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_curr_size | 248922234 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_elements | 102400 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | normal_queue_max_size | 21474836480 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio | 0.8539634687001242 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0 | + | 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0 | + +-------+---------------+----------------------------+----------------------------+--------------------+ + ``` + +2. Query cache hit ratio and sort by hit ratio + + ``` + select * from information_schema.file_cache_statistics where METRIC_NAME = "hits_ratio" order by METRIC_VALUE desc; + ``` diff --git a/versioned_docs/version-3.0/admin-manual/system-tables/table_options.md b/versioned_docs/version-3.0/admin-manual/system-tables/table_options.md new file mode 100644 index 00000000000..34958b077ae --- /dev/null +++ b/versioned_docs/version-3.0/admin-manual/system-tables/table_options.md @@ -0,0 +1,69 @@ +--- +{ + "title": "table_options", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +`table_options` is a built-in system table of doris, which is stored under the information_schema database. You can view the current table options information of each table through the `table_options` system table. + +## Database + +`information_schema` + +## Table Information + ++-----------------+-------------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++-----------------+-------------+------+-------+---------+-------+ +| TABLE_CATALOG | VARCHAR(64) | Yes | false | NULL | | +| TABLE_SCHEMA | VARCHAR(64) | Yes | false | NULL | | +| TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +| TABLE_MODEL | TEXT | Yes | false | NULL | | +| TABLE_MODEL_KEY | TEXT | Yes | false | NULL | | +| DISTRIBUTE_KEY | TEXT | Yes | false | NULL | | +| DISTRIBUTE_TYPE | TEXT | Yes | false | NULL | | +| BUCKETS_NUM | INT | Yes | false | NULL | | +| PARTITION_NUM | INT | Yes | false | NULL | | ++-----------------+-------------+------+-------+---------+-------+ + + +### Example +mysql> select * from information_schema.table_options where table_schema="test_table_options_db"; ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ +| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_MODEL | TABLE_MODEL_KEY | DISTRIBUTE_KEY | DISTRIBUTE_TYPE | BUCKETS_NUM | PARTITION_NUM | ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ +| internal | test_table_options_db | randomtable | DUP | user_id,date,timestamp | RANDOM | RANDOM | 16 | 1 | +| internal | test_table_options_db | test_row_column_page_size1 | DUP | aaa | aaa | HASH | 1 | 1 | +| internal | test_table_options_db | aggregate_table | AGG | user_id,date,city,age,sex | user_id | HASH | 1 | 1 | +| internal | test_table_options_db | rangetable | AGG | user_id,date,timestamp,city,age,sex | user_id | HASH | 8 | 3 | +| internal | test_table_options_db | unique_table | UNI | user_id,username | user_id | HASH | 1 | 1 | +| internal | test_table_options_db | listtable | AGG | user_id,date,timestamp,city,age,sex | user_id | HASH | 16 | 3 | +| internal | test_table_options_db | duplicate_table | DUP | timestamp,type,error_code | type | HASH | 1 | 1 | ++---------------+-----------------------+----------------------------+-------------+-------------------------------------+----------------+-----------------+-------------+---------------+ + + +### KeyWords + + table_options, information_schema diff --git a/versioned_docs/version-3.0/admin-manual/system-tables/table_properties.md b/versioned_docs/version-3.0/admin-manual/system-tables/table_properties.md new file mode 100644 index 00000000000..182a9fe3d6e --- /dev/null +++ b/versioned_docs/version-3.0/admin-manual/system-tables/table_properties.md @@ -0,0 +1,110 @@ +--- +{ + "title": "table_properties", + "language": "en" +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +### Overview + +Used to view the properties of tables (including internal and external tables). + +:::tip +This system table is supported from versions 2.1.6 and 3.0.2. +::: + +### Database + +`information_schema` + +### Table Information + +| Column Name | Type | Description | +|-----------------|--------------|---------------------| +| TABLE_CATALOG | VARCHAR(64) | Catalog to which the table belongs | +| TABLE_SCHEMA | VARCHAR(64) | Database to which the table belongs | +| TABLE_NAME | VARCHAR(64) | Name of the table | +| PROPERTY_NAME | STRING | Name of the property | +| PROPERTY_VALUE | STRING | Value of the property | + +:::tip +For specific details on table properties, refer to the **Create Table** documentation. +::: + +### Examples + +1. Query all table properties + + ``` + mysql> select * from information_schema.table_properties; + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ... + | internal | test_database | test_table | min_load_replica_num | -1 | + | internal | test_database | test_table | data_sort.col_num | 3 | + | internal | test_database | test_table | group_commit_interval_ms | 10000 | + | internal | test_database | test_table | data_sort.sort_type | LEXICAL | + | internal | test_database | test_table | is_being_synced | false | + | internal | test_database | test_table | binlog.enable | false | + | internal | test_database | test_table | enable_mow_light_delete | false | + | internal | test_database | test_table | binlog.ttl_seconds | 86400 | + | internal | test_database | test_table | inverted_index_storage_format | V2 | + | internal | test_database | test_table | time_series_compaction_empty_rowsets_threshold | 5 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | time_series_compaction_level_threshold | 1 | + | internal | test_database | test_table | time_series_compaction_time_threshold_seconds | 3600 | + | internal | test_database | test_table | storage_format | V2 | + | internal | test_database | test_table | store_row_column | false | + | internal | test_database | test_table | light_schema_change | true | + | internal | test_database | test_table | enable_unique_key_merge_on_write | false | + | internal | test_database | test_table | in_memory | false | + | internal | test_database | test_table | file_cache_ttl_seconds | 0 | + | internal | test_database | test_table | group_commit_data_bytes | 134217728 | + | internal | test_database | test_table | compaction_policy | size_based | + | internal | test_database | test_table | _auto_bucket | false | + | internal | test_database | test_table | binlog.max_history_nums | 9223372036854775807 | + | internal | test_database | test_table | time_series_compaction_file_count_threshold | 2000 | + | internal | test_database | test_table | skip_write_index_on_load | false | + | internal | test_database | test_table | disable_auto_compaction | false | + | internal | test_database | test_table | row_store_page_size | 16384 | + | internal | test_database | test_table | time_series_compaction_goal_size_mbytes | 1024 | + | internal | test_database | test_table | storage_medium | HDD | + | internal | test_database | test_table | enable_single_replica_compaction | false | + | internal | test_database | test_table | compression | LZ4F | + | internal | test_database | test_table | binlog.max_bytes | 9223372036854775807 | + +---------------+---------------+----------------------+------------------------------------------------+-------------------------+ + ``` + +2. Query the default replication number + + ``` + mysql> select * from information_schema.table_properties where PROPERTY_NAME="default.replication_allocation"; + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | PROPERTY_NAME | PROPERTY_VALUE | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + | internal | __internal_schema | column_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | partition_statistics | default.replication_allocation | tag.location.default: 1 | + | internal | __internal_schema | audit_log | default.replication_allocation | tag.location.default: 1 | + | internal | test_database | test_table | default.replication_allocation | tag.location.default: 1 | + +---------------+----------------------+----------------------+--------------------------------+-------------------------+ + ``` diff --git a/versioned_sidebars/version-2.1-sidebars.json b/versioned_sidebars/version-2.1-sidebars.json index 0c9bd387e84..e4606be7697 100644 --- a/versioned_sidebars/version-2.1-sidebars.json +++ b/versioned_sidebars/version-2.1-sidebars.json @@ -428,6 +428,15 @@ "admin-manual/config/user-property" ] }, + { + "type": "category", + "label": "System Tables", + "items": [ + "admin-manual/system-tables/file_cache_statistics", + "admin-manual/system-tables/tables", + "admin-manual/system-tables/table_properties" + ] + }, "admin-manual/audit-plugin", { "type": "category", diff --git a/versioned_sidebars/version-3.0-sidebars.json b/versioned_sidebars/version-3.0-sidebars.json index c04253915e1..2ae79667c61 100644 --- a/versioned_sidebars/version-3.0-sidebars.json +++ b/versioned_sidebars/version-3.0-sidebars.json @@ -482,7 +482,9 @@ "type": "category", "label": "System Tables", "items": [ - "admin-manual/system-tables/tables" + "admin-manual/system-tables/file_cache_statistics", + "admin-manual/system-tables/tables", + "admin-manual/system-tables/table_properties" ] }, "admin-manual/audit-plugin", --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org