This is an automated email from the ASF dual-hosted git repository. kassiez pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new 6750fe484f update getting-started (#1542) 6750fe484f is described below commit 6750fe484f79c643616ffe6c5fd8249a87d176c0 Author: wangtianyi2004 <376612...@qq.com> AuthorDate: Wed Dec 25 11:46:07 2024 +0800 update getting-started (#1542) ## Versions - [ ] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 ## Languages - [ ] Chinese - [ ] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built --------- Co-authored-by: KassieZ <139741991+kass...@users.noreply.github.com> --- docs/gettingStarted/what-is-apache-doris.md | 109 +++++++++++++++------ .../current/gettingStarted/what-is-apache-doris.md | 106 +++++++++++++------- .../gettingStarted/what-is-apache-doris.md | 8 +- .../gettingStarted/what-is-apache-doris.md | 8 +- .../gettingStarted/what-is-apache-doris.md | 93 +++++++++++------- .../gettingStarted/what-is-apache-doris.md | 104 +++++++++++++------- .../apache-doris-query-engine-1.png | Bin .../apache-doris-query-engine-2.png | Bin ...technical-compute-storage-decouple-overview.png | Bin 0 -> 171485 bytes .../apache-doris-technical-overview.png | Bin .../apache-doris-usage-scenarios-pipeline.png | Bin .../gettingStarted/what-is-apache-doris.md | 8 +- .../gettingStarted/what-is-apache-doris.md | 8 +- .../gettingStarted/what-is-apache-doris.md | 97 ++++++++++++------ .../gettingStarted/what-is-apache-doris.md | 109 +++++++++++++++------ 15 files changed, 445 insertions(+), 205 deletions(-) diff --git a/docs/gettingStarted/what-is-apache-doris.md b/docs/gettingStarted/what-is-apache-doris.md index a0b62e6d32..aeacf6ce4f 100644 --- a/docs/gettingStarted/what-is-apache-doris.md +++ b/docs/gettingStarted/what-is-apache-doris.md @@ -1,5 +1,8 @@ --- -{ "title": "What's Apache Doris", "language": "en" } +{ +'title': 'Apache Doris 简介', +'language': 'zh-CN' +} --- <!-- @@ -22,6 +25,7 @@ under the License. --> +## What's Apache Doris Apache Doris is an MPP-based real-time data warehouse known for its high query speed. For queries on large datasets, it returns results in sub-seconds. It supports both high-concurrent point queries and high-throughput complex analysis. It can be used for report analysis, ad-hoc queries, unified data warehouse, and data lake query acceleration. Based on Apache Doris, users can build applications for user behavior analysis, A/B testing platform, log analysis, user profile analysis, and e- [...] @@ -31,64 +35,111 @@ Apache Doris has a wide user base. It has been used in production environments o ## Usage Scenarios -The figure below shows what Apache Doris can do in a data pipeline. Data sources, after integration and processing, are ingested into the Apache Doris real-time data warehouse and offline data lakehouses such as Hive, Iceberg, and Hudi. Apache Doris can be used for the following purposes: +As shown in the figure below, after various data integrations and processing, data sources are typically ingested into the real-time data warehouse Doris and offline lakehouses (such as Hive, Iceberg, and Hudi). These are widely used in OLAP analysis scenarios. - + -- **Report analysis** - - Real-time dashboards - - Reports for internal analysts and managers - - Customer-facing reports: such as site analysis for website owners and advertising reports for advertisers. Such cases typically require high concurrency (thousands of QPS) and low query latency (measured in milliseconds). For example, the e-commerce giant JD.com uses Apache Doris for ad reporting. It ingests 10 billion rows of data per day and achieves over 10,000 QPS and P99 latency of 150ms. -- **Ad-hoc query**: analyst-facing self-service analytics with irregular query patterns and high throughput requirements. For example, Xiaomi builds a Growth Analytics platform based on Apache Doris. Handling 10,000s of SQL queries every day, it delivers an average query latency of 10 seconds and a P95 latency of 30 seconds. -- **Data Lakehouse**: Apache Doris allows federated queries on external tables in offline data lakehouses such as Hive, Hudi, and Iceberg and achieves outstanding query performance by avoiding data copying. -- **Log analysis**: Apache Doris supports inverted index and full-text search since version 2.0. Relying on its highly efficient query and storage engines, Apache Doris enables 10 times higher cost-effectiveness than common log analytic solutions. -- **Unified data warehouse**: Apache Doris can work as a unified data processing platform for various analytic workloads, saving users from handling complicated data components and tech stacks. For example, Haidilao, a world-renowned chain restaurant, replaces its old architecture consisting of Spark, Hive, Kudu, HBase, and Phoenix with Apache Doris. +Apache Doris is widely used in the following scenarios: + +- Reporting Services and Ad-Hoc Queries: Doris provides multidimensional data analysis capabilities, supporting internal enterprise reports and ad-hoc report queries. It offers stable and high-performance service support for high-concurrency report access by users. + +- Real-Time Data Warehouse Analysis: Doris can be applied to real-time data processing and analysis scenarios, providing second-level synchronization of data changes from TP databases and sub-second data query capabilities. It serves scenarios such as real-time dashboards, real-time risk control, real-time order analysis, and real-time advertiser reports. + +- Unified Lakehouse Analysis: Doris’s external table federated analysis accesses data stored in offline lakehouses like Hive, Iceberg, and Hudi. This approach significantly enhances query performance without the need for data duplication. + +- User Profiling and Behavior Analysis: Utilizing Doris’s built-in behavior analysis functions and bitmap types supports user behavior analysis and profiling scenarios. It provides efficient query and real-time analysis capabilities, helping enterprises quickly gain user insights, optimize user experiences, and make informed business decisions. + +- Log Search and Analysis: Doris supports inverted indexing and full-text search, effectively meeting log search and analysis requirements. Leveraging its efficient query and storage engines, Doris offers a cost-performance advantage of up to tenfold compared to traditional log search and analysis solutions. ## Technical overview -Apache Doris has a simple and neat architecture with only two types of processes. +Apache Doris adopts the MySQL protocol and is highly compatible with MySQL syntax, supporting standard SQL. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools. When deploying Apache Doris, users can choose between a compute-storage integrated architecture or a compute-storage separated architecture based on their hardware environment and business requirements. + +### Compute-storage decoupled +Apache Doris has a simple and neat architecture with only two types of processes: + +- Frontend(FE): Mainly responsible for handling user request access, query parsing and planning, metadata management, and node management-related tasks. + +- Backend(BE): Mainly responsible for data storage and the execution of query plans. Data is partitioned into shards and stored with multiple replicas in the BE. -- **Frontend (FE)**: user request processing, query parsing and planning, metadata management, and node management -- **Backend (BE)**: data storage and query execution + -Both frontend and backend processes are scalable, supporting up to hundreds of machines and tens of petabytes of storage capacity in a single cluster. Both types of processes guarantee high service availability and high data reliability through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance costs of a distributed system. +In a production environment, multiple FE nodes can be deployed for disaster recovery and backup, with each FE maintaining a full copy of the metadata. The FE has three roles: - + | Role | Function | + | -------- | ------------------------------------------------------------ | + | Master | The FE Master node is responsible for reading and writing metadata. When the Master metadata undergoes changes, it synchronizes these changes to Follower or Observer nodes using the BDB JE protocol. | + | Follower | Follower nodes are responsible for reading metadata. In the event of a Master node failure, a Follower node can be selected as the new Master node. | + | Observer | Observer nodes are responsible for reading metadata, primarily to enhance the cluster's query concurrency and performance. They do not participate in the cluster's leader election. | -## Interface +Both FE and BE processes are horizontally scalable, allowing a single cluster to support hundreds of machines and tens of petabytes of storage capacity. FE and BE processes ensure high availability and data reliability through consistency protocols. The compute-storage integrated architecture is highly consolidated, significantly reducing the operational costs of distributed systems. -Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. +### Compute-Storage Decoupled +Starting from version 3.0, a compute-storage decoupled deployment architecture can be chosen. The compute-storage decoupled version of Apache Doris utilizes a unified shared storage layer as the data storage space. By separating storage and computation, users can independently scale storage capacity and computing resources, thereby achieving optimal performance and cost efficiency. As shown in the figure below, the compute-storage decoupled architecture is divided into three layers: -## Storage engine +- Metadata Layer: The metadata layer is primarily responsible for request planning, query parsing and planning, as well as metadata storage and management. + +- Compute Layer: The compute layer consists of multiple compute groups, each of which can operate as an independent tenant handling business computations. Within each compute group, there are multiple stateless BE nodes, and BE nodes can be elastically scaled up or down at any time. + +- Storage Layer: The storage layer can use shared storage solutions such as S3, HDFS, OSS, COS, OBS, Minio, and Ceph to store Doris's data files, including Segment files and inverted index files. -Apache Doris has a columnar **storage engine**, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. + -Apache Doris supports various **index** structures to minimize data scans: +## Core Features of Apache Doris + +- High Availability: In Apache Doris, both metadata and data are stored with multiple replicas, synchronizing data logs via the quorum protocol. Data write is considered successful once a majority of replicas have completed the write, ensuring that the cluster remains available even if a few nodes fail. Apache Doris supports both same-city and cross-region disaster recovery, enabling dual-cluster master-slave modes. When some nodes experience failures, the cluster can automatically isola [...] + +- High Compatibility: Apache Doris is highly compatible with the MySQL protocol and supports standard SQL syntax, covering most MySQL and Hive functions. This high compatibility allows users to seamlessly migrate and integrate existing applications and tools. Apache Doris supports the MySQL ecosystem, enabling users to connect Doris using MySQL Client tools for more convenient operations and maintenance. It also supports MySQL protocol compatibility for BI reporting tools and data transm [...] + +- Real-Time Data Warehouse: Based on Apache Doris, a real-time data warehouse service can be built. Apache Doris offers second-level data ingestion capabilities, capturing incremental changes from upstream online transactional databases into Doris within seconds. Leveraging vectorized engines, MPP architecture, and Pipeline execution engines, Doris provides sub-second data query capabilities, thereby constructing a high-performance, low-latency real-time data warehouse platform. + +- Unified Lakehouse: Apache Doris can build a unified lakehouse architecture based on external data sources such as data lakes or relational databases. The Doris unified lakehouse solution enables seamless integration and free data flow between data lakes and data warehouses, helping users directly utilize data warehouse capabilities to solve data analysis problems in data lakes while fully leveraging data lake data management capabilities to enhance data value. + +- Flexible Modeling: Apache Doris offers various modeling approaches, such as wide table models, pre-aggregation models, star/snowflake schemas, etc. During data import, data can be flattened into wide tables and written into Doris through compute engines like Flink or Spark, or data can be directly imported into Doris, performing data modeling operations through views, materialized views, or real-time multi-table joins. + +## Technical overview + +Doris provides an efficient SQL interface and is fully compatible with the MySQL protocol. Its query engine is based on an MPP (Massively Parallel Processing) architecture, capable of efficiently executing complex analytical queries and achieving low-latency real-time queries. Through columnar storage technology for data encoding and compression, it significantly optimizes query performance and storage compression ratio. + +### Interface + +Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. + +### Storage engine + +Apache Doris has a columnar storage engine, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. + +Apache Doris supports various index structures to minimize data scans: - Sorted Compound Key Index: Users can specify three columns at most to form a compound sort key. This can effectively prune data to better support highly concurrent reporting scenarios. + - Min/Max Index: This enables effective data filtering in equivalence and range queries of numeric types. + - BloomFilter Index: This is very effective in equivalence filtering and pruning of high-cardinality columns. + - Inverted Index: This enables fast searching for any field. -Apache Doris supports a variety of **data models** and has optimized them for different scenarios: +- Apache Doris supports a variety of data models and has optimized them for different scenarios: + +Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation -- Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation - Unique Key Model: ensures uniqueness of keys and overwrites data with the same key to achieve row-level data updates + - Duplicate Key Model: stores data as it is without aggregation, capable of detailed storage of fact tables -Apache Doris also supports **strongly consistent materialized views**. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. +- Apache Doris also supports strongly consistent materialized views. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. -## Query engine +### Query engine Apache Doris has an MPP-based query engine for parallel execution between and within nodes. It supports distributed shuffle join for large tables to better handle complicated queries. - + The query engine of Apache Doris is fully vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, increase cache hit rates, and make efficient use of SIMD instructions. Apache Doris delivers a 5~10 times higher performance in wide table aggregation scenarios than non-vectorized engines. - + -Apache Doris uses **adaptive query execution** technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. +Apache Doris uses adaptive query execution technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. -The query **optimizer** of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. +The query optimizer of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/gettingStarted/what-is-apache-doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/gettingStarted/what-is-apache-doris.md index 809ad7b7b6..601c64c369 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/gettingStarted/what-is-apache-doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/gettingStarted/what-is-apache-doris.md @@ -24,46 +24,82 @@ specific language governing permissions and limitations under the License. --> + ## Apache Doris 的发展 -Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下的查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、湖仓一体等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 +Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目,2017 年正式对外开源,2018 年 7 月由百度捐赠给 Apache 基金会进行孵化,之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。2022 年 6 月,Apache Doris 成功从 Apache 孵化器毕业,正式成为 Apache 顶级项目(Top-Level Project,TLP)。目前 Apache Doris 社区已经聚集了来自不同行业数百家企业的 600 余位贡献者,并且每月活跃贡献者人数也超过 120 位。 -Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云厂商比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 +Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云商厂比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 + +## 使用场景 + +如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Doris 和离线湖仓(Hive、Iceberg 与 Hudi 等),广泛应用于 OLAP 分析场景。 + + + +Apache Doris 被广泛应用在以下场景中: + +- 报表服务与即席查询:Doris 提供多维数据分析能力,为企业内部的报表、面相分析是的即系报表查询,面相用户的高并发报表提供稳定高性能服务支撑; + +- 实时数仓分析:Doris 可以应用于实时数据处理与分析场景,提供秒级同步 TP 数据库的数据变更及亚秒级数据查询能力,服务于实时大屏、实时风控、实时订单分析、实时广告主报表等场景; + +- 湖仓一体统一分析:Doris 外表联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升; + +- 用户画像与行为分析:利用 Doris 内置的行为分析函数与 bitmap 类型可以支撑用户行为分析与画像圈人场景,可以提供高效的查询与实时分析能力,帮助企业快速获取用户洞察,优化用户体验和企业决策; + +- 日志检索与分析:Doris 支持倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 -## Apache Doris 的使用场景 +## 整体架构 -如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Apache Doris 和离线湖仓(Hive, Iceberg, Hudi 中),Apache Doris 被广泛应用在以下场景中。 +Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL,用户可以通过各类客户端工具来访问 Apache Doris,并支持与 BI 工具的无缝对接。在部署 Apache 时,可以根据硬件环境与业务需求选择存算一体架构或存算分离架构。 - +### 存算一体 +Apache Doris 存算一体架构精简易于维护,如下图所示,只有两类进程: -- 报表分析 +- Frontend(FE):主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 - - 实时看板(Dashboards); +- Backend(BE):主要负责数据存储、查询计划的执行。数据会被切分成分片,在 BE 中多副本存储。 + + + +在生产环境中可以部署多个 FE 节点做容灾备份,每个 FE 中都会维护全量的元数据副本。FE 分为三种角色: + + | 角色 | 功能 | + | -------- | ------------------------------------------------------------ | + | Master | FE Master 节点负责元数据的读写,在 Master 元数据发生变更后,会通过 BDB JE 协议同步给 Follower 或 Observer 节点。 | + | Follower | Follower 节点负责读取元数据,在 Master 节点发生故障时,Follower 节点可以被选取作为新的 Master 节点。 | + | Observer | Observer 节点负责读取元数据,主要为了增加集群的查询并发行。不参加集群选主。 | + +FE 与 BE 进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。FE 与 BE 进程通过一致性协议来保证服务的高可用和数据的高可靠。存算一体架构高度集成,大幅降低了分布式系统的运维成本。 + +### 存算分离 +自 3.0 版本后,可以选择存算分离部署架构。Apache Doris 存算分离版使用统一的共享存储层作为数据存储空间。存储和计算分离,用户可以独立扩展存储容量和计算资源,从而实现最佳性能和成本效益。如下图所示,存算分离架构分为三层: + +- 元数据层:元数据层主要负责请求规划,查询解析规划与元数据的存储与管理; - - 面向企业内部分析师和管理者的报表; +- 计算层:计算层由多个计算组组成,每个计算组可以作为一个独立的租户承担业务计算。在每一个计算组中,有多个无状态的 BE 节点,计算组中可以随时弹性扩缩容 BE 节点; - - 面向用户或者客户的高并发报表分析(Customer Facing Analytics)。比如面向网站主的站点分析、面向广告主的广告报表,并发通常要求成千上万的 QPS,查询延时要求毫秒级响应。著名的电商公司京东在广告报表中使用 Apache Doris,每天写入 100 亿行数据,查询并发 QPS 上万,99 分位的查询延时 150ms。 +- 存储层:存储层可以使用 S3、HDFS、OSS、COS、OBS、Minio、Ceph 等共享存储存放 Doris 的数据文件,包含包括 Segment 文件、反向索引的索引文件等。 -- 即席查询(Ad-hoc Query):面向分析师的自助分析,查询模式不固定,要求较高的吞吐。小米公司基于 Apache Doris 构建了增长分析平台(Growing Analytics,GA),利用用户行为数据对业务进行增长分析,平均查询延时 10s,95 分位的查询延时 30s 以内,每天的 SQL 查询量为数万条。 + -- 湖仓一体(Data Lakehouse):通过外表的方式联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升。 +## Apache Doris 的核心特性 -- 日志检索分析:在 Apache Doris 2.0 版本中,支持了倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 +- 高可用:在 Apache Doris 中,元数据和数据均采用多副本存储,通过 quorum 协议同步数据日志。当大多数副本完成写入后,即认定数据写入成功,从而确保即使少数节点发生故障,集群仍能保持可用性。Apache Doris 支持同城和异地容灾,能够实现双集群主备模式。当部分节点发生异常时,集群可以自动隔离故障节点,避免影响整体集群的可用性。 -- 统一数仓构建:一个平台满足统一的数据仓库建设需求,简化繁琐的大数据软件栈。海底捞基于 Apache Doris 构建的统一数仓,替换了原来由 Spark、Hive、Kudu、Hbase、Phoenix 组成的旧架构,架构大大简化。 +- 高兼容:Apache Doris 高度兼容 MySQL 协议,支持标准 SQL 语法,涵盖绝大部分 MySQL 和 Hive 函数。通过这种高兼容性,用户可以无缝地迁移和集成现有的应用和工具。Apache Doris 支持 MySQL 生态,用户可以通过 MySQL Client 工具链接 Doris,使得操作和维护更加便捷,也可以使用 MySQL 协议对 BI 报表工具与数据传输工具进行兼容适配,确保数据分析和数据传输过程中的高效性和稳定性; -## 整体架构和技术特点 +- 实时数仓:基于 Apache Doris 可以构建实时数据仓库服务。Apache Doris 提供了秒级数入库能力,上游在线联机事务库中的增量变更可以秒级捕获到 Doris 中,依靠向量化引擎、MPP 架构及 Pipeline 执行引擎等加速手段可以提供亚秒级数据查询能力,从而构建高性能低延迟的实时数仓平台; -Apache Doris 的**整体架构**非常简单,如下图所示,只有两类进程: +- 湖仓一体:Apache 可以基于外部数据源,如数据湖或关系型数据库构建湖仓一体的架构。Apache Doris 湖仓一体解决了数据能够在数据湖和数据仓库之间进行无缝的集成和自由的流转,从而帮助用户直接利用数据仓库的能力来解决数据湖中的数据分析问题,同时又能充分利用数据湖的数据管理能力来提升数据的价值; -- **Frontend(FE)**:主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 -- **Backend(BE)**:主要负责数据存储、查询计划的执行。 +- 灵活建模:Apache Doris 提供多种建模方式,如宽表模型、预聚合模型、星型/雪花模型等。数据导入时,可以通过 Flink、Spark 等计算引擎将数据打平成宽表写入到 Doris 中,也可以将数据直接导入到 Doris 中,通过视图、物化视图或实时多表关联等方式进行数据的建模操作。 -这两类进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大地降低了一款分布式系统的运维成本。 +## 技术特点 - +Doris 提供了高效的 SQL 接口,并完全兼容 MySQL 协议。其查询引擎基于 MPP(大规模并行处理)架构,能够高效执行复杂的分析查询,并实现低延迟的实时查询。通过列式存储技术对数据进行编码与压缩,显著优化了查询性能和存储压缩比。 ### 使用接口 @@ -71,39 +107,41 @@ Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL ### 存储引擎 -在**存储引擎**方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 +在存储引擎方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 -Apache Doris 也支持比较丰富的**索引结构**,来减少数据的扫描: +Apache Doris 也支持比较丰富的索引结构,来减少数据的扫描: - Sorted Compound Key Index,可以最多指定三个列组成复合排序键,通过该索引,能够有效进行数据裁剪,从而能够更好支持高并发的报表场景 - + - Min/Max Index:有效过滤数值类型的等值和范围查询 - + - BloomFilter Index:对高基数列的等值过滤裁剪非常有效 - + - Inverted Index:能够对任意字段实现快速检索 -在**存储模型**方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: +在存储模型方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: - 聚合模型(Aggregate Key Model):相同 Key 的 Value 列合并,通过提前聚合大幅提升性能 - + - 主键模型(Unique Key Model):Key 唯一,相同 Key 的数据覆盖,实现行级别数据更新 - + - 明细模型(Duplicate Key Model):明细数据模型,满足事实表的明细存储 -Apache Doris 也支持**强一致的物化视图**,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 +Apache Doris 也支持强一致的物化视图,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 ### 查询引擎 -在**查询引擎**方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 +在查询引擎方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 + + + +Apache Doris 查询引擎是向量化的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 - + -**Apache Doris 查询引擎是向量化**的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 +Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术, 可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 - +在优化器方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 -**Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术,**可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 -在**优化器**方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.2/gettingStarted/what-is-apache-doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.2/gettingStarted/what-is-apache-doris.md index 809ad7b7b6..b47121e05b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.2/gettingStarted/what-is-apache-doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.2/gettingStarted/what-is-apache-doris.md @@ -36,7 +36,7 @@ Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群 如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Apache Doris 和离线湖仓(Hive, Iceberg, Hudi 中),Apache Doris 被广泛应用在以下场景中。 - + - 报表分析 @@ -63,7 +63,7 @@ Apache Doris 的**整体架构**非常简单,如下图所示,只有两类进 这两类进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大地降低了一款分布式系统的运维成本。 - + ### 使用接口 @@ -97,12 +97,12 @@ Apache Doris 也支持**强一致的物化视图**,物化视图的更新和选 在**查询引擎**方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 - + **Apache Doris 查询引擎是向量化**的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 - + **Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术,**可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/gettingStarted/what-is-apache-doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/gettingStarted/what-is-apache-doris.md index 809ad7b7b6..b47121e05b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/gettingStarted/what-is-apache-doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/gettingStarted/what-is-apache-doris.md @@ -36,7 +36,7 @@ Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群 如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Apache Doris 和离线湖仓(Hive, Iceberg, Hudi 中),Apache Doris 被广泛应用在以下场景中。 - + - 报表分析 @@ -63,7 +63,7 @@ Apache Doris 的**整体架构**非常简单,如下图所示,只有两类进 这两类进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大地降低了一款分布式系统的运维成本。 - + ### 使用接口 @@ -97,12 +97,12 @@ Apache Doris 也支持**强一致的物化视图**,物化视图的更新和选 在**查询引擎**方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 - + **Apache Doris 查询引擎是向量化**的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 - + **Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术,**可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/gettingStarted/what-is-apache-doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/gettingStarted/what-is-apache-doris.md index 809ad7b7b6..f9bf076980 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/gettingStarted/what-is-apache-doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/gettingStarted/what-is-apache-doris.md @@ -24,46 +24,71 @@ specific language governing permissions and limitations under the License. --> + ## Apache Doris 的发展 -Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下的查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、湖仓一体等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 +Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目,2017 年正式对外开源,2018 年 7 月由百度捐赠给 Apache 基金会进行孵化,之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。2022 年 6 月,Apache Doris 成功从 Apache 孵化器毕业,正式成为 Apache 顶级项目(Top-Level Project,TLP)。目前 Apache Doris 社区已经聚集了来自不同行业数百家企业的 600 余位贡献者,并且每月活跃贡献者人数也超过 120 位。 -Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云厂商比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 +Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云商厂比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 + +## 使用场景 -## Apache Doris 的使用场景 +如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Doris 和离线湖仓(Hive、Iceberg 与 Hudi 等),广泛应用于 OLAP 分析场景。 -如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Apache Doris 和离线湖仓(Hive, Iceberg, Hudi 中),Apache Doris 被广泛应用在以下场景中。 + - +Apache Doris 被广泛应用在以下场景中: -- 报表分析 +- 报表服务与即席查询:Doris 提供多维数据分析能力,为企业内部的报表、面相分析是的即系报表查询,面相用户的高并发报表提供稳定高性能服务支撑; + +- 实时数仓分析:Doris 可以应用于实时数据处理与分析场景,提供秒级同步 TP 数据库的数据变更及亚秒级数据查询能力,服务于实时大屏、实时风控、实时订单分析、实时广告主报表等场景; - - 实时看板(Dashboards); +- 湖仓一体统一分析:Doris 外表联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升; - - 面向企业内部分析师和管理者的报表; +- 用户画像与行为分析:利用 Doris 内置的行为分析函数与 bitmap 类型可以支撑用户行为分析与画像圈人场景,可以提供高效的查询与实时分析能力,帮助企业快速获取用户洞察,优化用户体验和企业决策; - - 面向用户或者客户的高并发报表分析(Customer Facing Analytics)。比如面向网站主的站点分析、面向广告主的广告报表,并发通常要求成千上万的 QPS,查询延时要求毫秒级响应。著名的电商公司京东在广告报表中使用 Apache Doris,每天写入 100 亿行数据,查询并发 QPS 上万,99 分位的查询延时 150ms。 +- 日志检索与分析:Doris 支持倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 + +## 整体架构 + +Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL,用户可以通过各类客户端工具来访问 Apache Doris,并支持与 BI 工具的无缝对接。 + + +Apache Doris 存算一体架构精简易于维护,如下图所示,只有两类进程: + +- Frontend(FE):主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 + +- Backend(BE):主要负责数据存储、查询计划的执行。数据会被切分成分片,在 BE 中多副本存储。 + + + +在生产环境中可以部署多个 FE 节点做容灾备份,每个 FE 中都会维护全量的元数据副本。FE 分为三种角色: -- 即席查询(Ad-hoc Query):面向分析师的自助分析,查询模式不固定,要求较高的吞吐。小米公司基于 Apache Doris 构建了增长分析平台(Growing Analytics,GA),利用用户行为数据对业务进行增长分析,平均查询延时 10s,95 分位的查询延时 30s 以内,每天的 SQL 查询量为数万条。 + | 角色 | 功能 | + | -------- | ------------------------------------------------------------ | + | Master | FE Master 节点负责元数据的读写,在 Master 元数据发生变更后,会通过 BDB JE 协议同步给 Follower 或 Observer 节点。 | + | Follower | Follower 节点负责读取元数据,在 Master 节点发生故障时,Follower 节点可以被选取作为新的 Master 节点。 | + | Observer | Observer 节点负责读取元数据,主要为了增加集群的查询并发行。不参加集群选主。 | -- 湖仓一体(Data Lakehouse):通过外表的方式联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升。 +FE 与 BE 进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。FE 与 BE 进程通过一致性协议来保证服务的高可用和数据的高可靠。存算一体架构高度集成,大幅降低了分布式系统的运维成本。 -- 日志检索分析:在 Apache Doris 2.0 版本中,支持了倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 +## Apache Doris 的核心特性 -- 统一数仓构建:一个平台满足统一的数据仓库建设需求,简化繁琐的大数据软件栈。海底捞基于 Apache Doris 构建的统一数仓,替换了原来由 Spark、Hive、Kudu、Hbase、Phoenix 组成的旧架构,架构大大简化。 +- 高可用:在 Apache Doris 中,元数据和数据均采用多副本存储,通过 quorum 协议同步数据日志。当大多数副本完成写入后,即认定数据写入成功,从而确保即使少数节点发生故障,集群仍能保持可用性。Apache Doris 支持同城和异地容灾,能够实现双集群主备模式。当部分节点发生异常时,集群可以自动隔离故障节点,避免影响整体集群的可用性。 -## 整体架构和技术特点 +- 高兼容:Apache Doris 高度兼容 MySQL 协议,支持标准 SQL 语法,涵盖绝大部分 MySQL 和 Hive 函数。通过这种高兼容性,用户可以无缝地迁移和集成现有的应用和工具。Apache Doris 支持 MySQL 生态,用户可以通过 MySQL Client 工具链接 Doris,使得操作和维护更加便捷,也可以使用 MySQL 协议对 BI 报表工具与数据传输工具进行兼容适配,确保数据分析和数据传输过程中的高效性和稳定性; -Apache Doris 的**整体架构**非常简单,如下图所示,只有两类进程: +- 实时数仓:基于 Apache Doris 可以构建实时数据仓库服务。Apache Doris 提供了秒级数入库能力,上游在线联机事务库中的增量变更可以秒级捕获到 Doris 中,依靠向量化引擎、MPP 架构及 Pipeline 执行引擎等加速手段可以提供亚秒级数据查询能力,从而构建高性能低延迟的实时数仓平台; -- **Frontend(FE)**:主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 -- **Backend(BE)**:主要负责数据存储、查询计划的执行。 +- 湖仓一体:Apache 可以基于外部数据源,如数据湖或关系型数据库构建湖仓一体的架构。Apache Doris 湖仓一体解决了数据能够在数据湖和数据仓库之间进行无缝的集成和自由的流转,从而帮助用户直接利用数据仓库的能力来解决数据湖中的数据分析问题,同时又能充分利用数据湖的数据管理能力来提升数据的价值; -这两类进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大地降低了一款分布式系统的运维成本。 +- 灵活建模:Apache Doris 提供多种建模方式,如宽表模型、预聚合模型、星型/雪花模型等。数据导入时,可以通过 Flink、Spark 等计算引擎将数据打平成宽表写入到 Doris 中,也可以将数据直接导入到 Doris 中,通过视图、物化视图或实时多表关联等方式进行数据的建模操作。 - +## 技术特点 + +Doris 提供了高效的 SQL 接口,并完全兼容 MySQL 协议。其查询引擎基于 MPP(大规模并行处理)架构,能够高效执行复杂的分析查询,并实现低延迟的实时查询。通过列式存储技术对数据进行编码与压缩,显著优化了查询性能和存储压缩比。 ### 使用接口 @@ -71,39 +96,39 @@ Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL ### 存储引擎 -在**存储引擎**方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 +在存储引擎方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 -Apache Doris 也支持比较丰富的**索引结构**,来减少数据的扫描: +Apache Doris 也支持比较丰富的索引结构,来减少数据的扫描: - Sorted Compound Key Index,可以最多指定三个列组成复合排序键,通过该索引,能够有效进行数据裁剪,从而能够更好支持高并发的报表场景 - + - Min/Max Index:有效过滤数值类型的等值和范围查询 - + - BloomFilter Index:对高基数列的等值过滤裁剪非常有效 - + - Inverted Index:能够对任意字段实现快速检索 -在**存储模型**方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: +在存储模型方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: - 聚合模型(Aggregate Key Model):相同 Key 的 Value 列合并,通过提前聚合大幅提升性能 - + - 主键模型(Unique Key Model):Key 唯一,相同 Key 的数据覆盖,实现行级别数据更新 - + - 明细模型(Duplicate Key Model):明细数据模型,满足事实表的明细存储 -Apache Doris 也支持**强一致的物化视图**,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 +Apache Doris 也支持强一致的物化视图,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 ### 查询引擎 -在**查询引擎**方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 +在查询引擎方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 - + -**Apache Doris 查询引擎是向量化**的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 +Apache Doris 查询引擎是向量化的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 + - +Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术, 可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 -**Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术,**可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 +在优化器方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 -在**优化器**方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/gettingStarted/what-is-apache-doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/gettingStarted/what-is-apache-doris.md index 809ad7b7b6..f0b4571f4c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/gettingStarted/what-is-apache-doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/gettingStarted/what-is-apache-doris.md @@ -24,46 +24,82 @@ specific language governing permissions and limitations under the License. --> + ## Apache Doris 的发展 -Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下的查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、湖仓一体等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 +Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库,以高效、简单、统一的特点被人们所熟知,仅需亚秒级响应时间即可返回海量数据下查询结果,不仅可以支持高并发的点查询场景,也能支持高吞吐的复杂分析场景。基于此,Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景,用户可以在此之上构建大屏看板、用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。 Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目,2017 年正式对外开源,2018 年 7 月由百度捐赠给 Apache 基金会进行孵化,之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。2022 年 6 月,Apache Doris 成功从 Apache 孵化器毕业,正式成为 Apache 顶级项目(Top-Level Project,TLP)。目前 Apache Doris 社区已经聚集了来自不同行业数百家企业的 600 余位贡献者,并且每月活跃贡献者人数也超过 120 位。 -Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云厂商比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 +Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体,截止目前,Apache Doris 已经在全球超过 4000 家中大型企业的生产环境中得到应用,在中国市值或估值排行前 50 的互联网公司中,有超过 80% 长期使用 Apache Doris,包括百度、美团、小米、京东、字节跳动、阿里巴巴、腾讯、网易、快手、微博等。同时在一些传统行业如金融、消费、电信、工业制造、能源、医疗、政务等领域也有着丰富的应用。在中国几乎所有的云商厂比如阿里云、华为云、天翼云、腾讯云、百度云、火山引擎等都在提供托管的 Apache Doris 的云服务。 + +## 使用场景 -## Apache Doris 的使用场景 +如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Doris 和离线湖仓(Hive、Iceberg 与 Hudi 等),广泛应用于 OLAP 分析场景。 -如下图所示,数据源经过各种数据集成和加工处理后,通常会入库到实时数据仓库 Apache Doris 和离线湖仓(Hive, Iceberg, Hudi 中),Apache Doris 被广泛应用在以下场景中。 + - +Apache Doris 被广泛应用在以下场景中: -- 报表分析 +- 报表服务与即席查询:Doris 提供多维数据分析能力,为企业内部的报表、面相分析是的即系报表查询,面相用户的高并发报表提供稳定高性能服务支撑; + +- 实时数仓分析:Doris 可以应用于实时数据处理与分析场景,提供秒级同步 TP 数据库的数据变更及亚秒级数据查询能力,服务于实时大屏、实时风控、实时订单分析、实时广告主报表等场景; + +- 湖仓一体统一分析:Doris 外表联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升; - - 实时看板(Dashboards); +- 用户画像与行为分析:利用 Doris 内置的行为分析函数与 bitmap 类型可以支撑用户行为分析与画像圈人场景,可以提供高效的查询与实时分析能力,帮助企业快速获取用户洞察,优化用户体验和企业决策; + +- 日志检索与分析:Doris 支持倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 + +## 整体架构 + +Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL,用户可以通过各类客户端工具来访问 Apache Doris,并支持与 BI 工具的无缝对接。在部署 Apache 时,可以根据硬件环境与业务需求选择存算一体架构或存算分离架构。 + +### 存算一体 +Apache Doris 存算一体架构精简易于维护,如下图所示,只有两类进程: + +- Frontend(FE):主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 + +- Backend(BE):主要负责数据存储、查询计划的执行。数据会被切分成分片,在 BE 中多副本存储。 + + + +在生产环境中可以部署多个 FE 节点做容灾备份,每个 FE 中都会维护全量的元数据副本。FE 分为三种角色: + + | 角色 | 功能 | + | -------- | ------------------------------------------------------------ | + | Master | FE Master 节点负责元数据的读写,在 Master 元数据发生变更后,会通过 BDB JE 协议同步给 Follower 或 Observer 节点。 | + | Follower | Follower 节点负责读取元数据,在 Master 节点发生故障时,Follower 节点可以被选取作为新的 Master 节点。 | + | Observer | Observer 节点负责读取元数据,主要为了增加集群的查询并发行。不参加集群选主。 | + +FE 与 BE 进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。FE 与 BE 进程通过一致性协议来保证服务的高可用和数据的高可靠。存算一体架构高度集成,大幅降低了分布式系统的运维成本。 + +### 存算分离 +自 3.0 版本后,可以选择存算分离部署架构。Apache Doris 存算分离版使用统一的共享存储层作为数据存储空间。存储和计算分离,用户可以独立扩展存储容量和计算资源,从而实现最佳性能和成本效益。如下图所示,存算分离架构分为三层: + +- 元数据层:元数据层主要负责请求规划,查询解析规划与元数据的存储与管理; - - 面向企业内部分析师和管理者的报表; +- 计算层:计算层由多个计算组组成,每个计算组可以作为一个独立的租户承担业务计算。在每一个计算组中,有多个无状态的 BE 节点,计算组中可以随时弹性扩缩容 BE 节点; - - 面向用户或者客户的高并发报表分析(Customer Facing Analytics)。比如面向网站主的站点分析、面向广告主的广告报表,并发通常要求成千上万的 QPS,查询延时要求毫秒级响应。著名的电商公司京东在广告报表中使用 Apache Doris,每天写入 100 亿行数据,查询并发 QPS 上万,99 分位的查询延时 150ms。 +- 存储层:存储层可以使用 S3、HDFS、OSS、COS、OBS、Minio、Ceph 等共享存储存放 Doris 的数据文件,包含包括 Segment 文件、反向索引的索引文件等。 -- 即席查询(Ad-hoc Query):面向分析师的自助分析,查询模式不固定,要求较高的吞吐。小米公司基于 Apache Doris 构建了增长分析平台(Growing Analytics,GA),利用用户行为数据对业务进行增长分析,平均查询延时 10s,95 分位的查询延时 30s 以内,每天的 SQL 查询量为数万条。 + -- 湖仓一体(Data Lakehouse):通过外表的方式联邦分析位于 Hive、Iceberg、Hudi 等离线湖仓中的数据,在避免数据拷贝的前提下,查询性能大幅提升。 +## Apache Doris 的核心特性 -- 日志检索分析:在 Apache Doris 2.0 版本中,支持了倒排索引和全文检索,能够很好的满足日志检索分析的场景,并且依赖其高效的查询引擎和存储引擎,相比传统的日志检索分析的方案可以有 10 倍性价比的优势。 +- 高可用:在 Apache Doris 中,元数据和数据均采用多副本存储,通过 quorum 协议同步数据日志。当大多数副本完成写入后,即认定数据写入成功,从而确保即使少数节点发生故障,集群仍能保持可用性。Apache Doris 支持同城和异地容灾,能够实现双集群主备模式。当部分节点发生异常时,集群可以自动隔离故障节点,避免影响整体集群的可用性。 -- 统一数仓构建:一个平台满足统一的数据仓库建设需求,简化繁琐的大数据软件栈。海底捞基于 Apache Doris 构建的统一数仓,替换了原来由 Spark、Hive、Kudu、Hbase、Phoenix 组成的旧架构,架构大大简化。 +- 高兼容:Apache Doris 高度兼容 MySQL 协议,支持标准 SQL 语法,涵盖绝大部分 MySQL 和 Hive 函数。通过这种高兼容性,用户可以无缝地迁移和集成现有的应用和工具。Apache Doris 支持 MySQL 生态,用户可以通过 MySQL Client 工具链接 Doris,使得操作和维护更加便捷,也可以使用 MySQL 协议对 BI 报表工具与数据传输工具进行兼容适配,确保数据分析和数据传输过程中的高效性和稳定性; -## 整体架构和技术特点 +- 实时数仓:基于 Apache Doris 可以构建实时数据仓库服务。Apache Doris 提供了秒级数入库能力,上游在线联机事务库中的增量变更可以秒级捕获到 Doris 中,依靠向量化引擎、MPP 架构及 Pipeline 执行引擎等加速手段可以提供亚秒级数据查询能力,从而构建高性能低延迟的实时数仓平台; -Apache Doris 的**整体架构**非常简单,如下图所示,只有两类进程: +- 湖仓一体:Apache 可以基于外部数据源,如数据湖或关系型数据库构建湖仓一体的架构。Apache Doris 湖仓一体解决了数据能够在数据湖和数据仓库之间进行无缝的集成和自由的流转,从而帮助用户直接利用数据仓库的能力来解决数据湖中的数据分析问题,同时又能充分利用数据湖的数据管理能力来提升数据的价值; -- **Frontend(FE)**:主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。 -- **Backend(BE)**:主要负责数据存储、查询计划的执行。 +- 灵活建模:Apache Doris 提供多种建模方式,如宽表模型、预聚合模型、星型/雪花模型等。数据导入时,可以通过 Flink、Spark 等计算引擎将数据打平成宽表写入到 Doris 中,也可以将数据直接导入到 Doris 中,通过视图、物化视图或实时多表关联等方式进行数据的建模操作。 -这两类进程都是可以横向扩展的,单集群可以支持到数百台机器,数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大地降低了一款分布式系统的运维成本。 +## 技术特点 - +Doris 提供了高效的 SQL 接口,并完全兼容 MySQL 协议。其查询引擎基于 MPP(大规模并行处理)架构,能够高效执行复杂的分析查询,并实现低延迟的实时查询。通过列式存储技术对数据进行编码与压缩,显著优化了查询性能和存储压缩比。 ### 使用接口 @@ -71,39 +107,39 @@ Apache Doris 采用 MySQL 协议,高度兼容 MySQL 语法,支持标准 SQL ### 存储引擎 -在**存储引擎**方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 +在存储引擎方面,Apache Doris 采用列式存储,按列进行数据的编码压缩和读取,能够实现极高的压缩比,同时减少大量非相关数据的扫描,从而更加有效利用 IO 和 CPU 资源。 -Apache Doris 也支持比较丰富的**索引结构**,来减少数据的扫描: +Apache Doris 也支持比较丰富的索引结构,来减少数据的扫描: - Sorted Compound Key Index,可以最多指定三个列组成复合排序键,通过该索引,能够有效进行数据裁剪,从而能够更好支持高并发的报表场景 - + - Min/Max Index:有效过滤数值类型的等值和范围查询 - + - BloomFilter Index:对高基数列的等值过滤裁剪非常有效 - + - Inverted Index:能够对任意字段实现快速检索 -在**存储模型**方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: +在存储模型方面,Apache Doris 支持多种存储模型,针对不同的场景做了针对性的优化: - 聚合模型(Aggregate Key Model):相同 Key 的 Value 列合并,通过提前聚合大幅提升性能 - + - 主键模型(Unique Key Model):Key 唯一,相同 Key 的数据覆盖,实现行级别数据更新 - + - 明细模型(Duplicate Key Model):明细数据模型,满足事实表的明细存储 -Apache Doris 也支持**强一致的物化视图**,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 +Apache Doris 也支持强一致的物化视图,物化视图的更新和选择都在系统内自动进行,不需要用户手动选择,从而大幅减少了物化视图维护的代价。 ### 查询引擎 -在**查询引擎**方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 +在查询引擎方面,Apache Doris 采用 MPP 的模型,节点间和节点内都并行执行,也支持多个大表的分布式 Shuffle Join,从而能够更好应对复杂查询。 - + -**Apache Doris 查询引擎是向量化**的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 +Apache Doris 查询引擎是向量化的查询引擎,所有的内存结构能够按照列式布局,能够达到大幅减少虚函数调用、提升 Cache 命中率,高效利用 SIMD 指令的效果。在宽表聚合场景下性能是非向量化引擎的 5-10 倍。 + - +Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术, 可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 -**Apache Doris 采用了自适应查询执行(Adaptive Query Execution)技术,**可以根据 Runtime Statistics 来动态调整执行计划,比如通过 Runtime Filter 技术能够在运行时生成 Filter 推到 Probe 侧,并且能够将 Filter 自动穿透到 Probe 侧最底层的 Scan 节点,从而大幅减少 Probe 的数据量,加速 Join 性能。Apache Doris 的 Runtime Filter 支持 In/Min/Max/Bloom Filter。 +在优化器方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 -在**优化器**方面,Apache Doris 使用 CBO 和 RBO 结合的优化策略,RBO 支持常量折叠、子查询改写、谓词下推等,CBO 支持 Join Reorder。目前 CBO 还在持续优化中,主要集中在更加精准的统计信息收集和推导,更加精准的代价模型预估等方面。 diff --git a/static/images/apache-doris-query-engine-1.png b/static/images/getting-started/apache-doris-query-engine-1.png similarity index 100% rename from static/images/apache-doris-query-engine-1.png rename to static/images/getting-started/apache-doris-query-engine-1.png diff --git a/static/images/apache-doris-query-engine-2.png b/static/images/getting-started/apache-doris-query-engine-2.png similarity index 100% rename from static/images/apache-doris-query-engine-2.png rename to static/images/getting-started/apache-doris-query-engine-2.png diff --git a/static/images/getting-started/apache-doris-technical-compute-storage-decouple-overview.png b/static/images/getting-started/apache-doris-technical-compute-storage-decouple-overview.png new file mode 100644 index 0000000000..f21562ddfe Binary files /dev/null and b/static/images/getting-started/apache-doris-technical-compute-storage-decouple-overview.png differ diff --git a/static/images/apache-doris-technical-overview.png b/static/images/getting-started/apache-doris-technical-overview.png similarity index 100% rename from static/images/apache-doris-technical-overview.png rename to static/images/getting-started/apache-doris-technical-overview.png diff --git a/static/images/apache-doris-usage-scenarios-pipeline.png b/static/images/getting-started/apache-doris-usage-scenarios-pipeline.png similarity index 100% rename from static/images/apache-doris-usage-scenarios-pipeline.png rename to static/images/getting-started/apache-doris-usage-scenarios-pipeline.png diff --git a/versioned_docs/version-1.2/gettingStarted/what-is-apache-doris.md b/versioned_docs/version-1.2/gettingStarted/what-is-apache-doris.md index a0b62e6d32..9e56128285 100644 --- a/versioned_docs/version-1.2/gettingStarted/what-is-apache-doris.md +++ b/versioned_docs/version-1.2/gettingStarted/what-is-apache-doris.md @@ -33,7 +33,7 @@ Apache Doris has a wide user base. It has been used in production environments o The figure below shows what Apache Doris can do in a data pipeline. Data sources, after integration and processing, are ingested into the Apache Doris real-time data warehouse and offline data lakehouses such as Hive, Iceberg, and Hudi. Apache Doris can be used for the following purposes: - + - **Report analysis** - Real-time dashboards @@ -53,7 +53,7 @@ Apache Doris has a simple and neat architecture with only two types of processes Both frontend and backend processes are scalable, supporting up to hundreds of machines and tens of petabytes of storage capacity in a single cluster. Both types of processes guarantee high service availability and high data reliability through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance costs of a distributed system. - + ## Interface @@ -82,11 +82,11 @@ Apache Doris also supports **strongly consistent materialized views**. Materiali Apache Doris has an MPP-based query engine for parallel execution between and within nodes. It supports distributed shuffle join for large tables to better handle complicated queries. - + The query engine of Apache Doris is fully vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, increase cache hit rates, and make efficient use of SIMD instructions. Apache Doris delivers a 5~10 times higher performance in wide table aggregation scenarios than non-vectorized engines. - + Apache Doris uses **adaptive query execution** technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. diff --git a/versioned_docs/version-2.0/gettingStarted/what-is-apache-doris.md b/versioned_docs/version-2.0/gettingStarted/what-is-apache-doris.md index a0b62e6d32..e07d3f5ee7 100644 --- a/versioned_docs/version-2.0/gettingStarted/what-is-apache-doris.md +++ b/versioned_docs/version-2.0/gettingStarted/what-is-apache-doris.md @@ -33,7 +33,7 @@ Apache Doris has a wide user base. It has been used in production environments o The figure below shows what Apache Doris can do in a data pipeline. Data sources, after integration and processing, are ingested into the Apache Doris real-time data warehouse and offline data lakehouses such as Hive, Iceberg, and Hudi. Apache Doris can be used for the following purposes: - + - **Report analysis** - Real-time dashboards @@ -53,7 +53,7 @@ Apache Doris has a simple and neat architecture with only two types of processes Both frontend and backend processes are scalable, supporting up to hundreds of machines and tens of petabytes of storage capacity in a single cluster. Both types of processes guarantee high service availability and high data reliability through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance costs of a distributed system. - + ## Interface @@ -82,11 +82,11 @@ Apache Doris also supports **strongly consistent materialized views**. Materiali Apache Doris has an MPP-based query engine for parallel execution between and within nodes. It supports distributed shuffle join for large tables to better handle complicated queries. - + The query engine of Apache Doris is fully vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, increase cache hit rates, and make efficient use of SIMD instructions. Apache Doris delivers a 5~10 times higher performance in wide table aggregation scenarios than non-vectorized engines. - + Apache Doris uses **adaptive query execution** technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. diff --git a/versioned_docs/version-2.1/gettingStarted/what-is-apache-doris.md b/versioned_docs/version-2.1/gettingStarted/what-is-apache-doris.md index a0b62e6d32..7d6b16c1f7 100644 --- a/versioned_docs/version-2.1/gettingStarted/what-is-apache-doris.md +++ b/versioned_docs/version-2.1/gettingStarted/what-is-apache-doris.md @@ -1,5 +1,8 @@ --- -{ "title": "What's Apache Doris", "language": "en" } +{ +'title': 'Apache Doris 简介', +'language': 'zh-CN' +} --- <!-- @@ -22,6 +25,7 @@ under the License. --> +## What's Apache Doris Apache Doris is an MPP-based real-time data warehouse known for its high query speed. For queries on large datasets, it returns results in sub-seconds. It supports both high-concurrent point queries and high-throughput complex analysis. It can be used for report analysis, ad-hoc queries, unified data warehouse, and data lake query acceleration. Based on Apache Doris, users can build applications for user behavior analysis, A/B testing platform, log analysis, user profile analysis, and e- [...] @@ -31,64 +35,99 @@ Apache Doris has a wide user base. It has been used in production environments o ## Usage Scenarios -The figure below shows what Apache Doris can do in a data pipeline. Data sources, after integration and processing, are ingested into the Apache Doris real-time data warehouse and offline data lakehouses such as Hive, Iceberg, and Hudi. Apache Doris can be used for the following purposes: +As shown in the figure below, after various data integrations and processing, data sources are typically ingested into the real-time data warehouse Doris and offline lakehouses (such as Hive, Iceberg, and Hudi). These are widely used in OLAP analysis scenarios. - + -- **Report analysis** - - Real-time dashboards - - Reports for internal analysts and managers - - Customer-facing reports: such as site analysis for website owners and advertising reports for advertisers. Such cases typically require high concurrency (thousands of QPS) and low query latency (measured in milliseconds). For example, the e-commerce giant JD.com uses Apache Doris for ad reporting. It ingests 10 billion rows of data per day and achieves over 10,000 QPS and P99 latency of 150ms. -- **Ad-hoc query**: analyst-facing self-service analytics with irregular query patterns and high throughput requirements. For example, Xiaomi builds a Growth Analytics platform based on Apache Doris. Handling 10,000s of SQL queries every day, it delivers an average query latency of 10 seconds and a P95 latency of 30 seconds. -- **Data Lakehouse**: Apache Doris allows federated queries on external tables in offline data lakehouses such as Hive, Hudi, and Iceberg and achieves outstanding query performance by avoiding data copying. -- **Log analysis**: Apache Doris supports inverted index and full-text search since version 2.0. Relying on its highly efficient query and storage engines, Apache Doris enables 10 times higher cost-effectiveness than common log analytic solutions. -- **Unified data warehouse**: Apache Doris can work as a unified data processing platform for various analytic workloads, saving users from handling complicated data components and tech stacks. For example, Haidilao, a world-renowned chain restaurant, replaces its old architecture consisting of Spark, Hive, Kudu, HBase, and Phoenix with Apache Doris. +Apache Doris is widely used in the following scenarios: + +- Reporting Services and Ad-Hoc Queries: Doris provides multidimensional data analysis capabilities, supporting internal enterprise reports and ad-hoc report queries. It offers stable and high-performance service support for high-concurrency report access by users. + +- Real-Time Data Warehouse Analysis: Doris can be applied to real-time data processing and analysis scenarios, providing second-level synchronization of data changes from TP databases and sub-second data query capabilities. It serves scenarios such as real-time dashboards, real-time risk control, real-time order analysis, and real-time advertiser reports. + +- Unified Lakehouse Analysis: Doris’s external table federated analysis accesses data stored in offline lakehouses like Hive, Iceberg, and Hudi. This approach significantly enhances query performance without the need for data duplication. + +- User Profiling and Behavior Analysis: Utilizing Doris’s built-in behavior analysis functions and bitmap types supports user behavior analysis and profiling scenarios. It provides efficient query and real-time analysis capabilities, helping enterprises quickly gain user insights, optimize user experiences, and make informed business decisions. + +- Log Search and Analysis: Doris supports inverted indexing and full-text search, effectively meeting log search and analysis requirements. Leveraging its efficient query and storage engines, Doris offers a cost-performance advantage of up to tenfold compared to traditional log search and analysis solutions. ## Technical overview -Apache Doris has a simple and neat architecture with only two types of processes. +Apache Doris adopts the MySQL protocol and is highly compatible with MySQL syntax, supporting standard SQL. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools. + +Apache Doris has a simple and neat architecture with only two types of processes: + +- Frontend(FE): Mainly responsible for handling user request access, query parsing and planning, metadata management, and node management-related tasks. + +- Backend(BE): Mainly responsible for data storage and the execution of query plans. Data is partitioned into shards and stored with multiple replicas in the BE. + + + +In a production environment, multiple FE nodes can be deployed for disaster recovery and backup, with each FE maintaining a full copy of the metadata. The FE has three roles: + + | Role | Function | + | -------- | ------------------------------------------------------------ | + | Master | The FE Master node is responsible for reading and writing metadata. When the Master metadata undergoes changes, it synchronizes these changes to Follower or Observer nodes using the BDB JE protocol. | + | Follower | Follower nodes are responsible for reading metadata. In the event of a Master node failure, a Follower node can be selected as the new Master node. | + | Observer | Observer nodes are responsible for reading metadata, primarily to enhance the cluster's query concurrency and performance. They do not participate in the cluster's leader election. | + +Both FE and BE processes are horizontally scalable, allowing a single cluster to support hundreds of machines and tens of petabytes of storage capacity. FE and BE processes ensure high availability and data reliability through consistency protocols. The compute-storage integrated architecture is highly consolidated, significantly reducing the operational costs of distributed systems. -- **Frontend (FE)**: user request processing, query parsing and planning, metadata management, and node management -- **Backend (BE)**: data storage and query execution +## Core Features of Apache Doris -Both frontend and backend processes are scalable, supporting up to hundreds of machines and tens of petabytes of storage capacity in a single cluster. Both types of processes guarantee high service availability and high data reliability through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance costs of a distributed system. +- High Availability: In Apache Doris, both metadata and data are stored with multiple replicas, synchronizing data logs via the quorum protocol. Data write is considered successful once a majority of replicas have completed the write, ensuring that the cluster remains available even if a few nodes fail. Apache Doris supports both same-city and cross-region disaster recovery, enabling dual-cluster master-slave modes. When some nodes experience failures, the cluster can automatically isola [...] - +- High Compatibility: Apache Doris is highly compatible with the MySQL protocol and supports standard SQL syntax, covering most MySQL and Hive functions. This high compatibility allows users to seamlessly migrate and integrate existing applications and tools. Apache Doris supports the MySQL ecosystem, enabling users to connect Doris using MySQL Client tools for more convenient operations and maintenance. It also supports MySQL protocol compatibility for BI reporting tools and data transm [...] -## Interface +- Real-Time Data Warehouse: Based on Apache Doris, a real-time data warehouse service can be built. Apache Doris offers second-level data ingestion capabilities, capturing incremental changes from upstream online transactional databases into Doris within seconds. Leveraging vectorized engines, MPP architecture, and Pipeline execution engines, Doris provides sub-second data query capabilities, thereby constructing a high-performance, low-latency real-time data warehouse platform. -Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. +- Unified Lakehouse: Apache Doris can build a unified lakehouse architecture based on external data sources such as data lakes or relational databases. The Doris unified lakehouse solution enables seamless integration and free data flow between data lakes and data warehouses, helping users directly utilize data warehouse capabilities to solve data analysis problems in data lakes while fully leveraging data lake data management capabilities to enhance data value. -## Storage engine +- Flexible Modeling: Apache Doris offers various modeling approaches, such as wide table models, pre-aggregation models, star/snowflake schemas, etc. During data import, data can be flattened into wide tables and written into Doris through compute engines like Flink or Spark, or data can be directly imported into Doris, performing data modeling operations through views, materialized views, or real-time multi-table joins. -Apache Doris has a columnar **storage engine**, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. +## Technical overview + +Doris provides an efficient SQL interface and is fully compatible with the MySQL protocol. Its query engine is based on an MPP (Massively Parallel Processing) architecture, capable of efficiently executing complex analytical queries and achieving low-latency real-time queries. Through columnar storage technology for data encoding and compression, it significantly optimizes query performance and storage compression ratio. + +### Interface + +Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. + +### Storage engine + +Apache Doris has a columnar storage engine, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. -Apache Doris supports various **index** structures to minimize data scans: +Apache Doris supports various index structures to minimize data scans: - Sorted Compound Key Index: Users can specify three columns at most to form a compound sort key. This can effectively prune data to better support highly concurrent reporting scenarios. + - Min/Max Index: This enables effective data filtering in equivalence and range queries of numeric types. + - BloomFilter Index: This is very effective in equivalence filtering and pruning of high-cardinality columns. + - Inverted Index: This enables fast searching for any field. -Apache Doris supports a variety of **data models** and has optimized them for different scenarios: +- Apache Doris supports a variety of data models and has optimized them for different scenarios: + +Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation -- Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation - Unique Key Model: ensures uniqueness of keys and overwrites data with the same key to achieve row-level data updates + - Duplicate Key Model: stores data as it is without aggregation, capable of detailed storage of fact tables -Apache Doris also supports **strongly consistent materialized views**. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. +- Apache Doris also supports strongly consistent materialized views. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. -## Query engine +### Query engine Apache Doris has an MPP-based query engine for parallel execution between and within nodes. It supports distributed shuffle join for large tables to better handle complicated queries. - + The query engine of Apache Doris is fully vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, increase cache hit rates, and make efficient use of SIMD instructions. Apache Doris delivers a 5~10 times higher performance in wide table aggregation scenarios than non-vectorized engines. - + -Apache Doris uses **adaptive query execution** technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. +Apache Doris uses adaptive query execution technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. -The query **optimizer** of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. +The query optimizer of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. diff --git a/versioned_docs/version-3.0/gettingStarted/what-is-apache-doris.md b/versioned_docs/version-3.0/gettingStarted/what-is-apache-doris.md index a0b62e6d32..aeacf6ce4f 100644 --- a/versioned_docs/version-3.0/gettingStarted/what-is-apache-doris.md +++ b/versioned_docs/version-3.0/gettingStarted/what-is-apache-doris.md @@ -1,5 +1,8 @@ --- -{ "title": "What's Apache Doris", "language": "en" } +{ +'title': 'Apache Doris 简介', +'language': 'zh-CN' +} --- <!-- @@ -22,6 +25,7 @@ under the License. --> +## What's Apache Doris Apache Doris is an MPP-based real-time data warehouse known for its high query speed. For queries on large datasets, it returns results in sub-seconds. It supports both high-concurrent point queries and high-throughput complex analysis. It can be used for report analysis, ad-hoc queries, unified data warehouse, and data lake query acceleration. Based on Apache Doris, users can build applications for user behavior analysis, A/B testing platform, log analysis, user profile analysis, and e- [...] @@ -31,64 +35,111 @@ Apache Doris has a wide user base. It has been used in production environments o ## Usage Scenarios -The figure below shows what Apache Doris can do in a data pipeline. Data sources, after integration and processing, are ingested into the Apache Doris real-time data warehouse and offline data lakehouses such as Hive, Iceberg, and Hudi. Apache Doris can be used for the following purposes: +As shown in the figure below, after various data integrations and processing, data sources are typically ingested into the real-time data warehouse Doris and offline lakehouses (such as Hive, Iceberg, and Hudi). These are widely used in OLAP analysis scenarios. - + -- **Report analysis** - - Real-time dashboards - - Reports for internal analysts and managers - - Customer-facing reports: such as site analysis for website owners and advertising reports for advertisers. Such cases typically require high concurrency (thousands of QPS) and low query latency (measured in milliseconds). For example, the e-commerce giant JD.com uses Apache Doris for ad reporting. It ingests 10 billion rows of data per day and achieves over 10,000 QPS and P99 latency of 150ms. -- **Ad-hoc query**: analyst-facing self-service analytics with irregular query patterns and high throughput requirements. For example, Xiaomi builds a Growth Analytics platform based on Apache Doris. Handling 10,000s of SQL queries every day, it delivers an average query latency of 10 seconds and a P95 latency of 30 seconds. -- **Data Lakehouse**: Apache Doris allows federated queries on external tables in offline data lakehouses such as Hive, Hudi, and Iceberg and achieves outstanding query performance by avoiding data copying. -- **Log analysis**: Apache Doris supports inverted index and full-text search since version 2.0. Relying on its highly efficient query and storage engines, Apache Doris enables 10 times higher cost-effectiveness than common log analytic solutions. -- **Unified data warehouse**: Apache Doris can work as a unified data processing platform for various analytic workloads, saving users from handling complicated data components and tech stacks. For example, Haidilao, a world-renowned chain restaurant, replaces its old architecture consisting of Spark, Hive, Kudu, HBase, and Phoenix with Apache Doris. +Apache Doris is widely used in the following scenarios: + +- Reporting Services and Ad-Hoc Queries: Doris provides multidimensional data analysis capabilities, supporting internal enterprise reports and ad-hoc report queries. It offers stable and high-performance service support for high-concurrency report access by users. + +- Real-Time Data Warehouse Analysis: Doris can be applied to real-time data processing and analysis scenarios, providing second-level synchronization of data changes from TP databases and sub-second data query capabilities. It serves scenarios such as real-time dashboards, real-time risk control, real-time order analysis, and real-time advertiser reports. + +- Unified Lakehouse Analysis: Doris’s external table federated analysis accesses data stored in offline lakehouses like Hive, Iceberg, and Hudi. This approach significantly enhances query performance without the need for data duplication. + +- User Profiling and Behavior Analysis: Utilizing Doris’s built-in behavior analysis functions and bitmap types supports user behavior analysis and profiling scenarios. It provides efficient query and real-time analysis capabilities, helping enterprises quickly gain user insights, optimize user experiences, and make informed business decisions. + +- Log Search and Analysis: Doris supports inverted indexing and full-text search, effectively meeting log search and analysis requirements. Leveraging its efficient query and storage engines, Doris offers a cost-performance advantage of up to tenfold compared to traditional log search and analysis solutions. ## Technical overview -Apache Doris has a simple and neat architecture with only two types of processes. +Apache Doris adopts the MySQL protocol and is highly compatible with MySQL syntax, supporting standard SQL. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools. When deploying Apache Doris, users can choose between a compute-storage integrated architecture or a compute-storage separated architecture based on their hardware environment and business requirements. + +### Compute-storage decoupled +Apache Doris has a simple and neat architecture with only two types of processes: + +- Frontend(FE): Mainly responsible for handling user request access, query parsing and planning, metadata management, and node management-related tasks. + +- Backend(BE): Mainly responsible for data storage and the execution of query plans. Data is partitioned into shards and stored with multiple replicas in the BE. -- **Frontend (FE)**: user request processing, query parsing and planning, metadata management, and node management -- **Backend (BE)**: data storage and query execution + -Both frontend and backend processes are scalable, supporting up to hundreds of machines and tens of petabytes of storage capacity in a single cluster. Both types of processes guarantee high service availability and high data reliability through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance costs of a distributed system. +In a production environment, multiple FE nodes can be deployed for disaster recovery and backup, with each FE maintaining a full copy of the metadata. The FE has three roles: - + | Role | Function | + | -------- | ------------------------------------------------------------ | + | Master | The FE Master node is responsible for reading and writing metadata. When the Master metadata undergoes changes, it synchronizes these changes to Follower or Observer nodes using the BDB JE protocol. | + | Follower | Follower nodes are responsible for reading metadata. In the event of a Master node failure, a Follower node can be selected as the new Master node. | + | Observer | Observer nodes are responsible for reading metadata, primarily to enhance the cluster's query concurrency and performance. They do not participate in the cluster's leader election. | -## Interface +Both FE and BE processes are horizontally scalable, allowing a single cluster to support hundreds of machines and tens of petabytes of storage capacity. FE and BE processes ensure high availability and data reliability through consistency protocols. The compute-storage integrated architecture is highly consolidated, significantly reducing the operational costs of distributed systems. -Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. +### Compute-Storage Decoupled +Starting from version 3.0, a compute-storage decoupled deployment architecture can be chosen. The compute-storage decoupled version of Apache Doris utilizes a unified shared storage layer as the data storage space. By separating storage and computation, users can independently scale storage capacity and computing resources, thereby achieving optimal performance and cost efficiency. As shown in the figure below, the compute-storage decoupled architecture is divided into three layers: -## Storage engine +- Metadata Layer: The metadata layer is primarily responsible for request planning, query parsing and planning, as well as metadata storage and management. + +- Compute Layer: The compute layer consists of multiple compute groups, each of which can operate as an independent tenant handling business computations. Within each compute group, there are multiple stateless BE nodes, and BE nodes can be elastically scaled up or down at any time. + +- Storage Layer: The storage layer can use shared storage solutions such as S3, HDFS, OSS, COS, OBS, Minio, and Ceph to store Doris's data files, including Segment files and inverted index files. -Apache Doris has a columnar **storage engine**, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. + -Apache Doris supports various **index** structures to minimize data scans: +## Core Features of Apache Doris + +- High Availability: In Apache Doris, both metadata and data are stored with multiple replicas, synchronizing data logs via the quorum protocol. Data write is considered successful once a majority of replicas have completed the write, ensuring that the cluster remains available even if a few nodes fail. Apache Doris supports both same-city and cross-region disaster recovery, enabling dual-cluster master-slave modes. When some nodes experience failures, the cluster can automatically isola [...] + +- High Compatibility: Apache Doris is highly compatible with the MySQL protocol and supports standard SQL syntax, covering most MySQL and Hive functions. This high compatibility allows users to seamlessly migrate and integrate existing applications and tools. Apache Doris supports the MySQL ecosystem, enabling users to connect Doris using MySQL Client tools for more convenient operations and maintenance. It also supports MySQL protocol compatibility for BI reporting tools and data transm [...] + +- Real-Time Data Warehouse: Based on Apache Doris, a real-time data warehouse service can be built. Apache Doris offers second-level data ingestion capabilities, capturing incremental changes from upstream online transactional databases into Doris within seconds. Leveraging vectorized engines, MPP architecture, and Pipeline execution engines, Doris provides sub-second data query capabilities, thereby constructing a high-performance, low-latency real-time data warehouse platform. + +- Unified Lakehouse: Apache Doris can build a unified lakehouse architecture based on external data sources such as data lakes or relational databases. The Doris unified lakehouse solution enables seamless integration and free data flow between data lakes and data warehouses, helping users directly utilize data warehouse capabilities to solve data analysis problems in data lakes while fully leveraging data lake data management capabilities to enhance data value. + +- Flexible Modeling: Apache Doris offers various modeling approaches, such as wide table models, pre-aggregation models, star/snowflake schemas, etc. During data import, data can be flattened into wide tables and written into Doris through compute engines like Flink or Spark, or data can be directly imported into Doris, performing data modeling operations through views, materialized views, or real-time multi-table joins. + +## Technical overview + +Doris provides an efficient SQL interface and is fully compatible with the MySQL protocol. Its query engine is based on an MPP (Massively Parallel Processing) architecture, capable of efficiently executing complex analytical queries and achieving low-latency real-time queries. Through columnar storage technology for data encoding and compression, it significantly optimizes query performance and storage compression ratio. + +### Interface + +Apache Doris adopts the MySQL protocol, supports standard SQL, and is highly compatible with MySQL syntax. Users can access Apache Doris through various client tools and seamlessly integrate it with BI tools, including but not limited to Smartbi, DataEase, FineBI, Tableau, Power BI, and Apache Superset. Apache Doris can work as the data source for any BI tools that support the MySQL protocol. + +### Storage engine + +Apache Doris has a columnar storage engine, which encodes, compresses, and reads data by column. This enables a very high data compression ratio and largely reduces unnecessary data scanning, thus making more efficient use of IO and CPU resources. + +Apache Doris supports various index structures to minimize data scans: - Sorted Compound Key Index: Users can specify three columns at most to form a compound sort key. This can effectively prune data to better support highly concurrent reporting scenarios. + - Min/Max Index: This enables effective data filtering in equivalence and range queries of numeric types. + - BloomFilter Index: This is very effective in equivalence filtering and pruning of high-cardinality columns. + - Inverted Index: This enables fast searching for any field. -Apache Doris supports a variety of **data models** and has optimized them for different scenarios: +- Apache Doris supports a variety of data models and has optimized them for different scenarios: + +Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation -- Aggregate Key Model: merges the value columns with the same keys and improves performance by pre-aggregation - Unique Key Model: ensures uniqueness of keys and overwrites data with the same key to achieve row-level data updates + - Duplicate Key Model: stores data as it is without aggregation, capable of detailed storage of fact tables -Apache Doris also supports **strongly consistent materialized views**. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. +- Apache Doris also supports strongly consistent materialized views. Materialized views are automatically selected and updated within the system without manual efforts, which reduces maintenance costs for users. -## Query engine +### Query engine Apache Doris has an MPP-based query engine for parallel execution between and within nodes. It supports distributed shuffle join for large tables to better handle complicated queries. - + The query engine of Apache Doris is fully vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, increase cache hit rates, and make efficient use of SIMD instructions. Apache Doris delivers a 5~10 times higher performance in wide table aggregation scenarios than non-vectorized engines. - + -Apache Doris uses **adaptive query execution** technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. +Apache Doris uses adaptive query execution technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate a runtime filter and push it to the probe side. Specifically, it pushes the filters to the lowest-level scan node on the probe side, which largely reduces the data amount to be processed and increases join performance. The runtime filter of Apache Doris supports In/Min/Max/Bloom Filter. -The query **optimizer** of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. +The query optimizer of Apache Doris is a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, and predicate pushdown while CBO supports join reorder. The Apache Doris CBO is under continuous optimization for more accurate statistics collection and inference as well as a more accurate cost model. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org