johnpyp commented on code in PR #937:
URL: https://github.com/apache/doris-website/pull/937#discussion_r1707836600
##########
versioned_docs/version-3.0/releasenotes/release-3.0.0.md:
##########
@@ -4,3 +4,166 @@
"language": "en"
}
---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Thanks to our devoted developers and supportive community users, the
much-expected Apache Doris 3.0.0 is now available!
+
+## Compute-Storage Decoupled
+
+From version 3.0.0, Doris supports a compute-storage decoupled mode, allowing
users to physically isolate business workloads and separate read and write
operations through multiple computing clusters. This mode substantially reduces
storage costs by utilizing mature object storage or HDFS. It mitigates
operational complexities such as disk balancing and data loss due to multiple
BE nodes going offline.
+
+**Future-Oriented Architecture**: The architecture is selected considering
complexity, performance, and scalability:
+
+**1. Reducing Complexity**
+
+- Decoupling OLAP layer semantics from storage implementation simplifies
system maintenance and reduces operational complexity.
+
+- Choosing production-verified mature storage systems.
+
+- Single source of truth for metadata, greatly simplifying system complexity,
such as avoiding write publishes in integrated storage-computation modes and
providing strong consistency in design.
+
+**2. Performance**
+
+Capable of achieving the performance of integrated storage-computation mode.
+
+**3. Scalability**
+
+- Data scale support is no longer limited by FE memory.
+
+- Architecture can evolve and integrate both integrated and separated storage
computation.
+
+- Easily supports time travel, data sharing, and load isolation.
+
+
+
+Based on these considerations, the architectural choices made include:
+
+1. A single metadata service divided into semantic and storage layers, with
the storage layer choosing production-verified and transaction-capable
FoundationDB.
+
+2. Storage options include mature, low-cost object storage and HDFS.
+
+3. To compensate for performance loss due to remote storage, local data
caching has been introduced at computing nodes, combining the simplicity of a
Shared Disk architecture with the performance of a Share Nothing architecture.
+
+
+The overall architecture is a three-layer structure: metadata layer, computing
layer, and data storage layer.
+
+- Metadata Layer: Provides the system's metadata services, such as database
tables, schemas, rowset metadata, and transactions. Currently, BE's metadata
has fully transitioned to MetaService, and FE's transaction metadata has
entered MetaService. Future versions will incorporate other metadata into
MetaService.
+
+- Computing Layer: Responsible for executing query plans and providing load
isolation through computing clusters. Computing nodes are completely stateless,
with local data acting as a cache for remote storage to accelerate queries.
+
+- Data Storage Layer: Data is persisted to shared storage, currently
supporting S3, OSS, COS, GCS, Azure Blob, MinIO, BOS, and HDFS.
+
+[](https://private-user-images.githubusercontent.com/6919662/346635704-76ee422f-95a8-48a0-808c-08817e5e46c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodW
IuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMzkyNTcsIm5iZiI6MTcyMjMzODk1NywicGF0aCI6Ii82OTE5NjYyLzM0NjYzNTcwNC03NmVlNDIyZi05NWE4LTQ4YTAtODA4Yy0wODgxN2U1ZTQ2YzgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzBUMTEyOTE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTgxMGI4NTcxNzM0Y2JiZDAyNDBhZGU1ZmQwNmQzOGUzYzU4MjY3N2JjNTEwOTM4MDljNDZmMDdmZDA5NzFlZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.K0UF5zzpp-qxY4YpU8vOxDfjn1VexTpiyxcl5Bv5Puc)
+
+
+Production Insights
+
+**1. Operations-Friendly**
+
+- Cache TTL: In real-time scenarios, setting the Cache TTL ensures that data
within the TTL provides low-latency queries from the local cache.
+
+- Seamless Scaling: One benefit of the compute-storage decoupled mode is the
rapid elasticity of computing resources. Doris uses progressive active cache
preheating technology, ensuring no impact on ongoing queries and imports during
scalability periods.
+
+- Seamless Upgrades: During upgrades, the multi-process mode operation ensures
ongoing queries and imports do not fail.
+
+
+
+**2. Load Isolation**
+
+- Multiple Computing Clusters: Different business characteristics (e.g., batch
processing, real-time) or business units (different businesses within a group)
can use independent computing clusters for physical isolation.
+
+- Read-Write Separation: Imports and queries can be completed by different
computing clusters, supporting automatic (notification for preheating during
import) and manual (periodic manual preheating) methods to preheat the query
cluster's cache.
+
+
+
+**3. Positive Data Deletion**
+
+In distributed systems, common data deletion involves reverse GC, comparing
files in shared storage with those recorded in metadata and deleting unrecorded
files. This method can easily lead to data deletion errors under concurrent
conditions. Relying on MetaService's transaction capabilities, Doris adopts
asynchronous forward data deletion technology. Data generation is bound to a
transaction, as is metadata deletion, and the asynchronous deletion process
simply deletes data based on transaction records, eliminating the need for
reverse GC.
+
+
+
+## Optimizer
+
+- All DQL and DML, except for load operations, will use the new optimizer.
Default settings disable fallback to the old optimizer, which will be removed
in subsequent versions.
+
+- Comprehensive top-N runtime filter applicable to all query modes, enhancing
performance when a top-N operator is present.
+
+- Experimental Feature: Refactored logic for synchronized materialized view
selection, transitioning from RBO to CBO to maintain consistency with
asynchronous materialized views. If issues arise, the switch `set global
enable_sync_mv_cost_based_rewrite = false` can be used to revert to RBO mode.
+
+- Experimental Feature: Partition-level statistics. By activating the switch
`set global enable_partition_analyze = true`, partition-level statistics can be
collected and utilized, improving query performance in cases of data partition
skew.
+
+- Experimental Feature: Generated columns. When creating tables, columns can
be designated as generated columns, which automatically compute results based
on defined expressions at write time, offering more complex expressions
compared to default values but disallowing explicit writing of specified values.
Review Comment:
Suggestion: Link to the corresponding docs for Generated Columns
##########
versioned_docs/version-3.0/releasenotes/release-3.0.0.md:
##########
@@ -4,3 +4,166 @@
"language": "en"
}
---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Thanks to our devoted developers and supportive community users, the
much-expected Apache Doris 3.0.0 is now available!
+
+## Compute-Storage Decoupled
+
+From version 3.0.0, Doris supports a compute-storage decoupled mode, allowing
users to physically isolate business workloads and separate read and write
operations through multiple computing clusters. This mode substantially reduces
storage costs by utilizing mature object storage or HDFS. It mitigates
operational complexities such as disk balancing and data loss due to multiple
BE nodes going offline.
+
+**Future-Oriented Architecture**: The architecture is selected considering
complexity, performance, and scalability:
+
+**1. Reducing Complexity**
+
+- Decoupling OLAP layer semantics from storage implementation simplifies
system maintenance and reduces operational complexity.
+
+- Choosing production-verified mature storage systems.
+
+- Single source of truth for metadata, greatly simplifying system complexity,
such as avoiding write publishes in integrated storage-computation modes and
providing strong consistency in design.
+
+**2. Performance**
+
+Capable of achieving the performance of integrated storage-computation mode.
+
+**3. Scalability**
+
+- Data scale support is no longer limited by FE memory.
+
+- Architecture can evolve and integrate both integrated and separated storage
computation.
+
+- Easily supports time travel, data sharing, and load isolation.
+
+
+
+Based on these considerations, the architectural choices made include:
+
+1. A single metadata service divided into semantic and storage layers, with
the storage layer choosing production-verified and transaction-capable
FoundationDB.
+
+2. Storage options include mature, low-cost object storage and HDFS.
+
+3. To compensate for performance loss due to remote storage, local data
caching has been introduced at computing nodes, combining the simplicity of a
Shared Disk architecture with the performance of a Share Nothing architecture.
+
+
+The overall architecture is a three-layer structure: metadata layer, computing
layer, and data storage layer.
+
+- Metadata Layer: Provides the system's metadata services, such as database
tables, schemas, rowset metadata, and transactions. Currently, BE's metadata
has fully transitioned to MetaService, and FE's transaction metadata has
entered MetaService. Future versions will incorporate other metadata into
MetaService.
+
+- Computing Layer: Responsible for executing query plans and providing load
isolation through computing clusters. Computing nodes are completely stateless,
with local data acting as a cache for remote storage to accelerate queries.
+
+- Data Storage Layer: Data is persisted to shared storage, currently
supporting S3, OSS, COS, GCS, Azure Blob, MinIO, BOS, and HDFS.
+
+[](https://private-user-images.githubusercontent.com/6919662/346635704-76ee422f-95a8-48a0-808c-08817e5e46c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodW
IuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMzkyNTcsIm5iZiI6MTcyMjMzODk1NywicGF0aCI6Ii82OTE5NjYyLzM0NjYzNTcwNC03NmVlNDIyZi05NWE4LTQ4YTAtODA4Yy0wODgxN2U1ZTQ2YzgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzBUMTEyOTE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTgxMGI4NTcxNzM0Y2JiZDAyNDBhZGU1ZmQwNmQzOGUzYzU4MjY3N2JjNTEwOTM4MDljNDZmMDdmZDA5NzFlZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.K0UF5zzpp-qxY4YpU8vOxDfjn1VexTpiyxcl5Bv5Puc)
+
+
+Production Insights
+
+**1. Operations-Friendly**
+
+- Cache TTL: In real-time scenarios, setting the Cache TTL ensures that data
within the TTL provides low-latency queries from the local cache.
+
+- Seamless Scaling: One benefit of the compute-storage decoupled mode is the
rapid elasticity of computing resources. Doris uses progressive active cache
preheating technology, ensuring no impact on ongoing queries and imports during
scalability periods.
+
+- Seamless Upgrades: During upgrades, the multi-process mode operation ensures
ongoing queries and imports do not fail.
+
+
+
+**2. Load Isolation**
+
+- Multiple Computing Clusters: Different business characteristics (e.g., batch
processing, real-time) or business units (different businesses within a group)
can use independent computing clusters for physical isolation.
+
+- Read-Write Separation: Imports and queries can be completed by different
computing clusters, supporting automatic (notification for preheating during
import) and manual (periodic manual preheating) methods to preheat the query
cluster's cache.
+
+
+
+**3. Positive Data Deletion**
+
+In distributed systems, common data deletion involves reverse GC, comparing
files in shared storage with those recorded in metadata and deleting unrecorded
files. This method can easily lead to data deletion errors under concurrent
conditions. Relying on MetaService's transaction capabilities, Doris adopts
asynchronous forward data deletion technology. Data generation is bound to a
transaction, as is metadata deletion, and the asynchronous deletion process
simply deletes data based on transaction records, eliminating the need for
reverse GC.
+
+
+
+## Optimizer
+
+- All DQL and DML, except for load operations, will use the new optimizer.
Default settings disable fallback to the old optimizer, which will be removed
in subsequent versions.
Review Comment:
Suggestion: Elaborate on what the new optimizer is / link to docs to learn
more.
##########
versioned_docs/version-3.0/releasenotes/release-3.0.0.md:
##########
@@ -4,3 +4,166 @@
"language": "en"
}
---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Thanks to our devoted developers and supportive community users, the
much-expected Apache Doris 3.0.0 is now available!
+
+## Compute-Storage Decoupled
+
+From version 3.0.0, Doris supports a compute-storage decoupled mode, allowing
users to physically isolate business workloads and separate read and write
operations through multiple computing clusters. This mode substantially reduces
storage costs by utilizing mature object storage or HDFS. It mitigates
operational complexities such as disk balancing and data loss due to multiple
BE nodes going offline.
+
+**Future-Oriented Architecture**: The architecture is selected considering
complexity, performance, and scalability:
+
+**1. Reducing Complexity**
+
+- Decoupling OLAP layer semantics from storage implementation simplifies
system maintenance and reduces operational complexity.
+
+- Choosing production-verified mature storage systems.
+
+- Single source of truth for metadata, greatly simplifying system complexity,
such as avoiding write publishes in integrated storage-computation modes and
providing strong consistency in design.
+
+**2. Performance**
+
+Capable of achieving the performance of integrated storage-computation mode.
+
+**3. Scalability**
+
+- Data scale support is no longer limited by FE memory.
+
+- Architecture can evolve and integrate both integrated and separated storage
computation.
+
+- Easily supports time travel, data sharing, and load isolation.
+
+
+
+Based on these considerations, the architectural choices made include:
+
+1. A single metadata service divided into semantic and storage layers, with
the storage layer choosing production-verified and transaction-capable
FoundationDB.
+
+2. Storage options include mature, low-cost object storage and HDFS.
+
+3. To compensate for performance loss due to remote storage, local data
caching has been introduced at computing nodes, combining the simplicity of a
Shared Disk architecture with the performance of a Share Nothing architecture.
+
+
+The overall architecture is a three-layer structure: metadata layer, computing
layer, and data storage layer.
+
+- Metadata Layer: Provides the system's metadata services, such as database
tables, schemas, rowset metadata, and transactions. Currently, BE's metadata
has fully transitioned to MetaService, and FE's transaction metadata has
entered MetaService. Future versions will incorporate other metadata into
MetaService.
+
+- Computing Layer: Responsible for executing query plans and providing load
isolation through computing clusters. Computing nodes are completely stateless,
with local data acting as a cache for remote storage to accelerate queries.
+
+- Data Storage Layer: Data is persisted to shared storage, currently
supporting S3, OSS, COS, GCS, Azure Blob, MinIO, BOS, and HDFS.
+
+[](https://private-user-images.githubusercontent.com/6919662/346635704-76ee422f-95a8-48a0-808c-08817e5e46c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodW
IuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMzkyNTcsIm5iZiI6MTcyMjMzODk1NywicGF0aCI6Ii82OTE5NjYyLzM0NjYzNTcwNC03NmVlNDIyZi05NWE4LTQ4YTAtODA4Yy0wODgxN2U1ZTQ2YzgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzBUMTEyOTE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTgxMGI4NTcxNzM0Y2JiZDAyNDBhZGU1ZmQwNmQzOGUzYzU4MjY3N2JjNTEwOTM4MDljNDZmMDdmZDA5NzFlZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.K0UF5zzpp-qxY4YpU8vOxDfjn1VexTpiyxcl5Bv5Puc)
+
+
+Production Insights
+
+**1. Operations-Friendly**
+
+- Cache TTL: In real-time scenarios, setting the Cache TTL ensures that data
within the TTL provides low-latency queries from the local cache.
+
+- Seamless Scaling: One benefit of the compute-storage decoupled mode is the
rapid elasticity of computing resources. Doris uses progressive active cache
preheating technology, ensuring no impact on ongoing queries and imports during
scalability periods.
+
+- Seamless Upgrades: During upgrades, the multi-process mode operation ensures
ongoing queries and imports do not fail.
+
+
+
+**2. Load Isolation**
+
+- Multiple Computing Clusters: Different business characteristics (e.g., batch
processing, real-time) or business units (different businesses within a group)
can use independent computing clusters for physical isolation.
+
+- Read-Write Separation: Imports and queries can be completed by different
computing clusters, supporting automatic (notification for preheating during
import) and manual (periodic manual preheating) methods to preheat the query
cluster's cache.
+
+
+
+**3. Positive Data Deletion**
+
+In distributed systems, common data deletion involves reverse GC, comparing
files in shared storage with those recorded in metadata and deleting unrecorded
files. This method can easily lead to data deletion errors under concurrent
conditions. Relying on MetaService's transaction capabilities, Doris adopts
asynchronous forward data deletion technology. Data generation is bound to a
transaction, as is metadata deletion, and the asynchronous deletion process
simply deletes data based on transaction records, eliminating the need for
reverse GC.
+
+
+
+## Optimizer
+
+- All DQL and DML, except for load operations, will use the new optimizer.
Default settings disable fallback to the old optimizer, which will be removed
in subsequent versions.
+
+- Comprehensive top-N runtime filter applicable to all query modes, enhancing
performance when a top-N operator is present.
+
+- Experimental Feature: Refactored logic for synchronized materialized view
selection, transitioning from RBO to CBO to maintain consistency with
asynchronous materialized views. If issues arise, the switch `set global
enable_sync_mv_cost_based_rewrite = false` can be used to revert to RBO mode.
+
+- Experimental Feature: Partition-level statistics. By activating the switch
`set global enable_partition_analyze = true`, partition-level statistics can be
collected and utilized, improving query performance in cases of data partition
skew.
+
+- Experimental Feature: Generated columns. When creating tables, columns can
be designated as generated columns, which automatically compute results based
on defined expressions at write time, offering more complex expressions
compared to default values but disallowing explicit writing of specified values.
+
+## Execution Engine
+
+- Experimental Feature: Spill capability for core operators such as join,
aggregate, and sort has reached GA usable status.
+
+- Support for UDTFs.
Review Comment:
Suggestion: Link to information about UDTFs
##########
versioned_docs/version-3.0/releasenotes/release-3.0.0.md:
##########
@@ -4,3 +4,166 @@
"language": "en"
}
---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Thanks to our devoted developers and supportive community users, the
much-expected Apache Doris 3.0.0 is now available!
+
+## Compute-Storage Decoupled
+
+From version 3.0.0, Doris supports a compute-storage decoupled mode, allowing
users to physically isolate business workloads and separate read and write
operations through multiple computing clusters. This mode substantially reduces
storage costs by utilizing mature object storage or HDFS. It mitigates
operational complexities such as disk balancing and data loss due to multiple
BE nodes going offline.
+
+**Future-Oriented Architecture**: The architecture is selected considering
complexity, performance, and scalability:
+
+**1. Reducing Complexity**
+
+- Decoupling OLAP layer semantics from storage implementation simplifies
system maintenance and reduces operational complexity.
+
+- Choosing production-verified mature storage systems.
+
+- Single source of truth for metadata, greatly simplifying system complexity,
such as avoiding write publishes in integrated storage-computation modes and
providing strong consistency in design.
+
+**2. Performance**
+
+Capable of achieving the performance of integrated storage-computation mode.
+
+**3. Scalability**
+
+- Data scale support is no longer limited by FE memory.
+
+- Architecture can evolve and integrate both integrated and separated storage
computation.
+
+- Easily supports time travel, data sharing, and load isolation.
+
+
+
+Based on these considerations, the architectural choices made include:
+
+1. A single metadata service divided into semantic and storage layers, with
the storage layer choosing production-verified and transaction-capable
FoundationDB.
+
+2. Storage options include mature, low-cost object storage and HDFS.
+
+3. To compensate for performance loss due to remote storage, local data
caching has been introduced at computing nodes, combining the simplicity of a
Shared Disk architecture with the performance of a Share Nothing architecture.
Review Comment:
If possible, information about the performance loss (with benchmarks) would
be helpful.
##########
versioned_docs/version-3.0/releasenotes/release-3.0.0.md:
##########
@@ -4,3 +4,166 @@
"language": "en"
}
---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Thanks to our devoted developers and supportive community users, the
much-expected Apache Doris 3.0.0 is now available!
+
+## Compute-Storage Decoupled
+
+From version 3.0.0, Doris supports a compute-storage decoupled mode, allowing
users to physically isolate business workloads and separate read and write
operations through multiple computing clusters. This mode substantially reduces
storage costs by utilizing mature object storage or HDFS. It mitigates
operational complexities such as disk balancing and data loss due to multiple
BE nodes going offline.
+
+**Future-Oriented Architecture**: The architecture is selected considering
complexity, performance, and scalability:
+
+**1. Reducing Complexity**
+
+- Decoupling OLAP layer semantics from storage implementation simplifies
system maintenance and reduces operational complexity.
+
+- Choosing production-verified mature storage systems.
+
+- Single source of truth for metadata, greatly simplifying system complexity,
such as avoiding write publishes in integrated storage-computation modes and
providing strong consistency in design.
+
+**2. Performance**
+
+Capable of achieving the performance of integrated storage-computation mode.
+
+**3. Scalability**
+
+- Data scale support is no longer limited by FE memory.
+
+- Architecture can evolve and integrate both integrated and separated storage
computation.
+
+- Easily supports time travel, data sharing, and load isolation.
+
+
+
+Based on these considerations, the architectural choices made include:
+
+1. A single metadata service divided into semantic and storage layers, with
the storage layer choosing production-verified and transaction-capable
FoundationDB.
+
+2. Storage options include mature, low-cost object storage and HDFS.
+
+3. To compensate for performance loss due to remote storage, local data
caching has been introduced at computing nodes, combining the simplicity of a
Shared Disk architecture with the performance of a Share Nothing architecture.
+
+
+The overall architecture is a three-layer structure: metadata layer, computing
layer, and data storage layer.
+
+- Metadata Layer: Provides the system's metadata services, such as database
tables, schemas, rowset metadata, and transactions. Currently, BE's metadata
has fully transitioned to MetaService, and FE's transaction metadata has
entered MetaService. Future versions will incorporate other metadata into
MetaService.
+
+- Computing Layer: Responsible for executing query plans and providing load
isolation through computing clusters. Computing nodes are completely stateless,
with local data acting as a cache for remote storage to accelerate queries.
+
+- Data Storage Layer: Data is persisted to shared storage, currently
supporting S3, OSS, COS, GCS, Azure Blob, MinIO, BOS, and HDFS.
+
+[](https://private-user-images.githubusercontent.com/6919662/346635704-76ee422f-95a8-48a0-808c-08817e5e46c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodW
IuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMzkyNTcsIm5iZiI6MTcyMjMzODk1NywicGF0aCI6Ii82OTE5NjYyLzM0NjYzNTcwNC03NmVlNDIyZi05NWE4LTQ4YTAtODA4Yy0wODgxN2U1ZTQ2YzgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzBUMTEyOTE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTgxMGI4NTcxNzM0Y2JiZDAyNDBhZGU1ZmQwNmQzOGUzYzU4MjY3N2JjNTEwOTM4MDljNDZmMDdmZDA5NzFlZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.K0UF5zzpp-qxY4YpU8vOxDfjn1VexTpiyxcl5Bv5Puc)
+
+
+Production Insights
+
+**1. Operations-Friendly**
+
+- Cache TTL: In real-time scenarios, setting the Cache TTL ensures that data
within the TTL provides low-latency queries from the local cache.
+
+- Seamless Scaling: One benefit of the compute-storage decoupled mode is the
rapid elasticity of computing resources. Doris uses progressive active cache
preheating technology, ensuring no impact on ongoing queries and imports during
scalability periods.
+
+- Seamless Upgrades: During upgrades, the multi-process mode operation ensures
ongoing queries and imports do not fail.
+
+
+
+**2. Load Isolation**
+
+- Multiple Computing Clusters: Different business characteristics (e.g., batch
processing, real-time) or business units (different businesses within a group)
can use independent computing clusters for physical isolation.
+
+- Read-Write Separation: Imports and queries can be completed by different
computing clusters, supporting automatic (notification for preheating during
import) and manual (periodic manual preheating) methods to preheat the query
cluster's cache.
+
+
+
+**3. Positive Data Deletion**
+
+In distributed systems, common data deletion involves reverse GC, comparing
files in shared storage with those recorded in metadata and deleting unrecorded
files. This method can easily lead to data deletion errors under concurrent
conditions. Relying on MetaService's transaction capabilities, Doris adopts
asynchronous forward data deletion technology. Data generation is bound to a
transaction, as is metadata deletion, and the asynchronous deletion process
simply deletes data based on transaction records, eliminating the need for
reverse GC.
+
+
+
+## Optimizer
+
+- All DQL and DML, except for load operations, will use the new optimizer.
Default settings disable fallback to the old optimizer, which will be removed
in subsequent versions.
+
+- Comprehensive top-N runtime filter applicable to all query modes, enhancing
performance when a top-N operator is present.
+
+- Experimental Feature: Refactored logic for synchronized materialized view
selection, transitioning from RBO to CBO to maintain consistency with
asynchronous materialized views. If issues arise, the switch `set global
enable_sync_mv_cost_based_rewrite = false` can be used to revert to RBO mode.
+
+- Experimental Feature: Partition-level statistics. By activating the switch
`set global enable_partition_analyze = true`, partition-level statistics can be
collected and utilized, improving query performance in cases of data partition
skew.
+
+- Experimental Feature: Generated columns. When creating tables, columns can
be designated as generated columns, which automatically compute results based
on defined expressions at write time, offering more complex expressions
compared to default values but disallowing explicit writing of specified values.
+
+## Execution Engine
+
+- Experimental Feature: Spill capability for core operators such as join,
aggregate, and sort has reached GA usable status.
Review Comment:
It's unclear to the reader what this means... is it experimental? or GA?
both?
I'd also link to the docs explaining.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
