This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 58bdc237a0b [update](blog) Update UMS Blog and Qifu Blog on relevant
pages (#386)
58bdc237a0b is described below
commit 58bdc237a0b85fa4071be8df03882aa1447bc573
Author: KassieZ <139741991+kass...@users.noreply.github.com>
AuthorDate: Wed Jan 10 10:19:02 2024 +0800
[update](blog) Update UMS Blog and Qifu Blog on relevant pages (#386)
---
...al-time-data-warehouse-based-on-apache-doris.md | 151 +++++++++++++++++++++
...ta-reporting-tagging-and-data-lake-analytics.md | 2 +-
...expect-from-apache-doris-as-a-data-warehouse.md | 2 +-
...s-a-next-generation-real-time-data-warehouse.md | 2 +-
blog/log-analysis-elasticsearch-vs-apache-doris.md | 2 -
src/constant/newsletter.data.ts | 14 +-
src/constant/users.data.json | 2 +-
src/constant/various-analytics.data.tsx | 4 +
...d-highly-available-real-time-data-warehouse.png | Bin 0 -> 307462 bytes
9 files changed, 166 insertions(+), 13 deletions(-)
diff --git
a/blog/a-fast-secure-high-available-real-time-data-warehouse-based-on-apache-doris.md
b/blog/a-fast-secure-high-available-real-time-data-warehouse-based-on-apache-doris.md
new file mode 100644
index 00000000000..bb0b035a5e0
--- /dev/null
+++
b/blog/a-fast-secure-high-available-real-time-data-warehouse-based-on-apache-doris.md
@@ -0,0 +1,151 @@
+---
+{
+ 'title': "The financial sector's choice: fast, secure, and highly
available real-time data warehousing based on Apache Doris",
+ 'summary': "A whole-journey guide for financial users looking for fast
data processing performance, data security, and high service availability.",
+ 'date': '2024-01-08',
+ 'author': 'Apache Doris',
+ 'tags': ['Best Practice'],
+ 'picked': "true",
+ 'order': "1",
+ "image":
'/images/apache-doris-a-fast-secure-and-highly-available-real-time-data-warehouse.png'
+}
+
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+This is a whole-journey guide for Apache Doris users, especially those from
the financial sector which requires a high level of data security and
availability. If you don't know how to build a real-time data pipeline and make
the most of the [Apache Doris](https://doris.apache.org/) functionalities,
start with this post and you will be loaded with inspiration after reading.
+
+This is the best practice of a non-banking payment service provider that
serves over 25 million retailers and processes data from 40 million end
devices. Data sources include MySQL, Oracle, and MongoDB. They were using
Apache Hive as an offline data warehouse but feeling the need to add a
real-time data processing pipeline. **After introducing Apache Doris, they
increase their data ingestion speed by 2~5 times, ETL performance by 3~12
times, and query execution speed by 10~15 times.**
+
+In this post, you will learn how to integrate Apache Doris into your data
architecture, including how to arrange data inside Doris, how to ingest data
into it, and how to enable efficient data updates. Plus, you will learn about
the enterprise features that Apache Doris provides to guarantee data security,
system stability, and service availability.
+
+
+
+## Building a real-time data warehouse with Apache Doris
+### Choice of data models
+
+Apache Doris arranges data with three data models. The main difference between
these models lies in whether or how they aggregate data.
+
+- **[Duplicate Key
model](https://doris.apache.org/docs/data-table/data-model#duplicate-model)**:
for detailed data queries. It supports ad-hoc queries of any dimension.
+- **[Unique Key
model](https://doris.apache.org/docs/data-table/data-model#unique-model)**: for
use cases with data uniqueness constraints. It supports precise deduplication,
multi-stream upserts, and partial column updates.
+- **[Aggregate Key
model](https://doris.apache.org/docs/data-table/data-model#aggregate-model)**:
for data reporting. It accelerates data reporting by pre-aggregating data.
+
+The financial user adopts different data models in different data warehouse
layers:
+
+- **ODS - Duplicate Key model**: As a payment service provider, the user
receives a million settlement data every day. Since the settlement cycle can
span a whole year, the relevant data needs to be kept intact for a year. Thus,
the proper way is to put it in the Duplicate Key model, which does not perform
any data aggregations. An exception is that some data is prone to constant
changes, like order status from retailers. Such data should be put into the
Unique Key model so that the newl [...]
+- **DWD & DWS - Unique Key model**: Data in the DWD and DWS layers are further
abstracted, but it is all put in the Unique Key model so that the settlement
data can be automatically updated.
+- **ADS - Aggregate Key model**: Data is highly abstracted in this layer. It
is pre-aggregated to mitigate the computation load of downstream analytics.
+
+
+### Partitioning and bucketing strategies
+
+The idea of partitioning and bucketing is to "cut" data into smaller pieces to
increase data processing speed. The key is to set an appropriate number of data
partitions and buckets. Based on their use case, the user tailors the bucketing
field and bucket number to each table. For example, they often need to query
the dimensional data of different retailers from the retailer flat table, so
they specify the retailer ID column as the bucketing field, and list the
recommended bucket number [...]
+
+
+
+### Multi-source data migration
+
+In the adoption of Apache Doris, the user had to migrate all local data from
their branches into Doris, which was when they found out their branches were
using **different databases** and had **data files of very different formats**,
so the migration could be a mess.
+
+
+
+Luckily, Apache Doris supports a rich collection of data integration methods
for both real-time data streaming and offline data import.
+
+- **Real-time data streaming**: Apache Doris fetches MySQL Binlogs in real
time. Part of them is written into Doris directly via Flink CDC, while the
high-volume ones are synchronized into Kafka for peak shaving, and then written
into Doris via the Flink-Doris-Connector.
+- **Offline data import**: This includes more diversified data sources and
data formats. Historical data and incremental data from S3 and HDFS will be
ingested into Doris via the [Broker
Load](https://doris.apache.org/docs/data-operate/import/import-way/broker-load-manual)
method, data from Hive or JDBC will be synchronized to Doris via the [Insert
Into](https://doris.apache.org/docs/data-operate/import/import-way/insert-into-manual)
method, and files will be loaded to Doris via the Flin [...]
+
+### Full data ingestion and incremental data ingestion
+
+To ensure business continuity and data accuracy, the user figures out the
following ways to ingest full data and incremental data:
+
+- **Full data ingestion**: Create a temporary table of the target schema in
Doris, ingest full data into the temporary table, and then use the `ALTER TABLE
t1 REPLACE WITH TABLE t2` statement for atomic replacement of the regular table
with the temporary table. This method prevents interruptions to queries on the
frontend.
+
+```SQL
+alter table ${DB_NAME}.${TBL_NAME} drop partition IF EXISTS p${P_DOWN_DATE};
+ALTER TABLE ${DB_NAME}.${TBL_NAME} ADD PARTITION IF NOT EXISTS
p${P_DOWN_DATE} VALUES [('${P_DOWN_DATE}'), ('${P_UP_DATE}'));
+
+LOAD LABEL ${TBL_NAME}_${load_timestamp} ...
+```
+
+- **Incremental data ingestion**: Create a new data partition to accommodate
incremental data.
+
+### Offline data processing
+
+The user has moved their offline data processing workload to Apache Doris and
thus **increased execution speed by 5 times**.
+
+
+
+- **Before**: The old Hive-based offline data warehouse used the TEZ execution
engine to process 30 million new data records every day. With 2TB computation
resources, the whole pipeline took 2.5 hours.
+- **After**: Apache Doris finishes the same tasks within only 30 minutes and
consumes only 1TB. Script execution takes only 10 seconds instead of 8 minutes.
+
+## Enterprise features for financial players
+
+### Multi-tenant resource isolation
+
+This is required because it often happens that the same piece of data is
requested by multiple teams or business systems. These tasks can lead to
resource preemption and thus performance decrease and system instability.
+
+**Resource limit for different workloads**
+
+The user classifies their analytics workloads into four types and sets a
resource limit for each of them. In particular, they have four different types
of Doris accounts and set a limit on the CPU and memory resources for each type
of account.
+
+
+
+In this way, when one tenant requires excessive resources, it will only
compromise its own efficiency but not affect other tenants.
+
+**Resource tag-based isolation**
+
+For data security under the parent-subsidiary company hierarchy, the user has
set isolated resource groups for the subsidiaries. Data of each subsidiary is
stored in its own resource group with three replicas, while data of the parent
company is stored with four replicas: three in the parent company resource
group, and the other one in the subsidiary resource group. Thus, when an
employee from a subsidiary requests data from the parent company, the query
will only executed in the subsidi [...]
+
+
+
+**Workload group**
+
+The resource tag-based isolation plan ensures isolation on a physical level,
but as Apache Doris developers, we want to further optimize resource
utilization and pursue more fine-grained resource isolation. For these
purposes, we released the [Workload
Group](https://doris.apache.org/docs/admin-manual/workload-group) feature in
[Apache Doris 2.0](https://doris.apache.org/blog/release-note-2.0.0).
+
+The Workload Group mechanism relates queries to workload groups, which limit
the share of CPU and memory resources of the backend nodes that a query can
use. When cluster resources are in short supply, the biggest queries will stop
execution. On the contrary, when there are plenty of available cluster
resources and a workload group requires more resources than the limit, it will
get assigned with the idle resources proportionately.
+
+The user is actively planning their transition to the Workload Group plan and
utilizing the task prioritizing mechanism and query queue feature to organize
the execution order.
+
+**Fine-grained user privilege management**
+
+For regulation and compliance reasons, this payment service provider
implements strict privilege control to make sure that everyone only has access
to what they are supposed to access. This is how they do it:
+
+- **User privilege setting**: System users of different subsidiaries or with
different business needs are granted different data access privileges.
+- **Privilege control over databases, tables, and rows**: The `ROW POLICY`
mechanism of Apache Doris makes these operations easy.
+- **Privilege control over columns**: This is done by creating views.
+
+
+
+### Cluster stability guarantee
+
+- **Circuit Breaking**: From time to time, system users might input faulty
SQL, causing excessive resource consumption. A circuit-breaking mechanism is in
place for that. It will promptly stop these resource-intensive queries and
prevent interruption to the system.
+- **Data ingestion concurrency control**: The user has a frequent need to
integrate historical data into their data platform. That involves a lot of data
modification tasks and might stress the cluster. To solve that, they turn on
the
[Merge-on-Write](https://doris.apache.org/docs/data-table/data-model#merge-on-write-of-unique-model)
mode in the Unique Key model, enable [Vertical
Compaction](https://doris.apache.org/docs/advanced/best-practice/compaction#vertical-compaction)
and [Segment [...]
+- **Network traffic control**: Considering their two clusters in different
cities, they employ Quality of Service (QoS) strategies tailored to different
scenarios for precise network isolation and ensuring network quality and
stability.
+- **Monitoring and alerting**: The user has integrated Doris with their
internal monitoring and alerting platform so any detected issues will be
notified via their messaging software and email in real time.
+
+### Cross-cluster replication
+
+Disaster recovery is crucial for the financial industry. The user leverages
the Cross-Cluster Replication (CCR) capability and builds a dual-cluster
solution. As the primary cluster undertakes all the queries, the major business
data is also synchronized into the backup cluster and updated in real time, so
that in the case of service downtime in the primary cluster, the backup cluster
will take over swiftly and ensure business continuity.
+
+## Conclusion
+
+We appreciate the user for their active
[communication](https://join.slack.com/t/apachedoriscommunity/shared_invite/zt-1t3wfymur-0soNPATWQ~gbU8xutFOLog)
with us along the way and are glad to see so many Apache Doris features fit in
their needs. They are also planning on exploring federated query,
compute-storage separation, and auto maintenance with Apache Doris. We look
forward to more best practice and feedback from them.
\ No newline at end of file
diff --git
a/blog/apache-doris-speeds-up-data-reporting-tagging-and-data-lake-analytics.md
b/blog/apache-doris-speeds-up-data-reporting-tagging-and-data-lake-analytics.md
index 64e5e5c2f4c..9fb2e466b84 100644
---
a/blog/apache-doris-speeds-up-data-reporting-tagging-and-data-lake-analytics.md
+++
b/blog/apache-doris-speeds-up-data-reporting-tagging-and-data-lake-analytics.md
@@ -6,7 +6,7 @@
'author': 'Apache Doris',
'tags': ['Best Practice'],
'picked': "true",
- 'order': "1",
+ 'order': "2",
"image":
'/images/apache-doris-speeds-up-data-reporting-data-lake-analytics.jpg'
}
diff --git
a/blog/apache-doris-summit-asia-2023-what-can-you-expect-from-apache-doris-as-a-data-warehouse.md
b/blog/apache-doris-summit-asia-2023-what-can-you-expect-from-apache-doris-as-a-data-warehouse.md
index 56f700ca510..abaafe1aa9e 100644
---
a/blog/apache-doris-summit-asia-2023-what-can-you-expect-from-apache-doris-as-a-data-warehouse.md
+++
b/blog/apache-doris-summit-asia-2023-what-can-you-expect-from-apache-doris-as-a-data-warehouse.md
@@ -6,7 +6,7 @@
'author': 'Apache Doris',
'tags': ['Top News'],
'picked': "true",
- 'order': "2",
+ 'order': "3",
"image": '/images/doris-summit-asia.png'
}
diff --git
a/blog/introduction-to-apache-doris-a-next-generation-real-time-data-warehouse.md
b/blog/introduction-to-apache-doris-a-next-generation-real-time-data-warehouse.md
index bd2653fc7b7..e303be90192 100644
---
a/blog/introduction-to-apache-doris-a-next-generation-real-time-data-warehouse.md
+++
b/blog/introduction-to-apache-doris-a-next-generation-real-time-data-warehouse.md
@@ -6,7 +6,7 @@
'author': 'Apache Doris',
'tags': ['Tech Sharing'],
'picked': "true",
- 'order': "3",
+ 'order': "4",
"image": '/images/doris-intro.png'
}
diff --git a/blog/log-analysis-elasticsearch-vs-apache-doris.md
b/blog/log-analysis-elasticsearch-vs-apache-doris.md
index 1868a545beb..8bf2e2bee35 100644
--- a/blog/log-analysis-elasticsearch-vs-apache-doris.md
+++ b/blog/log-analysis-elasticsearch-vs-apache-doris.md
@@ -5,8 +5,6 @@
'date': '2023-09-28',
'author': 'Apache Doris',
'tags': ['Tech Sharing'],
- 'picked': "true",
- 'order': "4",
"image": '/images/es-vs-doris.png'
}
diff --git a/src/constant/newsletter.data.ts b/src/constant/newsletter.data.ts
index 816cacef12e..893731a2508 100644
--- a/src/constant/newsletter.data.ts
+++ b/src/constant/newsletter.data.ts
@@ -1,4 +1,11 @@
export const NEWSLETTER_DATA = [
+ {
+ tags: ['Best Practice'],
+ title: "The financial sector's choice: fast, secure, and highly
available real-time data warehousing based on Apache Doris",
+ content: `A whole-journey guide for financial users looking for fast
data processing performance, data security, and high service availability.`,
+ to:
'/blog/a-fast-secure-high-available-real-time-data-warehouse-based-on-apache-doris',
+ image:
'apache-doris-a-fast-secure-and-highly-available-real-time-data-warehouse.png',
+ },
{
tags: ['Best Practice'],
title: 'Apache Doris speeds up data reporting, tagging, and data lake
analytics',
@@ -21,12 +28,5 @@ export const NEWSLETTER_DATA = [
'This is a technical overview of Apache Doris, introducing how it
enables fast query performance with its architectural design, features, and
mechanisms.',
to:
'/blog/introduction-to-apache-doris-a-next-generation-real-time-data-warehouse',
image: 'newsletter-2.png',
- },
- {
- tags: ['Tech Sharing'],
- title: 'Log analysis: Elasticsearch vs Apache Doris',
- content: `As a major part of a company's data asset, logs brings
values to businesses in three aspects: system observability, cyber security,
and data analysis. They are your first resort for troubleshooting, your
reference for improving system security, and your data mine where you can
extract information that points to business growth.`,
- to: '/blog/log-analysis-elasticsearch-vs-apache-doris',
- image: 'newsletter-3.png',
}
];
diff --git a/src/constant/users.data.json b/src/constant/users.data.json
index 19366e36b49..5d27d45f84a 100644
--- a/src/constant/users.data.json
+++ b/src/constant/users.data.json
@@ -977,7 +977,7 @@
"category": "Finance",
"logo": "UMS",
"order": 108,
- "to": null,
+ "to":
"https://doris.apache.org/blog/a-fast-secure-high-available-real-time-data-warehouse-based-on-apache-doris";,
"image": "/images/user-logo/Finance/UMS.jpg"
},
{
diff --git a/src/constant/various-analytics.data.tsx
b/src/constant/various-analytics.data.tsx
index 42b94ec17b7..010a02c16ba 100644
--- a/src/constant/various-analytics.data.tsx
+++ b/src/constant/various-analytics.data.tsx
@@ -54,6 +54,10 @@ export const VariousAnalyticsData = [
content: 'Deep dive',
to:
'/blog/Building-the-Next-Generation-Data-Lakehouse-10X-Performance',
},
+ {
+ content: 'Read use case',
+ to:
'/blog/apache-doris-speeds-up-data-reporting-tagging-and-data-lake-analytics',
+ },
],
icon: (
<svg xmlns="http://www.w3.org/2000/svg"; width="16" height="16"
viewBox="0 0 16 16" fill="none">
diff --git
a/static/images/apache-doris-a-fast-secure-and-highly-available-real-time-data-warehouse.png
b/static/images/apache-doris-a-fast-secure-and-highly-available-real-time-data-warehouse.png
new file mode 100644
index 00000000000..9fb6a9ffa18
Binary files /dev/null and
b/static/images/apache-doris-a-fast-secure-and-highly-available-real-time-data-warehouse.png
differ
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
