This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 22cf933747ca79a3979e8019c24f477b66f5fedc
Author: Mingyu Chen <morning...@163.com>
AuthorDate: Fri Oct 13 10:50:00 2023 +0800
[improvement](resource-tag) limit the default user's resource tag to
'default' (#25331)
In previous, if user property `'resource_tags.location'` is not set, the
can use Backends with any resource tag.
It may confuse that when the DBA set part of Backends to resource group A,
then the current existing user
should not be able to use this group A util it's `'resource_tags.location'`
is set.
So in this PR, I change the behavior, that if user property
`'resource_tags.location'` is not set, it can only use the
Backends with `default` tag.
---
docs/en/docs/admin-manual/multi-tenant.md | 2 ++
docs/zh-CN/docs/admin-manual/multi-tenant.md | 2 ++
.../mysql/privilege/CommonUserProperties.java | 4 ++++
.../org/apache/doris/catalog/UserPropertyTest.java | 3 +++
.../apache/doris/planner/ResourceTagQueryTest.java | 2 +-
.../org/apache/doris/statistics/CacheTest.java | 26 +++++++++++++---------
.../nereids_p0/outfile/test_outfile_expr.groovy | 2 +-
7 files changed, 28 insertions(+), 13 deletions(-)
diff --git a/docs/en/docs/admin-manual/multi-tenant.md
b/docs/en/docs/admin-manual/multi-tenant.md
index d0402bbf445..dc6900e06ac 100644
--- a/docs/en/docs/admin-manual/multi-tenant.md
+++ b/docs/en/docs/admin-manual/multi-tenant.md
@@ -138,6 +138,8 @@ Node resource division refers to setting tags for BE nodes
in a Doris cluster, a
After the setting is complete, when user1 initiates a query on the
UserTable table, it will only access the data copy on the nodes in the
`group_a` resource group, and the query will only use the node computing
resources in the `group_a` resource group. The query of user3 can use copies
and computing resources in any resource group.
+ > Note: By default, the user's `resource_tags.location` attribute is
empty. In versions prior to 2.0.2 (inclusive), by default, users are not
restricted by tags and can use any resource group. After version 2.0.3, users
can only use the `default` resource group by default.
+
In this way, we have achieved physical resource isolation for different
user queries by dividing nodes and restricting user resource usage.
Furthermore, we can create different users for different business departments
and restrict each user from using different resource groups. In order to avoid
the use of resource interference between different business parts. For example,
there is a business table in the cluster that needs to be shared by all 9
business departments, but it is hoped [...]
On the other hand, for the isolation of online and offline tasks. We can
use resource groups to achieve this. For example, we can divide nodes into two
resource groups, Online and Offline. The table data is still stored in 3
copies, of which 2 copies are stored in the Online resource group, and 1 copy
is stored in the Offline resource group. The Online resource group is mainly
used for online data services with high concurrency and low latency. Some large
queries or offline ETL opera [...]
diff --git a/docs/zh-CN/docs/admin-manual/multi-tenant.md
b/docs/zh-CN/docs/admin-manual/multi-tenant.md
index a7ae8314f59..97a54eda612 100644
--- a/docs/zh-CN/docs/admin-manual/multi-tenant.md
+++ b/docs/zh-CN/docs/admin-manual/multi-tenant.md
@@ -138,6 +138,8 @@ FE 不参与用户数据的处理计算等工作，因此是一个资源消耗
设置完成后，user1 在发起对 UserTable 表的查询时，只会访问 `group_a` 资源组内节点上的数据副本，并且查询仅会使用
`group_a` 资源组内的节点计算资源。而 user3 的查询可以使用任意资源组内的副本和计算资源。
+ > 注：默认情况下，用户的 `resource_tags.location` 属性为空，在2.0.2（含）之前的版本中，默认情况下，用户不受 tag
的限制，可以使用任意资源组。在 2.0.3 版本之后，默认情况下，用户只能使用 `default` 资源组。
+
这样，我们通过对节点的划分，以及对用户的资源使用限制，实现了不同用户查询上的物理资源隔离。更进一步，我们可以给不同的业务部门创建不同的用户，并限制每个用户使用不同的资源组。以避免不同业务部分之间使用资源干扰。比如集群内有一张业务表需要共享给所有9个业务部门使用，但是希望能够尽量避免不同部门之间的资源抢占。则我们可以为这张表创建3个副本，分别存储在3个资源组中。接下来，我们为9个业务部门创建9个用户，每3个用户限制使用一个资源组。这样，资源的竞争程度就由9降低到了3。
另一方面，针对在线和离线任务的隔离。我们可以利用资源组的方式实现。比如我们可以将节点划分为 Online 和 Offline
两个资源组。表数据依然以3副本的方式存储，其中 2 个副本存放在 Online 资源组，1 个副本存放在 Offline 资源组。Online
资源组主要用于高并发低延迟的在线数据服务，而一些大查询或离线ETL操作，则可以使用 Offline
资源组中的节点执行。从而实现在统一集群内同时提供在线和离线服务的能力。
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
index cf36639c0a9..8fa28485ae5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
@@ -110,6 +110,10 @@ public class CommonUserProperties implements Writable {
}
public Set<Tag> getResourceTags() {
+ // If resource tags in user properties is empty, use default backend
tag.
+ if (resourceTags.isEmpty()) {
+ return Sets.newHashSet(Tag.DEFAULT_BACKEND_TAG);
+ }
return resourceTags;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/UserPropertyTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/UserPropertyTest.java
index 793cdf6a7dd..3142d75691a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/UserPropertyTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/UserPropertyTest.java
@@ -24,8 +24,10 @@ import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.load.DppConfig;
import org.apache.doris.mysql.privilege.UserProperty;
+import org.apache.doris.resource.Tag;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
import mockit.Expectations;
import mockit.Mocked;
import org.junit.Assert;
@@ -116,6 +118,7 @@ public class UserPropertyTest {
Assert.assertEquals(new String[]{"rule1", "rule2"},
userProperty.getSqlBlockRules());
Assert.assertEquals(2, userProperty.getCpuResourceLimit());
Assert.assertEquals(500, userProperty.getQueryTimeout());
+ Assert.assertEquals(Sets.newHashSet(Tag.DEFAULT_BACKEND_TAG),
userProperty.getCopiedResourceTags());
// fetch property
List<List<String>> rows = userProperty.fetchProperty();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/planner/ResourceTagQueryTest.java
b/fe/fe-core/src/test/java/org/apache/doris/planner/ResourceTagQueryTest.java
index 8bcd302a181..9bbc7ab55eb 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/planner/ResourceTagQueryTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/planner/ResourceTagQueryTest.java
@@ -196,7 +196,7 @@ public class ResourceTagQueryTest {
OlapTable tbl = (OlapTable) db.getTableNullable("tbl1");
Set<Tag> userTags =
Env.getCurrentEnv().getAuth().getResourceTags(Auth.ROOT_USER);
- Assert.assertEquals(0, userTags.size());
+ Assert.assertEquals(1, userTags.size());
// set default tag for root
String setPropStr = "set property for 'root' 'resource_tags.location'
= 'default';";
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
b/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
index d6570ecebc5..24c811d01f1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
@@ -294,16 +294,20 @@ public class CacheTest extends TestWithFeService {
null, null, false, null, new Date().toString());
}
};
- StatisticsCache statisticsCache = new StatisticsCache();
- ColumnStatistic columnStatistic =
statisticsCache.getColumnStatistics(1, 1, 1, "col");
- Thread.sleep(3000);
- columnStatistic = statisticsCache.getColumnStatistics(1, 1, 1, "col");
- Assertions.assertEquals(1, columnStatistic.count);
- Assertions.assertEquals(2, columnStatistic.ndv);
- Assertions.assertEquals(3, columnStatistic.avgSizeByte);
- Assertions.assertEquals(4, columnStatistic.numNulls);
- Assertions.assertEquals(5, columnStatistic.dataSize);
- Assertions.assertEquals(6, columnStatistic.minValue);
- Assertions.assertEquals(7, columnStatistic.maxValue);
+ try {
+ StatisticsCache statisticsCache = new StatisticsCache();
+ ColumnStatistic columnStatistic =
statisticsCache.getColumnStatistics(1, 1, 1, "col");
+ Thread.sleep(3000);
+ columnStatistic = statisticsCache.getColumnStatistics(1, 1, 1,
"col");
+ Assertions.assertEquals(1, columnStatistic.count);
+ Assertions.assertEquals(2, columnStatistic.ndv);
+ Assertions.assertEquals(3, columnStatistic.avgSizeByte);
+ Assertions.assertEquals(4, columnStatistic.numNulls);
+ Assertions.assertEquals(5, columnStatistic.dataSize);
+ Assertions.assertEquals(6, columnStatistic.minValue);
+ Assertions.assertEquals(7, columnStatistic.maxValue);
+ } catch (Throwable t) {
+ t.printStackTrace();
+ }
}
}
diff --git a/regression-test/suites/nereids_p0/outfile/test_outfile_expr.groovy
b/regression-test/suites/nereids_p0/outfile/test_outfile_expr.groovy
index 6e539ad337b..7ac4b1b43b2 100644
--- a/regression-test/suites/nereids_p0/outfile/test_outfile_expr.groovy
+++ b/regression-test/suites/nereids_p0/outfile/test_outfile_expr.groovy
@@ -105,7 +105,7 @@ suite("test_outfile_expr") {
SELECT user_id+1, age+sex, repeat(char_col, 10) FROM ${tableName}
t ORDER BY user_id INTO OUTFILE "file://${outFile}/";
"""
- url = result[0][3]
+ def url = result[0][3]
urlHost = url.substring(8, url.indexOf("${outFile}"))
def filePrifix = url.split("${outFile}")[1]
csvFiles = "${outFile}${filePrifix}*.csv"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
