[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455020216 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on code in PR #17424:
URL: https://github.com/apache/doris/pull/17424#discussion_r1125617536
##
fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/RangerHiveAccessController.java:
##
@@ -33,33 +34,49 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
public class RangerHiveAccessController implements CatalogAccessController {
public static final String CLIENT_TYPE_DORIS = "doris";
private static final Logger LOG =
LogManager.getLogger(RangerHiveAccessController.class);
+private static ScheduledThreadPoolExecutor logFlushTimer =
ThreadPoolManager.newDaemonScheduledThreadPool(1,
+"ranger-hive-audit-log-flusher-timer", true);
private RangerHivePlugin hivePlugin;
private RangerHiveAuditHandler auditHandler;
public RangerHiveAccessController(Map properties) {
String serviceName = properties.get("ranger.service.name");
hivePlugin = new RangerHivePlugin(serviceName);
auditHandler = new RangerHiveAuditHandler(hivePlugin.getConfig());
+//start a timed log flusher
+logFlushTimer.scheduleAtFixedRate(new
RangerHiveAuditLogFlusher(auditHandler), 10, 20L, TimeUnit.SECONDS);
}
private RangerAccessRequestImpl createRequest(UserIdentity currentUser,
HiveAccessType accessType) {
RangerAccessRequestImpl request = new RangerAccessRequestImpl();
-request.setUser(currentUser.getQualifiedUser());
-request.setUserRoles(currentUser.getRoles());
+// currentUser.getQualifiedUser() is as of form:
default_cluster:user1, only use `user1`
+String[] userArray = currentUser.getQualifiedUser().split(":");
+request.setUser(userArray[1]);
+request.setClusterName(userArray[0]);
+Set roles = new HashSet<>();
+for (String role : currentUser.getRoles()) {
+// some roles are as of form: default_role_rbac_test@%, only use
`default_role_rbac_test`
+roles.add(role.split("@")[0]);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on code in PR #17424:
URL: https://github.com/apache/doris/pull/17424#discussion_r1125617736
##
fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/RangerHiveAccessController.java:
##
@@ -33,33 +34,49 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
public class RangerHiveAccessController implements CatalogAccessController {
public static final String CLIENT_TYPE_DORIS = "doris";
private static final Logger LOG =
LogManager.getLogger(RangerHiveAccessController.class);
+private static ScheduledThreadPoolExecutor logFlushTimer =
ThreadPoolManager.newDaemonScheduledThreadPool(1,
+"ranger-hive-audit-log-flusher-timer", true);
private RangerHivePlugin hivePlugin;
private RangerHiveAuditHandler auditHandler;
public RangerHiveAccessController(Map properties) {
String serviceName = properties.get("ranger.service.name");
hivePlugin = new RangerHivePlugin(serviceName);
auditHandler = new RangerHiveAuditHandler(hivePlugin.getConfig());
+//start a timed log flusher
+logFlushTimer.scheduleAtFixedRate(new
RangerHiveAuditLogFlusher(auditHandler), 10, 20L, TimeUnit.SECONDS);
}
private RangerAccessRequestImpl createRequest(UserIdentity currentUser,
HiveAccessType accessType) {
RangerAccessRequestImpl request = new RangerAccessRequestImpl();
-request.setUser(currentUser.getQualifiedUser());
-request.setUserRoles(currentUser.getRoles());
+// currentUser.getQualifiedUser() is as of form:
default_cluster:user1, only use `user1`
+String[] userArray = currentUser.getQualifiedUser().split(":");
+request.setUser(userArray[1]);
+request.setClusterName(userArray[0]);
+Set roles = new HashSet<>();
+for (String role : currentUser.getRoles()) {
+// some roles are as of form: default_role_rbac_test@%, only use
`default_role_rbac_test`
+roles.add(role.split("@")[0]);
+}
+request.setUserRoles(roles);
request.setAction(accessType.name());
if (accessType == HiveAccessType.USE) {
request.setAccessType(RangerPolicyEngine.ANY_ACCESS);
} else {
request.setAccessType(accessType.name().toLowerCase());
}
request.setClientIPAddress(currentUser.getHost());
+request.setClusterType(CLIENT_TYPE_DORIS);
Review Comment:
used for ranger audit, user does not need to set it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on code in PR #17424:
URL: https://github.com/apache/doris/pull/17424#discussion_r1125617875
##
fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/RangerHiveAccessController.java:
##
@@ -99,20 +112,20 @@ private boolean checkPrivilege(UserIdentity currentUser,
HiveAccessType accessTy
request.setResource(resource);
RangerAccessResult result = hivePlugin.isAccessAllowed(request,
auditHandler);
-auditHandler.flushAudit();
if (result == null) {
-LOG.warn(String.format("Error getting authorizer result, please
check your ranger config. Request: %s",
-request));
+LOG.warn(String.format("Error getting authorizer result, please
check your ranger config. Make sure "
++ "ranger policy engine is initialized. Request: %s",
request));
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17427: [fix](compatibility) fix unsigned int type compatibility value scope problem
ZhangYu0123 commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455020982 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 opened a new pull request, #17428: Update data-faq.md
ZhangYu0123 opened a new pull request, #17428: URL: https://github.com/apache/doris/pull/17428 Already support rename column. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on code in PR #17424:
URL: https://github.com/apache/doris/pull/17424#discussion_r1125620132
##
fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/RangerHiveAccessController.java:
##
@@ -33,33 +34,49 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
public class RangerHiveAccessController implements CatalogAccessController {
public static final String CLIENT_TYPE_DORIS = "doris";
private static final Logger LOG =
LogManager.getLogger(RangerHiveAccessController.class);
+private static ScheduledThreadPoolExecutor logFlushTimer =
ThreadPoolManager.newDaemonScheduledThreadPool(1,
+"ranger-hive-audit-log-flusher-timer", true);
private RangerHivePlugin hivePlugin;
private RangerHiveAuditHandler auditHandler;
public RangerHiveAccessController(Map properties) {
String serviceName = properties.get("ranger.service.name");
hivePlugin = new RangerHivePlugin(serviceName);
auditHandler = new RangerHiveAuditHandler(hivePlugin.getConfig());
+//start a timed log flusher
+logFlushTimer.scheduleAtFixedRate(new
RangerHiveAuditLogFlusher(auditHandler), 10, 20L, TimeUnit.SECONDS);
}
private RangerAccessRequestImpl createRequest(UserIdentity currentUser,
HiveAccessType accessType) {
RangerAccessRequestImpl request = new RangerAccessRequestImpl();
-request.setUser(currentUser.getQualifiedUser());
-request.setUserRoles(currentUser.getRoles());
+// currentUser.getQualifiedUser() is as of form:
default_cluster:user1, only use `user1`
+String[] userArray = currentUser.getQualifiedUser().split(":");
+request.setUser(userArray[1]);
+request.setClusterName(userArray[0]);
+Set roles = new HashSet<>();
+for (String role : currentUser.getRoles()) {
+// some roles are as of form: default_role_rbac_test@%, only use
`default_role_rbac_test`
+roles.add(role.split("@")[0]);
+}
+request.setUserRoles(roles);
request.setAction(accessType.name());
if (accessType == HiveAccessType.USE) {
request.setAccessType(RangerPolicyEngine.ANY_ACCESS);
} else {
request.setAccessType(accessType.name().toLowerCase());
}
request.setClientIPAddress(currentUser.getHost());
+request.setClusterType(CLIENT_TYPE_DORIS);
Review Comment:
it's a constant defined in RangerHiveAccessController
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17428: [docs](typo) fix docs, already support rename column.
ZhangYu0123 commented on PR #17428: URL: https://github.com/apache/doris/pull/17428#issuecomment-1455024687 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #17330: [Refactor](map) remove using column array in map to reduce offset column
xiaokang commented on code in PR #17330:
URL: https://github.com/apache/doris/pull/17330#discussion_r1125607634
##
be/src/vec/data_types/data_type_map.cpp:
##
@@ -21,12 +21,9 @@
namespace doris::vectorized {
-DataTypeMap::DataTypeMap(const DataTypePtr& keys_, const DataTypePtr& values_)
{
-key_type = make_nullable(keys_);
-value_type = make_nullable(values_);
-
-keys = std::make_shared(key_type);
-values = std::make_shared(value_type);
+DataTypeMap::DataTypeMap(const DataTypePtr& key, const DataTypePtr& value) {
Review Comment:
key -> key_type_
##
be/src/vec/columns/column_map.cpp:
##
@@ -25,50 +25,108 @@ namespace doris::vectorized {
/** A column of map values.
*/
std::string ColumnMap::get_name() const {
-return "Map(" + keys->get_name() + ", " + values->get_name() + ")";
+return "Map(" + keys_column->get_name() + ", " + values_column->get_name()
+ ")";
}
-ColumnMap::ColumnMap(MutableColumnPtr&& keys, MutableColumnPtr&& values)
-: keys(std::move(keys)), values(std::move(values)) {
-check_size();
-}
+ColumnMap::ColumnMap(MutableColumnPtr&& keys, MutableColumnPtr&& values,
MutableColumnPtr&& offsets)
+: keys_column(std::move(keys)),
+ values_column(std::move(values)),
+ offsets_column(std::move(offsets)) {
+const COffsets* offsets_concrete = typeid_cast(offsets_column.get());
-ColumnArray::Offsets64& ColumnMap::get_offsets() const {
-const ColumnArray& column_keys = assert_cast(get_keys());
-// todo . did here check size ?
-return const_cast(column_keys.get_offsets());
-}
+if (!offsets_concrete) {
+LOG(FATAL) << "offsets_column must be a ColumnUInt64";
+}
-void ColumnMap::check_size() const {
-const auto* key_array = typeid_cast(keys.get());
-const auto* value_array = typeid_cast(values.get());
-CHECK(key_array) << "ColumnMap keys can be created only from array";
-CHECK(value_array) << "ColumnMap values can be created only from array";
-CHECK_EQ(get_keys_ptr()->size(), get_values_ptr()->size());
+if (!offsets_concrete->empty() && keys && values) {
+auto last_offset = offsets_concrete->get_data().back();
+
+/// This will also prevent possible overflow in offset.
+if (keys_column->size() != last_offset) {
+LOG(FATAL) << "offsets_column has data inconsistent with
key_column";
+}
+if (values_column->size() != last_offset) {
+LOG(FATAL) << "offsets_column has data inconsistent with
value_column";
+}
+}
}
// todo. here to resize every row map
MutableColumnPtr ColumnMap::clone_resized(size_t to_size) const {
-auto res = ColumnMap::create(keys->clone_resized(to_size),
values->clone_resized(to_size));
+auto res = ColumnMap::create(get_keys().clone_empty(),
get_values().clone_empty(),
+ COffsets::create());
+if (to_size == 0) {
+return res;
+}
+
+size_t from_size = size();
+
+if (to_size <= from_size) {
+res->get_offsets().assign(get_offsets().begin(), get_offsets().begin()
+ to_size);
+res->get_keys().insert_range_from(get_keys(), 0, get_offsets()[to_size
- 1]);
+res->get_values().insert_range_from(get_values(), 0,
get_offsets()[to_size - 1]);
+} else {
+/// Copy column and append empty arrays for extra elements.
+Offset64 offset = 0;
+if (from_size > 0) {
+res->get_offsets().assign(get_offsets().begin(),
get_offsets().end());
+res->get_keys().insert_range_from(get_keys(), 0,
get_keys().size());
+res->get_values().insert_range_from(get_values(), 0,
get_values().size());
+offset = get_offsets().back();
+}
+res->get_offsets().resize(to_size);
+for (size_t i = from_size; i < to_size; ++i) {
+res->get_offsets()[i] = offset;
+}
+}
return res;
}
// to support field functions
Field ColumnMap::operator[](size_t n) const {
-// Map is FieldVector , see in field.h
-Map res(2);
-keys->get(n, res[0]);
-values->get(n, res[1]);
+// Map is FieldVector, now we keep key value in seperate , see in field.h
+Map m(2);
+size_t start_offset = offset_at(n);
+size_t element_size = size_at(n);
+
+if (element_size > max_array_size_as_field) {
+LOG(FATAL) << "element size " << start_offset
+ << " is too large to be manipulated as single map field,"
+ << "maximum size " << max_array_size_as_field;
+}
-return res;
+Array k(element_size), v(element_size);
+
+for (size_t i = 0; i < element_size; ++i) {
+k[i] = get_keys()[start_offset + i];
+v[i] = get_values()[start_offset + i];
+}
+
+m.push_back(k);
+m.push_back(v);
+return m;
}
// here to compare to below
void ColumnMap::get(size_t n, Field& res) const {
Review Comment:
re
[GitHub] [doris] hello-stephen commented on pull request #17427: [fix](compatibility) fix unsigned int type compatibility value scope problem
hello-stephen commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455025269 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.26 seconds stream load tsv: 469 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 39 seconds loaded 2358488459 Bytes, about 57 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230305083340_clickbench_pr_108678.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #17421: [refactor](functioncontext) remove duplicate type definition in function context
yiguolei commented on PR #17421: URL: https://github.com/apache/doris/pull/17421#issuecomment-1455025937 ./run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #17397: [fix](metrics)Delete the extra underline for metrics
yiguolei merged PR #17397: URL: https://github.com/apache/doris/pull/17397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (afb5def385 -> 779d94f932)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from afb5def385 [enhancement](timeout) replace query timeout with exec timeout (#17360) add 779d94f932 [fix](metrics)Delete the extra underline for metrics (#17397) No new revisions were added by this update. Summary of changes: be/src/util/system_metrics.cpp | 18 +- be/src/util/system_metrics.h | 6 +++--- 2 files changed, 12 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on a diff in pull request #17344: [feature](function) support type template in SQL function
zhangstar333 commented on code in PR #17344: URL: https://github.com/apache/doris/pull/17344#discussion_r1125623331 ## gensrc/script/doris_builtins_functions.py: ## @@ -100,7 +104,7 @@ [['element_at', '%element_extract%'], 'STRING', ['ARRAY_STRING', 'BIGINT'], 'ALWAYS_NULLABLE'], # map element -[['element_at', '%element_extract%'], 'INT', ['MAP_STRING_INT', 'STRING'], 'ALWAYS_NULLABLE'], +[['element_at', '%element_extract%'], 'V', ['MAP', 'K'], 'ALWAYS_NULLABLE', ['K', 'V']], Review Comment: `[['array_sort'], 'ARRAY', ['ARRAY'], '', ['K']]` Thanks very mush! , use this way will be easy to register array functions. In addition, I want to know how to register if have a function eg: it's input params is all array type, and return type is also all array type it means, the input param is `ARRAY`, so the return type not just K, like input is array, but return will be array, array.and so on -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on a diff in pull request #17344: [feature](function) support type template in SQL function
zhangstar333 commented on code in PR #17344: URL: https://github.com/apache/doris/pull/17344#discussion_r1125623331 ## gensrc/script/doris_builtins_functions.py: ## @@ -100,7 +104,7 @@ [['element_at', '%element_extract%'], 'STRING', ['ARRAY_STRING', 'BIGINT'], 'ALWAYS_NULLABLE'], # map element -[['element_at', '%element_extract%'], 'INT', ['MAP_STRING_INT', 'STRING'], 'ALWAYS_NULLABLE'], +[['element_at', '%element_extract%'], 'V', ['MAP', 'K'], 'ALWAYS_NULLABLE', ['K', 'V']], Review Comment: `[['array_sort'], 'ARRAY', ['ARRAY'], '', ['K']]` Thanks very mush! , use this way will be easy to register array functions. In addition, I want to know how to register if have a function eg: it's input params is all array type, and return type is also all array type it means, the input param is `ARRAY`, so the return type not just K, like input is array, but return will be `array, array`.and so on -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on a diff in pull request #17414: [enhancement](transaction) Reduce hold writeLock time for DatabaseTransactionMgr to clear transaction
wangbo commented on code in PR #17414:
URL: https://github.com/apache/doris/pull/17414#discussion_r1125626948
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -742,6 +742,37 @@ public void replayBatchRemoveTransaction(List
txnIds) {
}
}
+public void
replayBatchRemoveTransaction(BatchRemoveTransactionsOperationV2 operation) {
+writeLock();
+try {
+int numOfClearedTransaction = 0;
+if (operation.getLatestTxnIdForShort() != -1) {
+while (!finalStatusTransactionStateDequeShort.isEmpty()) {
+TransactionState transactionState =
finalStatusTransactionStateDequeShort.pop();
Review Comment:
In old version, when execute ```replayBatchRemoveTransaction```, txnId
could be removed when txnId locates in head of deque.
So in V2, how to ensure the consistency between
latestTxnIdForShort/latestTxnIdForLong and
finalStatusTransactionStateDequeShort/finalStatusTransactionStateDequeLong.
Is there the case that latestTxnIdForShort not in
finalStatusTransactionStateDequeShort?
How about add a check here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on a diff in pull request #17414: [enhancement](transaction) Reduce hold writeLock time for DatabaseTransactionMgr to clear transaction
wangbo commented on code in PR #17414:
URL: https://github.com/apache/doris/pull/17414#discussion_r1125626948
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -742,6 +742,37 @@ public void replayBatchRemoveTransaction(List
txnIds) {
}
}
+public void
replayBatchRemoveTransaction(BatchRemoveTransactionsOperationV2 operation) {
+writeLock();
+try {
+int numOfClearedTransaction = 0;
+if (operation.getLatestTxnIdForShort() != -1) {
+while (!finalStatusTransactionStateDequeShort.isEmpty()) {
+TransactionState transactionState =
finalStatusTransactionStateDequeShort.pop();
Review Comment:
In old version, when execute ```replayBatchRemoveTransaction```, txnId
could be removed when txnId locates in head of deque.
So in V2, how to ensure the consistency between
latestTxnIdForShort/latestTxnIdForLong and
finalStatusTransactionStateDequeShort/finalStatusTransactionStateDequeLong.
Is there the case that latestTxnIdForShort not in current
finalStatusTransactionStateDequeShort?
How about add a check here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue opened a new pull request, #17429: [chore](fe) enhance mysql data type
ByteYue opened a new pull request, #17429: URL: https://github.com/apache/doris/pull/17429 # Proposed changes Issue Number: close #xxx ## Problem summary Previously if we try to query the data type of one array/map/struct in mysql client, we would get one return type `unknown`. This pr makes be return the corresponding data type. Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue commented on pull request #17429: [chore](fe) enhance mysql data type
ByteYue commented on PR #17429: URL: https://github.com/apache/doris/pull/17429#issuecomment-1455036554 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17414: [enhancement](transaction) Reduce hold writeLock time for DatabaseTransactionMgr to clear transaction
caiconghui commented on code in PR #17414:
URL: https://github.com/apache/doris/pull/17414#discussion_r1125628974
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -742,6 +742,37 @@ public void replayBatchRemoveTransaction(List
txnIds) {
}
}
+public void
replayBatchRemoveTransaction(BatchRemoveTransactionsOperationV2 operation) {
+writeLock();
+try {
+int numOfClearedTransaction = 0;
+if (operation.getLatestTxnIdForShort() != -1) {
+while (!finalStatusTransactionStateDequeShort.isEmpty()) {
+TransactionState transactionState =
finalStatusTransactionStateDequeShort.pop();
Review Comment:
only when is there a bug, never encounter the unconsistent case here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17429: [chore](fe) enhance mysql data type
github-actions[bot] commented on PR #17429: URL: https://github.com/apache/doris/pull/17429#issuecomment-1455036614 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17414: [enhancement](transaction) Reduce hold writeLock time for DatabaseTransactionMgr to clear transaction
github-actions[bot] commented on PR #17414: URL: https://github.com/apache/doris/pull/17414#issuecomment-1455037492 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455039368 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #17391: [fix](regression) Parameterize the S3 info for segcompaction regressi…
freemandealer commented on PR #17391: URL: https://github.com/apache/doris/pull/17391#issuecomment-1455040394 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on pull request #17414: [enhancement](transaction) Reduce hold writeLock time for DatabaseTransactionMgr to clear transaction
caiconghui commented on PR #17414: URL: https://github.com/apache/doris/pull/17414#issuecomment-1455041170 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17429: [chore](fe) enhance mysql data type
hello-stephen commented on PR #17429: URL: https://github.com/apache/doris/pull/17429#issuecomment-1455042458 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.91 seconds stream load tsv: 472 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 74 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230305095634_clickbench_pr_108695.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455044749 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455044967 run compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17428: [docs](typo) fix docs, already support rename column.
ZhangYu0123 commented on PR #17428: URL: https://github.com/apache/doris/pull/17428#issuecomment-1455044994 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue commented on pull request #16813: [enhance](cooldown) turn write cooldown meta async
ByteYue commented on PR #16813: URL: https://github.com/apache/doris/pull/16813#issuecomment-1455045494 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17324: [function](vec)use const_col to opt current functions.
github-actions[bot] commented on PR #17324: URL: https://github.com/apache/doris/pull/17324#issuecomment-1455046306 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17324: [function](vec)use const_col to opt current functions.
github-actions[bot] commented on PR #17324: URL: https://github.com/apache/doris/pull/17324#issuecomment-1455046320 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455046454 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455046558 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16813: [enhance](cooldown) turn write cooldown meta async
github-actions[bot] commented on PR #16813: URL: https://github.com/apache/doris/pull/16813#issuecomment-1455046742 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang opened a new pull request, #17430: [bugfix](jsonb) Fix create mv using jsonb key cause be crash
xiaokang opened a new pull request, #17430: URL: https://github.com/apache/doris/pull/17430 # Proposed changes Issue Number: close #xxx ## Problem summary Add supportsTableKey() and supportsComparison() to Type. Forbid type that not supportsTableKey() to be used as mv key. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #17430: [bugfix](jsonb) Fix create mv using jsonb key cause be crash
xiaokang commented on PR #17430: URL: https://github.com/apache/doris/pull/17430#issuecomment-1455048597 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #17431: [fix](orc) fix heap-use-after-free and potential memory leak of orc reader
morningman opened a new pull request, #17431: URL: https://github.com/apache/doris/pull/17431 # Proposed changes Issue Number: close #xxx ## Problem summary 1. fix heap-use-after-free The OrcReader has a internal `FileInputStream`, If the file is empty, the memory of `FileInputStream` will leak. Besides, there is a `Statistics` instance in `FileInputStream`. `FileInputStream` maybe delete if the orc reader is inited failed, but `Statistics` maybe used when orc reader is closed, causing heap-use-after-free error. 2. Potential memory leak When init file scanner in file scan node, the file scanner prepare failed, the memory of file scanner will leak. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #17431: [fix](orc) fix heap-use-after-free and potential memory leak of orc reader
github-actions[bot] commented on code in PR #17431:
URL: https://github.com/apache/doris/pull/17431#discussion_r1125638322
##
be/src/vec/exec/format/orc/vorc_reader.cpp:
##
@@ -146,33 +151,32 @@ void OrcReader::_init_profile() {
}
}
-Status OrcReader::init_reader(
-std::unordered_map*
colname_to_value_range) {
-SCOPED_RAW_TIMER(&_statistics.parse_meta_time);
-if (_file_reader == nullptr) {
+Status OrcReader::_create_file_reader() {
+if (_file_input_stream.get() == nullptr) {
Review Comment:
warning: redundant get() call on smart pointer
[readability-redundant-smartptr-get]
```suggestion
if (_file_input_stream == nullptr) {
```
##
be/src/vec/exec/format/orc/vorc_reader.h:
##
@@ -261,6 +238,9 @@ class OrcReader : public GenericReader {
std::string _get_field_name_lower_case(const orc::Type* orc_type, int pos);
+void _collect_profile_on_close();
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the
previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
**be/src/vec/exec/format/orc/vorc_reader.h:73:** previously declared here
```cpp
private:
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #17421: [refactor](functioncontext) remove duplicate type definition in function context
yiguolei commented on PR #17421: URL: https://github.com/apache/doris/pull/17421#issuecomment-1455051070 ./run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #17431: [fix](orc) fix heap-use-after-free and potential memory leak of orc reader
morningman commented on PR #17431: URL: https://github.com/apache/doris/pull/17431#issuecomment-1455053867 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer opened a new pull request, #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
freemandealer opened a new pull request, #17432: URL: https://github.com/apache/doris/pull/17432 segcompaction is async and in parallel with load job. If the load job is canncelling, memory structures will be destroyed and cause segcompaction crash. This commit will wait segcompaction finished before destruction. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
freemandealer commented on PR #17432: URL: https://github.com/apache/doris/pull/17432#issuecomment-1455058984 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17421: [refactor](functioncontext) remove duplicate type definition in function context
hello-stephen commented on PR #17421: URL: https://github.com/apache/doris/pull/17421#issuecomment-1455059144 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.82 seconds stream load tsv: 473 seconds loaded 74807831229 Bytes, about 150 MB/s stream load json: 38 seconds loaded 2358488459 Bytes, about 59 MB/s stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230305110734_clickbench_pr_108736.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17431: [fix](orc) fix heap-use-after-free and potential memory leak of orc reader
hello-stephen commented on PR #17431: URL: https://github.com/apache/doris/pull/17431#issuecomment-1455059506 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.35 seconds stream load tsv: 469 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 76 seconds loaded 1101869774 Bytes, about 13 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230305110906_clickbench_pr_108744.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
github-actions[bot] commented on PR #17432: URL: https://github.com/apache/doris/pull/17432#issuecomment-1455059663 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on code in PR #17281: URL: https://github.com/apache/doris/pull/17281#discussion_r1125645907 ## docker/runtime/k8s/backend.yml: ## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +apiVersion: v1 +kind: Service +metadata: + name: dbe Review Comment: is namespace required filed in metadata? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on code in PR #17281: URL: https://github.com/apache/doris/pull/17281#discussion_r1125646253 ## docker/runtime/k8s/backend.yml: ## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +apiVersion: v1 +kind: Service +metadata: + name: dbe Review Comment: ```suggestion name: doris-be-demo ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on code in PR #17281: URL: https://github.com/apache/doris/pull/17281#discussion_r1125646296 ## docker/runtime/k8s/backend.yml: ## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +apiVersion: v1 +kind: Service +metadata: + name: dbe + labels: +app: dbe Review Comment: ```suggestion app: doris-be-demo ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on code in PR #17281: URL: https://github.com/apache/doris/pull/17281#discussion_r1125648253 ## docker/runtime/k8s/backend.yml: ## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +apiVersion: v1 +kind: Service +metadata: + name: dbe + labels: +app: dbe +spec: + ports: +- port: 9060 + name: be +- port: 8040 + name: webserver +- port: 9050 + name: heartbeat-port #This name should be fixed. Doris will get the port information through this name +- port: 8060 + name: brpc + clusterIP: None + selector: +app: dbe Review Comment: ```suggestion - port: 9060 name: be_thrift_port - port: 8040 name: be_webserver_port - port: 9050 name: be_heartbeat_port #This name should be fixed. Doris will get the port information through this name - port: 8060 name: be_brpc_port clusterIP: None selector: app: doris-be-demo ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on a diff in pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on code in PR #17281: URL: https://github.com/apache/doris/pull/17281#discussion_r1125648784 ## docker/runtime/k8s/backend.yml: ## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +apiVersion: v1 +kind: Service +metadata: + name: dbe + labels: +app: dbe +spec: + ports: +- port: 9060 + name: be +- port: 8040 + name: webserver +- port: 9050 + name: heartbeat-port #This name should be fixed. Doris will get the port information through this name +- port: 8060 + name: brpc + clusterIP: None + selector: +app: dbe +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: dbe + labels: +app: dbe +spec: + selector: +matchLabels: + app: dbe + serviceName: dbe + replicas: 3 + template: +metadata: + name: dbe + labels: +app: dbe +spec: + containers: +- name: dbe + #Need to change to real mirror information + image: be:test + imagePullPolicy: IfNotPresent + env: +#Specify the startup type as k8s to bypass some restrictions of the official image initialization script +- name: BUILD_TYPE + value: "k8s" + ports: +- containerPort: 9060 + name: be +- containerPort: 8040 + name: webserver +- containerPort: 9050 + name: heartbeat-port +- containerPort: 8060 + name: brpc + volumeMounts: + #Mount the configuration file in the way of configmap +- name: conf + mountPath: /opt/apache-doris/be/conf + #Ifnot mounted, when enable_profile, error will be reported when querying the data from jdbc catalog + #Error message: error setting certificate verify locations: CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none +- name: sys + mountPath: /etc/pki + readOnly: true + volumes: +- name: conf + configMap: +name: be-conf +- name: sys + hostPath: +path: /etc/pki +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: be-conf +data: + be.conf: | +PPROF_TMPDIR="$DORIS_HOME/log/" +sys_log_level = INFO + +be_port = 9060 +webserver_port = 8040 +heartbeat_service_port = 9050 +brpc_port = 8060 + +priority_networks = 172.16.0.0/24 Review Comment: ```suggestion apiVersion: apps/v1 kind: StatefulSet metadata: name: doris-be-demo labels: app: doris-be-demo spec: selector: matchLabels: app: doris-be-demo serviceName: doris-be-demo replicas: 3 template: metadata: name: doris-be-demo labels: app: doris-be-demo spec: containers: - name: doris-be-demo #Need to change to real mirror information image: apache-doris:test imagePullPolicy: IfNotPresent env: #Specify the startup type as k8s to bypass some restrictions of the official image initialization script - name: BUILD_TYPE value: "k8s" ports: - containerPort: 9060 name: be_thrift_port - containerPort: 8040 name: be_webserver_port - containerPort: 9050 name: be_heartbeat_port - containerPort: 8060 name: be_brpc_port volumeMounts: #Mount the configuration file in the way of configmap - name: conf mountPath: /opt/apache-doris/be/conf #Ifnot mounted, when enable_profile, error will be reported when querying the data from jdbc catalog #Error message: error setting certificate verify locations: CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none - name: sys mountPath: /etc/pki readOnly: true volumes: - name: conf configMap: name: be-conf - name: sys
[GitHub] [doris] caiconghui commented on pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on PR #17281: URL: https://github.com/apache/doris/pull/17281#issuecomment-1455066561 yaml file name could be uniforma like: doris_be.yaml doris_fe.yaml doris_cn.yaml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui commented on pull request #17281: [enhancement](k8s)K8s yml demo
caiconghui commented on PR #17281: URL: https://github.com/apache/doris/pull/17281#issuecomment-1455066939 and three compoment for fe be cn show be related if we want to fast to relate them in k8s cluster, doris-fe-$cluster_name doris-be-$cluster_name doris-cn-$cluster_name -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455067472 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
Yulei-Yang commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455068237 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17427: [fix](compatibility) fix unsigned int type compatibility value scope problem
ZhangYu0123 commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455069363 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #17334: [fix](merge-on-write) fix that delete bitmap is not calculated correctly when clone tablet
liaoxin01 commented on code in PR #17334:
URL: https://github.com/apache/doris/pull/17334#discussion_r1125652862
##
be/src/olap/tablet.cpp:
##
@@ -195,17 +195,55 @@ void Tablet::save_meta() {
}
Status Tablet::revise_tablet_meta(const std::vector& to_add,
- const std::vector&
to_delete) {
+ const std::vector&
to_delete,
+ bool is_incremental_clone) {
LOG(INFO) << "begin to revise tablet. tablet_id=" << tablet_id();
delete_rowsets(to_delete, false);
add_rowsets(to_add);
-// FIXME: How to reclaim delete bitmap of deleted rowsets and stale
rowsets?
if (keys_type() == UNIQUE_KEYS && enable_unique_key_merge_on_write()) {
auto new_rowset_tree = std::make_unique();
ModifyRowSetTree(*_rowset_tree, to_delete, to_add,
new_rowset_tree.get());
_rowset_tree = std::move(new_rowset_tree);
+std::vector calc_delete_bitmap_rowsets;
+int64_t to_add_min_version = INT64_MAX;
+int64_t to_add_max_version = INT64_MIN;
for (auto& rs : to_add) {
-RETURN_IF_ERROR(update_delete_bitmap_without_lock(rs));
+if (to_add_min_version > rs->start_version()) {
+to_add_min_version = rs->start_version();
+}
+if (to_add_max_version < rs->end_version()) {
+to_add_max_version = rs->end_version();
+}
+}
+Version calc_delete_bitmap_ver;
+if (is_incremental_clone) {
+// From the rowset of to_add with smallest version, all other
rowsets
+// need to recalculate the delete bitmap
+// For example:
+// local tablet: [0-1] [2-5] [6-6] [9-10]
+// clone tablet: [7-7] [8-8]
+// new tablet: [0-1] [2-5] [6-6] [7-7] [8-8] [9-10]
+// [7-7] [8-8] [9-10] need to recalculate delete bitmap
+calc_delete_bitmap_ver = Version(to_add_min_version,
max_version().second);
+} else {
+// the delete bitmap of to_add's rowsets has clone from remote
when full clone.
+// only other rowsets in local need to recalculate the delete
bitmap.
+// For example:
+// local tablet: [0-1]x [2-5]x [6-6]x [7-7]x [9-10]
+// clone tablet: [0-1] [2-4] [5-6] [7-8]
+// new tablet: [0-1] [2-4] [5-6] [7-8] [9-10]
+// only [9-10] need to recalculate delete bitmap
+CHECK_EQ(to_add_min_version, 0) << "to_add_min_version is: " <<
to_add_min_version;
+calc_delete_bitmap_ver = Version(to_add_max_version + 1,
max_version().second);
+}
+Status res =
+capture_consistent_rowsets(calc_delete_bitmap_ver,
&calc_delete_bitmap_rowsets);
+// Because the data in memory has been changed, can't return an error.
+CHECK(res.ok()) << "fail to capture_consistent_rowsets, res: " << res;
+
+for (auto rs : calc_delete_bitmap_rowsets) {
+res = update_delete_bitmap_without_lock(rs);
+CHECK(res.ok()) << "fail to update_delete_bitmap_without_lock,
res: " << res;
Review Comment:
Because the data in memory has changed, if do not use CHECK here, it may
cause unexpected errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Bingandbing commented on pull request #17305: [enhancement](planner) support case transition of timestamp datatype when create table
Bingandbing commented on PR #17305: URL: https://github.com/apache/doris/pull/17305#issuecomment-1455074157 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] bobhan1 commented on issue #17176: [Good First Issue] Doris' Future
bobhan1 commented on issue #17176: URL: https://github.com/apache/doris/issues/17176#issuecomment-1455086037 Task 44: [array function] support `array_flatten` function, please assign to me. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #17391: [fix](regression) Parameterize the S3 info for segcompaction regressi…
freemandealer commented on PR #17391: URL: https://github.com/apache/doris/pull/17391#issuecomment-1455090712 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
github-actions[bot] commented on PR #17432: URL: https://github.com/apache/doris/pull/17432#issuecomment-1455091309 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
github-actions[bot] commented on PR #17432: URL: https://github.com/apache/doris/pull/17432#issuecomment-1455091333 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17432: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731)
dataroaring merged PR #17432: URL: https://github.com/apache/doris/pull/17432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](segcompaction) core when doing segcompaction for cancelling load(#16731) (#17432)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d08b231073 [fix](segcompaction) core when doing segcompaction for
cancelling load(#16731) (#17432)
d08b231073 is described below
commit d08b2310737402b75a00f9d82fb4eb95916fa2f3
Author: zhengyu
AuthorDate: Sun Mar 5 21:24:32 2023 +0800
[fix](segcompaction) core when doing segcompaction for cancelling
load(#16731) (#17432)
segcompaction is async and in parallel with load job. If the load job is
canncelling, memory structures will be destroyed and cause segcompaction
crash. This commit will wait segcompaction finished before destruction.
---
be/src/olap/rowset/beta_rowset_writer.cpp | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/beta_rowset_writer.cpp
b/be/src/olap/rowset/beta_rowset_writer.cpp
index 91a9774560..684d42a1f8 100644
--- a/be/src/olap/rowset/beta_rowset_writer.cpp
+++ b/be/src/olap/rowset/beta_rowset_writer.cpp
@@ -62,7 +62,11 @@ BetaRowsetWriter::BetaRowsetWriter()
}
BetaRowsetWriter::~BetaRowsetWriter() {
-// OLAP_UNUSED_ARG(_wait_flying_segcompaction());
+/* Note that segcompaction is async and in parallel with load job. So we
should handle carefully
+ * when the job is cancelled. Although it is meaningless to continue
segcompaction when the job
+ * is cancelled, the objects involved in the job should be preserved
during segcompaction to
+ * avoid crashs for memory issues. */
+OLAP_UNUSED_ARG(_wait_flying_segcompaction());
// TODO(lingbin): Should wrapper exception logic, no need to know file ops
directly.
if (!_already_built) { // abnormal exit, remove all files generated
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] bobhan1 opened a new issue, #17433: [Feature] (array_function) support array_flatten function
bobhan1 opened a new issue, #17433: URL: https://github.com/apache/doris/issues/17433 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description _No response_ ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang opened a new pull request, #17434: [Improvement](restore) make timeout of restore job's dispatching task progress configuable
Yulei-Yang opened a new pull request, #17434: URL: https://github.com/apache/doris/pull/17434 when a restore job has a plenty of replicas, it may fail due to timeout. The error message is： `[RestoreJob.checkAndPrepareMeta():782] begin to send create replica tasks to BE for restore. total 381344 tasks. timeout: 60` Currently, the max value of timeout is fixed, it's not suitable for such cases. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [x] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] bobhan1 commented on issue #17433: [Feature] (array_function) support array_flatten function
bobhan1 commented on issue #17433: URL: https://github.com/apache/doris/issues/17433#issuecomment-1455093290 Do we need to do type conversion or promotion if the type of the values in innermost arrays are different with each other? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17427: [fix](compatibility) fix unsigned int type compatibility value scope problem
ZhangYu0123 commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455093682 run p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16617: [Feature](network interface) Support network interface
github-actions[bot] commented on PR #16617: URL: https://github.com/apache/doris/pull/16617#issuecomment-1455094890 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #17425: [fix](regression) Adjust the test_add_drop_index case to avoid that FE failed to start when replaying the log
Tanya-W commented on PR #17425: URL: https://github.com/apache/doris/pull/17425#issuecomment-1455097408 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
github-actions[bot] commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455097550 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
github-actions[bot] commented on PR #17424: URL: https://github.com/apache/doris/pull/17424#issuecomment-1455097543 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #17424: [Improvement](auth)(step-2) add ranger authorizer for hms catalog
morningman merged PR #17424: URL: https://github.com/apache/doris/pull/17424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (d08b231073 -> d8a231f340)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from d08b231073 [fix](segcompaction) core when doing segcompaction for
cancelling load(#16731) (#17432)
add d8a231f340 [Improvement](auth)(step-2) add ranger authorizer for hms
catalog (#17424)
No new revisions were added by this update.
Summary of changes:
docs/en/docs/lakehouse/multi-catalog/hive.md | 19 +
docs/zh-CN/docs/lakehouse/multi-catalog/hive.md| 17 +
.../authorizer/RangerHiveAccessController.java | 86 +-
.../authorizer/RangerHiveAuditLogFlusher.java} | 35 -
.../org/apache/doris/mysql/privilege/Role.java | 4 +
5 files changed, 105 insertions(+), 56 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/{statistics/util/BlockingCounter.java
=> catalog/authorizer/RangerHiveAuditLogFlusher.java} (63%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17418: [chore](config) Increase the default maximum depth limit for expressions
github-actions[bot] commented on PR #17418: URL: https://github.com/apache/doris/pull/17418#issuecomment-1455099706 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17418: [chore](config) Increase the default maximum depth limit for expressions
github-actions[bot] commented on PR #17418: URL: https://github.com/apache/doris/pull/17418#issuecomment-1455099690 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17381: [fix](rebalance) fix that the clone operation is not performed due to incorrect condition judgment
dataroaring merged PR #17381: URL: https://github.com/apache/doris/pull/17381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](rebalance) fix that the clone operation is not performed due to incorrect condition judgment (#17381)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5190a496ac [fix](rebalance) fix that the clone operation is not
performed due to incorrect condition judgment (#17381)
5190a496ac is described below
commit 5190a496ac4cea4d2da070ea6d85d77d91de1fbd
Author: Xin Liao
AuthorDate: Sun Mar 5 21:58:33 2023 +0800
[fix](rebalance) fix that the clone operation is not performed due to
incorrect condition judgment (#17381)
---
be/src/olap/task/engine_clone_task.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/task/engine_clone_task.cpp
b/be/src/olap/task/engine_clone_task.cpp
index 76b5690559..70f1bd2d2e 100644
--- a/be/src/olap/task/engine_clone_task.cpp
+++ b/be/src/olap/task/engine_clone_task.cpp
@@ -68,7 +68,7 @@ EngineCloneTask::EngineCloneTask(const TCloneReq& clone_req,
const TMasterInfo&
Status EngineCloneTask::execute() {
// register the tablet to avoid it is deleted by gc thread during clone
process
SCOPED_ATTACH_TASK(_mem_tracker);
-if
(StorageEngine::instance()->tablet_manager()->register_clone_tablet(_clone_req.tablet_id))
{
+if
(!StorageEngine::instance()->tablet_manager()->register_clone_tablet(_clone_req.tablet_id))
{
return Status::InternalError("tablet {} is under clone",
_clone_req.tablet_id);
}
Status st = _do_clone();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455101182 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #17434: [Improvement](restore) make timeout of restore job's dispatching task progress configuable
morningman commented on PR #17434: URL: https://github.com/apache/doris/pull/17434#issuecomment-1455101304 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17434: [Improvement](restore) make timeout of restore job's dispatching task progress configuable
github-actions[bot] commented on PR #17434: URL: https://github.com/apache/doris/pull/17434#issuecomment-1455101380 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17434: [Improvement](restore) make timeout of restore job's dispatching task progress configuable
github-actions[bot] commented on PR #17434: URL: https://github.com/apache/doris/pull/17434#issuecomment-1455101387 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17334: [fix](merge-on-write) fix that delete bitmap is not calculated correctly when clone tablet
dataroaring merged PR #17334: URL: https://github.com/apache/doris/pull/17334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (5190a496ac -> 0801883604)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5190a496ac [fix](rebalance) fix that the clone operation is not performed due to incorrect condition judgment (#17381) add 0801883604 [fix](merge-on-write) fix that delete bitmap is not calculated correctly when clone tablet (#17334) No new revisions were added by this update. Summary of changes: be/src/olap/snapshot_manager.cpp | 7 be/src/olap/tablet.cpp | 67 -- be/src/olap/tablet.h | 3 +- be/src/olap/tablet_meta.cpp| 13 +++ be/src/olap/tablet_meta.h | 8 be/src/olap/task/engine_clone_task.cpp | 7 +++- 6 files changed, 75 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch hadoop-libs created (now 113149a)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch hadoop-libs in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git at 113149a [chore](github) Use LLVM Clang to build the libraries by default on macOS (#41) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch hadoop-libs updated: [hadoop-libs] modify readme
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch hadoop-libs in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/hadoop-libs by this push: new 4f3d477 [hadoop-libs] modify readme 4f3d477 is described below commit 4f3d47776339ac1e18cb04e4a0137f3c4dee8792 Author: morningman AuthorDate: Sun Mar 5 22:13:05 2023 +0800 [hadoop-libs] modify readme --- LICENSE.txt | 749 README.md | 15 +- 2 files changed, 6 insertions(+), 758 deletions(-) diff --git a/LICENSE.txt b/LICENSE.txt deleted file mode 100644 index 5c9c62d..000 --- a/LICENSE.txt +++ /dev/null @@ -1,749 +0,0 @@ - Apache License - Version 2.0, January 2004 -http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except a
[doris-thirdparty] annotated tag hadoop-libs-3.3.4 updated (4f3d477 -> d4811ee)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag hadoop-libs-3.3.4 in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git *** WARNING: tag hadoop-libs-3.3.4 was modified! *** from 4f3d477 (commit) to d4811ee (tag) tagging 4f3d47776339ac1e18cb04e4a0137f3c4dee8792 (commit) by morningman on Sun Mar 5 22:15:16 2023 +0800 - Log - hadoop-libs-3.3.4 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on issue #17176: [Good First Issue] Doris' Future
Yukang-Lian commented on issue #17176: URL: https://github.com/apache/doris/issues/17176#issuecomment-1455106109 > Task 44: [array function] support `array_flatten` function, please assign to me. Thank you. #17433 Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang commented on pull request #17342: [Improvement](meta) support return total statistics of all databases for command show proc '/jobs
Yulei-Yang commented on PR #17342: URL: https://github.com/apache/doris/pull/17342#issuecomment-1455109957 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17427: [fix](compatibility) fix unsigned int type compatibility value scope problem
ZhangYu0123 commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455112600 run p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #17435: [deps](libhdfs) add official hadoop libhdfs for x86
morningman opened a new pull request, #17435: URL: https://github.com/apache/doris/pull/17435 # Proposed changes Issue Number: close #xxx ## Problem summary This is the first step to introduce official hadoop libhdfs to Doris. Because the current hdfs client `libhdfs3` lacks some important feature and is hard to maintain. Download the hadoop 3.3.4 binary from hadoop website: https://hadoop.apache.org/releases.html Extract libs and headers which are used for libhdfs, and pack them into `hadoop_lib_3.3.4-x86.tar.gz` Upload it to https://github.com/apache/doris-thirdparty/releases/tag/hadoop-libs-3.3.4 ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] subkanthi commented on issue #16392: Doris Roadmap 2023
subkanthi commented on issue #16392: URL: https://github.com/apache/doris/issues/16392#issuecomment-1455120428 Im interested in contributing to some of the work, how do I go about doing it, should I create a separate issue and just add it here in the comments, please let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] white123dog commented on issue #17176: [Good First Issue] Doris' Future
white123dog commented on issue #17176: URL: https://github.com/apache/doris/issues/17176#issuecomment-1455121851 Hi,I'd like to do task 42:[array function] Support array_reduce function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on issue #17433: [Feature] (array_function) support array_flatten function
eldenmoon commented on issue #17433: URL: https://github.com/apache/doris/issues/17433#issuecomment-1455122303 @bobhan1 you could reference to function `array()`, it has a promotion rule, you could try it to test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangYu0123 commented on pull request #17427: [fix](type compatibility) fix unsigned int type compatibility problem
ZhangYu0123 commented on PR #17427: URL: https://github.com/apache/doris/pull/17427#issuecomment-1455131795 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gitccl opened a new pull request, #17436: [Feature](array-function) Support array_concat function
gitccl opened a new pull request, #17436: URL: https://github.com/apache/doris/pull/17436 # Proposed changes Issue Number: close #17307 ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [X] Has document been added or modified * [ ] Does it need to update dependencies * [X] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gitccl commented on pull request #17436: [Feature](array-function) Support array_concat function
gitccl commented on PR #17436: URL: https://github.com/apache/doris/pull/17436#issuecomment-1455134897 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17436: [Feature](array-function) Support array_concat function
github-actions[bot] commented on PR #17436: URL: https://github.com/apache/doris/pull/17436#issuecomment-1455135119 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #17202: [Chore](third-party) upgrade thrift from 0.13 to 0.16
BiteThet commented on PR #17202: URL: https://github.com/apache/doris/pull/17202#issuecomment-1455135211 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #17262: [Enchancement](Materialized-View) add more error infomation for select materialized view fail
BiteThet commented on PR #17262: URL: https://github.com/apache/doris/pull/17262#issuecomment-1455135466 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
