Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
gortiz commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1507160348
##
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java:
##
@@ -467,7 +501,11 @@ public ColumnDataType getFinalResultColumnType() {
@Override
public Double extractFinalResult(Map
intermediateResult) {
if (intermediateResult.isEmpty()) {
- return DEFAULT_FINAL_RESULT;
+ if (_nullHandlingEnabled) {
Review Comment:
_this map_ = `intermediateResult`?
In that case, no. `intermediateResult` is created in
`extractAggregationResult(AggregationResultHolder)`, which calls
`extractIntermediateResult` which returns not null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
gortiz commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1507164446
##
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java:
##
@@ -467,7 +501,11 @@ public ColumnDataType getFinalResultColumnType() {
@Override
public Double extractFinalResult(Map
intermediateResult) {
if (intermediateResult.isEmpty()) {
- return DEFAULT_FINAL_RESULT;
+ if (_nullHandlingEnabled) {
Review Comment:
I'm adding a new test to show the result when all stored data is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Multi-stage] Reduce the stats transferred [pinot]
Jackie-Jiang merged PR #12517: URL: https://github.com/apache/pinot/pull/12517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: [Multi-stage] Reduce the stats transfered (#12517)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 35c89c87a5 [Multi-stage] Reduce the stats transfered (#12517)
35c89c87a5 is described below
commit 35c89c87a5473465ca273b576d5bf1e554cb0c35
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Thu Feb 29 00:07:41 2024 -0800
[Multi-stage] Reduce the stats transfered (#12517)
---
.../MultiStageBrokerRequestHandler.java| 20 +++
.../pinot/common/datablock/DataBlockUtils.java | 2 --
.../runtime/blocks/TransferableBlockUtils.java | 3 +-
.../runtime/operator/exchange/BlockExchange.java | 39 --
.../query/service/dispatch/QueryDispatcher.java| 25 ++
5 files changed, 55 insertions(+), 34 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
index 05fb1ddd52..35aff7efd2 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
@@ -19,8 +19,9 @@
package org.apache.pinot.broker.requesthandler;
import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.collect.Maps;
import java.util.ArrayList;
-import java.util.HashMap;
+import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -61,6 +62,7 @@ import org.apache.pinot.core.transport.ServerInstance;
import org.apache.pinot.query.QueryEnvironment;
import org.apache.pinot.query.catalog.PinotCatalog;
import org.apache.pinot.query.mailbox.MailboxService;
+import org.apache.pinot.query.planner.physical.DispatchablePlanFragment;
import org.apache.pinot.query.planner.physical.DispatchableSubPlan;
import org.apache.pinot.query.routing.WorkerManager;
import org.apache.pinot.query.service.dispatch.QueryDispatcher;
@@ -179,14 +181,20 @@ public class MultiStageBrokerRequestHandler extends
BaseBrokerRequestHandler {
Map queryOptions = sqlNodeAndOptions.getOptions();
boolean traceEnabled =
Boolean.parseBoolean(queryOptions.get(CommonConstants.Broker.Request.TRACE));
-
-ResultTable queryResults;
-Map stageIdStatsMap = new HashMap<>();
-for (int stageId = 0; stageId <
dispatchableSubPlan.getQueryStageList().size(); stageId++) {
- stageIdStatsMap.put(stageId, new ExecutionStatsAggregator(traceEnabled));
+Map stageIdStatsMap;
+if (!traceEnabled) {
+ stageIdStatsMap = Collections.singletonMap(0, new
ExecutionStatsAggregator(false));
+} else {
+ List stagePlans =
dispatchableSubPlan.getQueryStageList();
+ int numStages = stagePlans.size();
+ stageIdStatsMap = Maps.newHashMapWithExpectedSize(numStages);
+ for (int stageId = 0; stageId < numStages; stageId++) {
+stageIdStatsMap.put(stageId, new ExecutionStatsAggregator(true));
+ }
}
long executionStartTimeNs = System.nanoTime();
+ResultTable queryResults;
try {
queryResults = _queryDispatcher.submitAndReduce(requestContext,
dispatchableSubPlan, queryTimeoutMs, queryOptions,
stageIdStatsMap);
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java
b/pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java
index 5d38168712..27f1140328 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java
@@ -59,12 +59,10 @@ public final class DataBlockUtils {
}
public static MetadataBlock getEndOfStreamDataBlock() {
-// TODO: add query statistics metadata for the block.
return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS);
}
public static MetadataBlock getEndOfStreamDataBlock(Map
stats) {
-// TODO: add query statistics metadata for the block.
return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS, stats);
}
diff --git
a/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlockUtils.java
b/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlockUtils.java
index 01c5fd7ddd..355b6fe294 100644
---
a/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlockUtils.java
+++
b/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlockUtils.java
@@ -30,13 +30,14 @@ import org.apache.pinot.common.datablock.DataBlockUtils;
public final class TransferableBlockUtils {
private static final int MEDIAN_
Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
gortiz commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1507172781
##
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java:
##
@@ -0,0 +1,263 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockValSet;
+import org.roaringbitmap.IntIterator;
+import org.roaringbitmap.RoaringBitmap;
+
+
+public abstract class NullableSingleInputAggregationFunction
+extends BaseSingleInputAggregationFunction {
+ protected final boolean _nullHandlingEnabled;
+
+ public NullableSingleInputAggregationFunction(ExpressionContext expression,
boolean nullHandlingEnabled) {
+super(expression);
+_nullHandlingEnabled = nullHandlingEnabled;
+ }
+
+ /**
+ * A consumer that is being used to consume batch of indexes.
+ */
+ @FunctionalInterface
+ public interface BatchConsumer {
+/**
+ * Consumes a batch of indexes.
+ * @param fromInclusive the start index (inclusive)
+ * @param toExclusive the end index (exclusive)
+ */
+void consume(int fromInclusive, int toExclusive);
+ }
+
+ /**
+ * A reducer that is being used to fold over consecutive indexes.
+ * @param
+ */
+ @FunctionalInterface
+ public interface Reducer {
+/**
+ * Applies the reducer to the range of indexes.
+ * @param acum the initial value of the accumulator
+ * @param fromInclusive the start index (inclusive)
+ * @param toExclusive the end index (exclusive)
+ * @return the next value of the accumulator (maybe the same as the input)
+ */
+A apply(A acum, int fromInclusive, int toExclusive);
+ }
+
+ /**
+ * Iterates over the non-null ranges of the blockValSet and calls the
consumer for each range.
+ * @param blockValSet the blockValSet to iterate over
+ * @param consumer the consumer to call for each non-null range
+ */
+ public void forEachNotNull(int length, BlockValSet blockValSet,
BatchConsumer consumer) {
+if (!_nullHandlingEnabled) {
+ consumer.consume(0, length);
+ return;
+}
+
+RoaringBitmap roaringBitmap = blockValSet.getNullBitmap();
+if (roaringBitmap == null) {
+ consumer.consume(0, length);
+ return;
+}
+
+forEachNotNull(length, roaringBitmap.getIntIterator(), consumer);
+ }
+
+ /**
+ * Iterates over the non-null ranges of the nullIndexIterator and calls the
consumer for each range.
+ * @param nullIndexIterator an int iterator that returns values in ascending
order whose min value is 0.
+ * Rows are considered null if and only if their
index is emitted.
+ */
+ public void forEachNotNull(int length, IntIterator nullIndexIterator,
BatchConsumer consumer) {
+int prev = 0;
+while (nullIndexIterator.hasNext() && prev < length) {
+ int nextNull = Math.min(nullIndexIterator.next(), length);
+ if (nextNull > prev) {
+consumer.consume(prev, nextNull);
+ }
+ prev = nextNull + 1;
+}
+if (prev < length) {
+ consumer.consume(prev, length);
+}
+ }
+
+ /**
+ * Folds over the non-null ranges of the blockValSet using the reducer.
+ * @param initialAcum the initial value of the accumulator
+ * @param The type of the accumulator
+ */
+ public A foldNotNull(int length, @Nullable RoaringBitmap roaringBitmap,
A initialAcum, Reducer reducer) {
+IntIterator intIterator = roaringBitmap == null ? null :
roaringBitmap.getIntIterator();
+return foldNotNull(length, intIterator, initialAcum, reducer);
+ }
+
+ /**
+ * Folds over the non-null ranges of the nullIndexIterator using the reducer.
+ * @param nullIndexIterator an int iterator that returns values in ascending
order whose min value is 0.
+ * Rows are considered null if and only if their
index is emitted.
+ * @param initialAcum the initial value of the accumulator
+ * @param The type of the accumulator
+ */
+
Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
gortiz commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1507175271
##
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java:
##
@@ -0,0 +1,263 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockValSet;
+import org.roaringbitmap.IntIterator;
+import org.roaringbitmap.RoaringBitmap;
+
+
+public abstract class NullableSingleInputAggregationFunction
+extends BaseSingleInputAggregationFunction {
+ protected final boolean _nullHandlingEnabled;
+
+ public NullableSingleInputAggregationFunction(ExpressionContext expression,
boolean nullHandlingEnabled) {
+super(expression);
+_nullHandlingEnabled = nullHandlingEnabled;
+ }
+
+ /**
+ * A consumer that is being used to consume batch of indexes.
+ */
+ @FunctionalInterface
+ public interface BatchConsumer {
+/**
+ * Consumes a batch of indexes.
+ * @param fromInclusive the start index (inclusive)
+ * @param toExclusive the end index (exclusive)
+ */
+void consume(int fromInclusive, int toExclusive);
+ }
+
+ /**
+ * A reducer that is being used to fold over consecutive indexes.
+ * @param
+ */
+ @FunctionalInterface
+ public interface Reducer {
+/**
+ * Applies the reducer to the range of indexes.
+ * @param acum the initial value of the accumulator
+ * @param fromInclusive the start index (inclusive)
+ * @param toExclusive the end index (exclusive)
+ * @return the next value of the accumulator (maybe the same as the input)
+ */
+A apply(A acum, int fromInclusive, int toExclusive);
+ }
+
+ /**
+ * Iterates over the non-null ranges of the blockValSet and calls the
consumer for each range.
+ * @param blockValSet the blockValSet to iterate over
+ * @param consumer the consumer to call for each non-null range
+ */
+ public void forEachNotNull(int length, BlockValSet blockValSet,
BatchConsumer consumer) {
+if (!_nullHandlingEnabled) {
+ consumer.consume(0, length);
+ return;
+}
+
+RoaringBitmap roaringBitmap = blockValSet.getNullBitmap();
+if (roaringBitmap == null) {
+ consumer.consume(0, length);
+ return;
+}
+
+forEachNotNull(length, roaringBitmap.getIntIterator(), consumer);
+ }
+
+ /**
+ * Iterates over the non-null ranges of the nullIndexIterator and calls the
consumer for each range.
+ * @param nullIndexIterator an int iterator that returns values in ascending
order whose min value is 0.
+ * Rows are considered null if and only if their
index is emitted.
+ */
+ public void forEachNotNull(int length, IntIterator nullIndexIterator,
BatchConsumer consumer) {
+int prev = 0;
+while (nullIndexIterator.hasNext() && prev < length) {
+ int nextNull = Math.min(nullIndexIterator.next(), length);
+ if (nextNull > prev) {
+consumer.consume(prev, nextNull);
+ }
+ prev = nextNull + 1;
+}
+if (prev < length) {
+ consumer.consume(prev, length);
+}
+ }
+
+ /**
+ * Folds over the non-null ranges of the blockValSet using the reducer.
+ * @param initialAcum the initial value of the accumulator
+ * @param The type of the accumulator
+ */
+ public A foldNotNull(int length, @Nullable RoaringBitmap roaringBitmap,
A initialAcum, Reducer reducer) {
+IntIterator intIterator = roaringBitmap == null ? null :
roaringBitmap.getIntIterator();
+return foldNotNull(length, intIterator, initialAcum, reducer);
+ }
+
+ /**
+ * Folds over the non-null ranges of the nullIndexIterator using the reducer.
+ * @param nullIndexIterator an int iterator that returns values in ascending
order whose min value is 0.
+ * Rows are considered null if and only if their
index is emitted.
+ * @param initialAcum the initial value of the accumulator
+ * @param The type of the accumulator
+ */
+
Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]
gortiz commented on code in PR #12475:
URL: https://github.com/apache/pinot/pull/12475#discussion_r1507181049
##
pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.validate;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlCastFunction;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.util.Static;
+
+
+public class BytesCastVisitor extends SqlBasicVisitor {
+
+ private final SqlValidator _originalValidator;
+
+ public BytesCastVisitor(SqlValidator originalValidator) {
+_originalValidator = originalValidator;
+ }
+
+ @SuppressWarnings("checkstyle:WhitespaceAround")
+ @Override
+ public Void visit(SqlCall call) {
+if (call.getOperator() instanceof SqlCastFunction) {
+ List operands = call.getOperandList();
+ SqlNode sqlNode = operands.get(1);
+ Preconditions.checkState(sqlNode instanceof SqlDataTypeSpec);
+ RelDataType toType = ((SqlDataTypeSpec)
sqlNode).deriveType(_originalValidator);
+ if (!SqlTypeUtil.isBinary(toType)) {
+return super.visit(call);
+ }
+ SqlNode srcNode = operands.get(0);
+ RelDataType fromType =
_originalValidator.getValidatedNodeTypeIfKnown(srcNode);
+ if (fromType != null && SqlTypeUtil.isBinary(fromType)) {
Review Comment:
In Calcite there is BINARY and VARBINARY. In both cases
`SqlTypeUtil.isBinary` return true. I'm assuming we want to allow castings
between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
gortiz commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1507178148
##
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/FieldConfig.java:
##
@@ -254,6 +256,24 @@ public Builder withTierOverwrites(JsonNode tierOverwrites)
{
return this;
}
+public Builder withIndex(String indexId, IndexConfig indexConfig) {
Review Comment:
I think it is the same case than `orNull` int iterator. I'm going to remove
it for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]
gortiz commented on code in PR #12475:
URL: https://github.com/apache/pinot/pull/12475#discussion_r1507187948
##
pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.validate;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlCastFunction;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.util.Static;
+
+
+public class BytesCastVisitor extends SqlBasicVisitor {
+
+ private final SqlValidator _originalValidator;
+
+ public BytesCastVisitor(SqlValidator originalValidator) {
+_originalValidator = originalValidator;
+ }
+
+ @SuppressWarnings("checkstyle:WhitespaceAround")
+ @Override
+ public Void visit(SqlCall call) {
+if (call.getOperator() instanceof SqlCastFunction) {
+ List operands = call.getOperandList();
+ SqlNode sqlNode = operands.get(1);
+ Preconditions.checkState(sqlNode instanceof SqlDataTypeSpec);
+ RelDataType toType = ((SqlDataTypeSpec)
sqlNode).deriveType(_originalValidator);
+ if (!SqlTypeUtil.isBinary(toType)) {
+return super.visit(call);
Review Comment:
It is not the same, right?
`return null` just aborts the execution here while `return
super.visit(call)` does in fact return the same value, but has the desired side
effect of visiting the operators.
For example, an expression like: `CAST (someOperation(CAST ('someliteral' as
BINARY)) as STRING)` will be converted into:
1. CAST
1. someOperation
1. CAST
1. 'someliteral'
2. BINARY
2. STRING
In case we changed this line to `return null` we won't be able to detect the
inner cast to BINARY because we would just abort the visitor once the first
CAST to STRING was detected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]
gortiz commented on code in PR #12475:
URL: https://github.com/apache/pinot/pull/12475#discussion_r1507194978
##
pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.validate;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlCastFunction;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.util.Static;
+
+
+public class BytesCastVisitor extends SqlBasicVisitor {
+
+ private final SqlValidator _originalValidator;
+
+ public BytesCastVisitor(SqlValidator originalValidator) {
+_originalValidator = originalValidator;
+ }
+
+ @SuppressWarnings("checkstyle:WhitespaceAround")
+ @Override
+ public Void visit(SqlCall call) {
+if (call.getOperator() instanceof SqlCastFunction) {
+ List operands = call.getOperandList();
+ SqlNode sqlNode = operands.get(1);
+ Preconditions.checkState(sqlNode instanceof SqlDataTypeSpec);
+ RelDataType toType = ((SqlDataTypeSpec)
sqlNode).deriveType(_originalValidator);
+ if (!SqlTypeUtil.isBinary(toType)) {
+return super.visit(call);
+ }
+ SqlNode srcNode = operands.get(0);
+ RelDataType fromType =
_originalValidator.getValidatedNodeTypeIfKnown(srcNode);
+ if (fromType != null && SqlTypeUtil.isBinary(fromType)) {
+return super.visit(call);
+ }
+ String message = "Cannot cast " + srcNode + " as " + toType + ".";
+ if (srcNode instanceof SqlCharStringLiteral) {
+message += " Try to use binary literal instead (like x" + srcNode +
")";
+ } else if (fromType != null && SqlTypeUtil.isCharacter(fromType)) {
+message += " Try to wrap the expression in hexToBytes (like
hexToBytes(" + srcNode + "))";
+ }
+ SqlParserPos pos = call.getParserPosition();
+ RuntimeException ex = new RuntimeException(message);
Review Comment:
Changed to throw a new runtime exception called `InvalidCastException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Record enricher [pinot]
gortiz commented on PR #12243: URL: https://github.com/apache/pinot/pull/12243#issuecomment-1970696484 Do you suggest to rollback the change and modify `TransformPipeline` to support the same semantics defined in `ExpressionTransformer`? We are still in time to do that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Attempt to apply timestamp expr override in V2 [pinot]
gortiz closed pull request #12421: Attempt to apply timestamp expr override in V2 URL: https://github.com/apache/pinot/pull/12421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [draft] Null mode at query time [pinot]
gortiz closed pull request #11935: [draft] Null mode at query time URL: https://github.com/apache/pinot/pull/11935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
suddendust opened a new pull request, #12522:
URL: https://github.com/apache/pinot/pull/12522
This PR adds two metrics in `RealtimeSegmentDataManager`:
1. `REALTIME_LAST_FETCHED_BATCH_SIZE`: This is a new gauge that can assume
the following values:
- 0: When there were no messages in the last batch. This can happen when
there are no messages or very low vol of messages in the stream partition.
- 0: When messages were fetched in the last batch.
- -1: When there were exceptions fetching the message batch from the stream
partition.
This metric can be used to identify the 1st case of no messages as:
`sum by (table) (pinot_server_realtimeLastFetchedBatchSizeCount{}) == 0`. We
can use this source alert to inhibit other alerts. For example, an alert on
realtime ingestion stopped.
**Why can't we reuse `pinot_server_realtimeRowsConsumed`?**
`pinot_server_realtimeRowsConsumed` tracks the number of rows that were
successfully indexed. If there were problems transforming/indexing the row,
those rows aren't counted in this. So it makes it hard to calculate the total
number of rows being fetched from the partition stream.
3. `STREAM_CONSUMER_CREATE_EXCEPTIONS`: If we face exceptions trying to
create a stream consumer.
Testing:
`STREAM_CONSUMER_CREATE_EXCEPTIONS`:
https://github.com/apache/pinot/assets/84911643/f9427cfc-85ac-43c4-a40c-b33bd7a88198";>
`REALTIME_ROWS_FETCHED`:
https://github.com/apache/pinot/assets/84911643/5fb3020e-f5df-4272-87cc-fb85d5842e4a";>
https://github.com/apache/pinot/assets/84911643/fa636fcf-e665-479f-8344-09237f20054c";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
codecov-commenter commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1970764635 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `86 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`499222b`)](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 30 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [84 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...a/org/apache/pinot/common/metrics/ServerGauge.java](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9TZXJ2ZXJHYXVnZS5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...a/org/apache/pinot/common/metrics/ServerMeter.java](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9TZXJ2ZXJNZXRlci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12522?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12522 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2374 -62 Lines133233 129759 -3474 Branches 2063620131 -505 = - Hits 822740-82274 - Misses44911 129759+84848 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12522/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.
Re: [I] Timeout expired while fetching topic metadata [pinot]
sandeepjain98 commented on issue #8396: URL: https://github.com/apache/pinot/issues/8396#issuecomment-1970777144 Hi @xiangfu0 , is it solved . even I am facing same issue. " Caused by: org.apache.pinot.shaded.org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Modify consumingSegmentsInfo endpoint to indicate how many servers failed [pinot]
gortiz opened a new pull request, #12523: URL: https://github.com/apache/pinot/pull/12523 Controller endpoint /table/consumingSegmentsInfo may return partial information when servers timeout or there is a parsing error dealing with their response. These cases registered as a log, but there is no notice in the response to understand if it is complete or shows partial information, so it is easy to think that the current issue is that there are segments we are not actually consuming from. This PR tries to improve the usability of the endpoint by adding the new field `serversFailingToRespond` on /table/consumingSegmentsInfo that indicates the number of servers that have failed to answer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
gortiz commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1970846422 I like the idea, but I'm not sure if the current implementation is correct. Specifically, the scenario that concerns me is the one where: - we fetch batches far often than the Prometheus polling rate - the source mostly emits non empty batches In that scenario it seems possible that `REALTIME_LAST_FETCHED_BATCH_SIZE` would be 0 each time Prometheus polls it even if we are actually ingesting. Instead of being a gauge, we could have something like: `REALTIME_FETCHED_ROWS` which could be a counter. This metric would be non decrement and we could just apply `rate` (or some other operation) to that metric. If it doesn't increase in a while, we can be sure that there are no data in the source and therefore we can skip some alerts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Lvnszn commented on issue #12503: URL: https://github.com/apache/pinot/issues/12503#issuecomment-1970895940 > Do you have access to the server node? K8s log might not be enough because it won't log the actual stack trace, and it doesn't contain the non-error logs. The log has been recycled. . What I recorded here is OOM: map fail -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
suddendust commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1970950356 That's a valid case. >If it doesn't increase in a while, we can be sure that there are no data in the source and therefore we can skip some alerts. We can't differentiate b/w this (valid) case and when there are exceptions fetching a batch. Although such exceptions can be tracked using `REALTIME_CONSUMPTION_EXCEPTIONS`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] adding support of mega_bytes when configuring broker response size [pinot]
gortiz commented on PR #12510: URL: https://github.com/apache/pinot/pull/12510#issuecomment-1970892525 +1 to Jackie suggestion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Consistency in API response for live broker [pinot]
cypherean commented on PR #12201: URL: https://github.com/apache/pinot/pull/12201#issuecomment-1971105562 @Jackie-Jiang @walterddr please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
ege-st commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1971166374 This seems like a situation where a histogram would be right choice: it would let us do percentile breakdowns of rows read and I believe it also includes the counter that @gortiz suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Pass explicit TypeRef when evaluating MV jsonPath [pinot]
saurabhd336 opened a new pull request, #12524:
URL: https://github.com/apache/pinot/pull/12524
For a json column `myJsonCol` with values like
```
{"values": [1, 2, 3, 4, 5]}
```
```
select summv(json_extract_scalar(myJsonCol, '$.values[0:3]', 'INT_ARRAY'))
from wxcc_agent_activity limit 10
```
Fails with
```
Error Code: 200
QueryExecutionError:
java.lang.ClassCastException: class java.lang.Integer cannot be cast to
class java.lang.Double (java.lang.Integer and java.lang.Double are in module
java.base of loader 'bootstrap')
at
org.apache.pinot.core.common.evaluators.DefaultJsonPathEvaluator.processList(DefaultJsonPathEvaluator.java:633)
at
org.apache.pinot.core.common.evaluators.DefaultJsonPathEvaluator.evaluateBlock(DefaultJsonPathEvaluator.java:381)
at
org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readDoubleValuesMV(DataFetcher.java:724)
at
org.apache.pinot.core.common.DataFetcher.fetchDoubleValues(DataFetcher.java:385)
```
This is because the implicit cast to `List<>` class when performing the
`read(_jsonPath)` operation is not enough to correctly convert the data types.
This PR fixes this behaviour by passing explicit TypeRef objects to `read` when
reading json arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
gortiz commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1971375101 Small tip, instead of rate you can also use [increase](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase). I don't think the difference in performance would be important enough to change it, but it may be cleaner what you want to express. Anyway I think both functions are fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
gortiz commented on PR #12522: URL: https://github.com/apache/pinot/pull/12522#issuecomment-1971379244 LGTM. Unless there is some urgency, I'll wait til tomorrow to merge it just in case other people want to review it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Pass explicit TypeRef when evaluating MV jsonPath [pinot]
codecov-commenter commented on PR #12524: URL: https://github.com/apache/pinot/pull/12524#issuecomment-1971383094 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12524?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `37 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`34446a5`)](https://app.codecov.io/gh/apache/pinot/pull/12524?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 30 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://app.codecov.io/gh/apache/pinot/pull/12524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | 0.00% | [37 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12524 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2374 -62 Lines133233 129779 -3454 Branches 2063620130 -506 = - Hits 822740-82274 - Misses44911 129779+84868 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12524/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://ap
Re: [PR] Pass explicit TypeRef when evaluating MV jsonPath [pinot]
saurabhd336 merged PR #12524: URL: https://github.com/apache/pinot/pull/12524 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Pass explicit TypeRef when evaluating MV jsonPath (#12524)
This is an automated email from the ASF dual-hosted git repository.
saurabhd336 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bddd3612bc Pass explicit TypeRef when evaluating MV jsonPath (#12524)
bddd3612bc is described below
commit bddd3612bc0f8a5927f34f14d93fb969ac81e79b
Author: Saurabh Dubey
AuthorDate: Thu Feb 29 21:05:21 2024 +0530
Pass explicit TypeRef when evaluating MV jsonPath (#12524)
Co-authored-by: Saurabh Dubey
---
.../evaluators/DefaultJsonPathEvaluator.java | 89 ++---
.../evaluators/DefaultJsonPathEvaluatorTest.java | 146 +
2 files changed, 215 insertions(+), 20 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/common/evaluators/DefaultJsonPathEvaluator.java
b/pinot-core/src/main/java/org/apache/pinot/core/common/evaluators/DefaultJsonPathEvaluator.java
index f22474354c..cae1e3c17b 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/common/evaluators/DefaultJsonPathEvaluator.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/common/evaluators/DefaultJsonPathEvaluator.java
@@ -25,6 +25,7 @@ import com.jayway.jsonpath.InvalidPathException;
import com.jayway.jsonpath.JsonPath;
import com.jayway.jsonpath.Option;
import com.jayway.jsonpath.ParseContext;
+import com.jayway.jsonpath.TypeRef;
import com.jayway.jsonpath.spi.json.JacksonJsonProvider;
import com.jayway.jsonpath.spi.mapper.JacksonMappingProvider;
import java.math.BigDecimal;
@@ -58,6 +59,16 @@ public final class DefaultJsonPathEvaluator implements
JsonPathEvaluator {
private static final float[] EMPTY_FLOATS = new float[0];
private static final double[] EMPTY_DOUBLES = new double[0];
private static final String[] EMPTY_STRINGS = new String[0];
+ private static final TypeRef> INTEGER_LIST_TYPE = new
TypeRef>() {
+ };
+ private static final TypeRef> LONG_LIST_TYPE = new
TypeRef>() {
+ };
+ private static final TypeRef> FLOAT_LIST_TYPE = new
TypeRef>() {
+ };
+ private static final TypeRef> DOUBLE_LIST_TYPE = new
TypeRef>() {
+ };
+ private static final TypeRef> STRING_LIST_TYPE = new
TypeRef>() {
+ };
public static JsonPathEvaluator create(String jsonPath, @Nullable Object
defaultValue) {
try {
@@ -274,23 +285,23 @@ public final class DefaultJsonPathEvaluator implements
JsonPathEvaluator {
reader.readDictIds(docIds, length, dictIdsBuffer, context);
if (dictionary.getValueType() == FieldSpec.DataType.BYTES) {
for (int i = 0; i < length; i++) {
- processList(i, extractFromBytes(dictionary, dictIdsBuffer[i]),
valueBuffer);
+ processList(i, extractFromBytes(dictionary, dictIdsBuffer[i],
INTEGER_LIST_TYPE), valueBuffer);
}
} else {
for (int i = 0; i < length; i++) {
- processList(i, extractFromString(dictionary, dictIdsBuffer[i]),
valueBuffer);
+ processList(i, extractFromString(dictionary, dictIdsBuffer[i],
INTEGER_LIST_TYPE), valueBuffer);
}
}
} else {
switch (reader.getStoredType()) {
case STRING:
for (int i = 0; i < length; i++) {
-processList(i, extractFromString(reader, context, docIds[i]),
valueBuffer);
+processList(i, extractFromString(reader, context, docIds[i],
INTEGER_LIST_TYPE), valueBuffer);
}
break;
case BYTES:
for (int i = 0; i < length; i++) {
-processList(i, extractFromBytes(reader, context, docIds[i]),
valueBuffer);
+processList(i, extractFromBytes(reader, context, docIds[i],
INTEGER_LIST_TYPE), valueBuffer);
}
break;
default:
@@ -305,23 +316,23 @@ public final class DefaultJsonPathEvaluator implements
JsonPathEvaluator {
reader.readDictIds(docIds, length, dictIdsBuffer, context);
if (dictionary.getValueType() == FieldSpec.DataType.BYTES) {
for (int i = 0; i < length; i++) {
- processList(i, extractFromBytes(dictionary, dictIdsBuffer[i]),
valueBuffer);
+ processList(i, extractFromBytes(dictionary, dictIdsBuffer[i],
LONG_LIST_TYPE), valueBuffer);
}
} else {
for (int i = 0; i < length; i++) {
- processList(i, extractFromString(dictionary, dictIdsBuffer[i]),
valueBuffer);
+ processList(i, extractFromString(dictionary, dictIdsBuffer[i],
LONG_LIST_TYPE), valueBuffer);
}
}
} else {
switch (reader.getStoredType()) {
case STRING:
for (int i = 0; i < length; i++) {
-processList(i, extractFromString(reader, context, docIds[i]),
valueBuffer);
+processList(i, extractFromString(reader, context, docIds[i],
LONG_LIST_TYPE), valueBuffer);
}
break;
case BYTES:
for (int i = 0; i < length; i++) {
-processLi
Re: [PR] Rest Endpoint to Create ZNode [pinot]
suddendust commented on code in PR #12497:
URL: https://github.com/apache/pinot/pull/12497#discussion_r1507874579
##
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java:
##
@@ -256,6 +258,71 @@ public SuccessResponse putData(
}
}
+ @POST
+ @Path("/zk/create")
+ @Authorize(targetType = TargetType.CLUSTER, action =
Actions.Cluster.UPDATE_ZNODE)
+ @Authenticate(AccessType.CREATE)
+ @Produces(MediaType.APPLICATION_JSON)
+ @Consumes(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Create a node at a given path")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 204,
message = "No Content"),
+ @ApiResponse(code = 400, message = "Bad Request"), @ApiResponse(code =
500, message = "Internal server error")
+ })
+ public SuccessResponse createNode(
+ @ApiParam(value = "Zookeeper Path, must start with /", required = true)
@QueryParam("path") String path,
+ @ApiParam(value = "Content") @QueryParam("data") @Nullable String data,
+ @ApiParam(value = "TTL of the node. TTL are only honoured for persistent
znodes (access option = 0x40 or 0x80),"
+ + " in which case TTL should be > 0. If access option is not 0x40 or
0x80, it will be ignored, and we can "
+ + "set it to any value, or be ignored", defaultValue = "-1")
+ @QueryParam("ttl") @DefaultValue("-1") int ttl,
+ @ApiParam(value = "accessOption", defaultValue = "1")
@QueryParam("accessOption") @DefaultValue("1")
+ int accessOption, @Nullable String payload) {
+
+path = validateAndNormalizeZKPath(path, false);
+
+//validate ttl range
+if ((accessOption == AccessOption.PERSISTENT_WITH_TTL
+|| accessOption == AccessOption.PERSISTENT_SEQUENTIAL_WITH_TTL) && ttl
<= 0) {
+ throw new ControllerApplicationException(LOGGER, "TTL for persistent
nodes should be > 0",
+ Response.Status.BAD_REQUEST);
+}
+//check if node already exists
+if (_pinotHelixResourceManager.getZKStat(path) != null) {
Review Comment:
Yes, that's a good idea. Flipped it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch release-1.1.0-rc updated (7cec963a88 -> ce94496e2e)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git discard 7cec963a88 [maven-release-plugin] prepare for next development iteration omit 17e33d6cbf [maven-release-plugin] prepare release release-1.1.0-rc This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/release-1.1.0-rc (ce94496e2e) \ O -- O -- O (7cec963a88) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: contrib/pinot-fmpp-maven-plugin/pom.xml| 6 ++- pinot-broker/pom.xml | 5 ++- pinot-clients/pinot-java-client/pom.xml| 5 ++- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 ++- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 ++- pinot-connectors/pinot-flink-connector/pom.xml | 6 ++- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-common/pom.xml| 5 ++- pinot-connectors/pom.xml | 6 ++- pinot-controller/pom.xml | 5 ++- pinot-core/pom.xml | 5 ++- pinot-distribution/pom.xml | 9 +++-- pinot-integration-test-base/pom.xml| 5 ++- pinot-integration-tests/pom.xml| 5 ++- pinot-minion/pom.xml | 5 ++- pinot-perf/pom.xml | 5 ++- .../pinot-batch-ingestion-common/pom.xml | 6 ++- .../pinot-batch-ingestion-hadoop/pom.xml | 6 ++- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 ++- .../pinot-batch-ingestion-spark-3/pom.xml | 6 ++- .../pinot-batch-ingestion-standalone/pom.xml | 6 ++- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 ++- .../pinot-environment/pinot-azure/pom.xml | 6 ++- pinot-plugins/pinot-environment/pom.xml| 6 ++- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +++-- pinot-plugins/pinot-file-system/pom.xml| 6 ++- .../pinot-input-format/pinot-avro-base/pom.xml | 5 ++- .../pinot-input-format/pinot-avro/pom.xml | 5 ++- .../pinot-input-format/pinot-clp-log/pom.xml | 5 ++- .../pinot-confluent-avro/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 ++- .../pinot-input-format/pinot-json/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 ++- .../pinot-input-format/pinot-parquet/pom.xml | 5 ++- .../pinot-input-format/pinot-protobuf/pom.xml | 5 ++- .../pinot-input-format/pinot-thrift/pom.xml| 5 ++- pinot-plugins/pinot-input-format/pom.xml | 6 ++- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pom.xml| 6 ++- .../pinot-minion-builtin-tasks/pom.xml | 5 ++- pinot-plugins/pinot-minion-tasks/pom.xml | 6 ++- .../pinot-segment-uploader-default/pom.xml | 6 ++- pinot-plugins/pinot-segment-uploader/pom.xml | 6 ++- .../pinot-segment-writer-file-based/pom.xml| 6 ++- pinot-plugins/pinot-segment-writer/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++- .../pinot-kafka-base/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 -- .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 ++- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++- pinot-plugins/pom.xml | 11 -- pinot-query-planner/pom.xml| 5 ++- pinot-query-runtime/pom.xml| 5 ++- pinot-segment-local/pom.xml| 5 ++- pinot-segment-spi/pom.xml | 5 ++- pinot-server/pom.xml | 5 ++- pinot-spi/pom.xml | 5 ++- pinot-tools/pom.xml| 5 ++- pom.xml| 11 +++--- 66 files changed, 262 insertions(+), 162 deletions(-) - To unsubscribe, e-mail:
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
zhtaoxiang commented on code in PR #12522:
URL: https://github.com/apache/pinot/pull/12522#discussion_r1507951724
##
pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java:
##
@@ -35,12 +35,14 @@ public enum ServerMeter implements AbstractMetrics.Meter {
DELETED_SEGMENT_COUNT("segments", false),
DELETE_TABLE_FAILURES("tables", false),
REALTIME_ROWS_CONSUMED("rows", true),
+ REALTIME_ROWS_FETCHED("rows", false),
REALTIME_ROWS_FILTERED("rows", false),
INVALID_REALTIME_ROWS_DROPPED("rows", false),
INCOMPLETE_REALTIME_ROWS_CONSUMED("rows", false),
REALTIME_CONSUMPTION_EXCEPTIONS("exceptions", true),
REALTIME_OFFSET_COMMITS("commits", true),
REALTIME_OFFSET_COMMIT_EXCEPTIONS("exceptions", false),
+ STREAM_CONSUMER_CREATE_EXCEPTIONS("exceptions", false),
Review Comment:
this newly created metric is not used anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare release release-1.1.0-rc0
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new 3bae032004 [maven-release-plugin] prepare release release-1.1.0-rc0 3bae032004 is described below commit 3bae03200414c0c61c3c25892c577423cd1b7055 Author: Vivek Iyer Vaidyanathan AuthorDate: Thu Feb 29 10:01:33 2024 -0800 [maven-release-plugin] prepare release release-1.1.0-rc0 --- contrib/pinot-fmpp-maven-plugin/pom.xml| 6 +-- pinot-broker/pom.xml | 5 +-- pinot-clients/pinot-java-client/pom.xml| 5 +-- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 +-- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 +-- pinot-connectors/pinot-flink-connector/pom.xml | 6 +-- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-common/pom.xml| 5 +-- pinot-connectors/pom.xml | 6 +-- pinot-controller/pom.xml | 5 +-- pinot-core/pom.xml | 5 +-- pinot-distribution/pom.xml | 9 ++--- pinot-integration-test-base/pom.xml| 5 +-- pinot-integration-tests/pom.xml| 5 +-- pinot-minion/pom.xml | 5 +-- pinot-perf/pom.xml | 5 +-- .../pinot-batch-ingestion-common/pom.xml | 6 +-- .../pinot-batch-ingestion-hadoop/pom.xml | 6 +-- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 +-- .../pinot-batch-ingestion-spark-3/pom.xml | 6 +-- .../pinot-batch-ingestion-standalone/pom.xml | 6 +-- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 +-- .../pinot-environment/pinot-azure/pom.xml | 6 +-- pinot-plugins/pinot-environment/pom.xml| 6 +-- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 +-- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 ++--- pinot-plugins/pinot-file-system/pom.xml| 6 +-- .../pinot-input-format/pinot-avro-base/pom.xml | 5 +-- .../pinot-input-format/pinot-avro/pom.xml | 5 +-- .../pinot-input-format/pinot-clp-log/pom.xml | 5 +-- .../pinot-confluent-avro/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 +-- .../pinot-input-format/pinot-json/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 +-- .../pinot-input-format/pinot-parquet/pom.xml | 5 +-- .../pinot-input-format/pinot-protobuf/pom.xml | 5 +-- .../pinot-input-format/pinot-thrift/pom.xml| 5 +-- pinot-plugins/pinot-input-format/pom.xml | 6 +-- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pom.xml| 6 +-- .../pinot-minion-builtin-tasks/pom.xml | 5 +-- pinot-plugins/pinot-minion-tasks/pom.xml | 6 +-- .../pinot-segment-uploader-default/pom.xml | 6 +-- pinot-plugins/pinot-segment-uploader/pom.xml | 6 +-- .../pinot-segment-writer-file-based/pom.xml| 6 +-- pinot-plugins/pinot-segment-writer/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 +-- .../pinot-kafka-base/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 ++ .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 +-- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 +-- pinot-plugins/pom.xml | 11 ++ pinot-query-planner/pom.xml| 5 +-- pinot-query-runtime/pom.xml| 5 +-- pinot-segment-local/pom.xml| 5 +-- pinot-segment-spi/pom.xml | 5 +-- pinot-server/pom.xml | 5 +-- pinot-spi/pom.xml | 5 +-- pinot-tools/pom.xml| 5 +-- pom.xml| 13 +++ 66 files changed, 163 insertions(+), 263 deletions(-) diff --git a/contrib/pinot-fmpp-maven-plugin/pom.xml b/contrib/pinot-fmpp-maven-plugin/pom.xml index 94df4afc49..94fc21d142 100644 --- a/contrib/pinot-fmpp-maven-plugin/pom.xml +++ b/contrib/pinot-fmpp-maven-plugin/pom.xml @@ -19,14 +19,12 @@ under the License. --> -http://maven.apache.org/POM/4.0.0"; -
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
suddendust commented on code in PR #12522:
URL: https://github.com/apache/pinot/pull/12522#discussion_r1507986026
##
pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java:
##
@@ -35,12 +35,14 @@ public enum ServerMeter implements AbstractMetrics.Meter {
DELETED_SEGMENT_COUNT("segments", false),
DELETE_TABLE_FAILURES("tables", false),
REALTIME_ROWS_CONSUMED("rows", true),
+ REALTIME_ROWS_FETCHED("rows", false),
REALTIME_ROWS_FILTERED("rows", false),
INVALID_REALTIME_ROWS_DROPPED("rows", false),
INCOMPLETE_REALTIME_ROWS_CONSUMED("rows", false),
REALTIME_CONSUMPTION_EXCEPTIONS("exceptions", true),
REALTIME_OFFSET_COMMITS("commits", true),
REALTIME_OFFSET_COMMIT_EXCEPTIONS("exceptions", false),
+ STREAM_CONSUMER_CREATE_EXCEPTIONS("exceptions", false),
Review Comment:
That code wasn't checked-in, sorry about that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
zhtaoxiang commented on code in PR #12522:
URL: https://github.com/apache/pinot/pull/12522#discussion_r1508002971
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##
@@ -1661,22 +1666,35 @@ private void makeStreamConsumer(String reason) {
closePartitionGroupConsumer();
}
_segmentLogger.info("Creating new stream consumer for topic partition {} ,
reason: {}", _clientId, reason);
-_partitionGroupConsumer =
-_streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
-_partitionGroupConsumer.start(_currentOffset);
+try {
+ _partitionGroupConsumer =
+ _streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
+ _partitionGroupConsumer.start(_currentOffset);
+} catch (Exception e) {
+ _segmentLogger.error("Faced exception while trying to recreate stream
consumer for topic partition {} reason {}",
+ _clientId, reason, e);
+ _serverMetrics.addMeteredTableValue(_clientId,
ServerMeter.STREAM_CONSUMER_CREATE_EXCEPTIONS, 1L);
+ throw e;
+}
}
/**
* Checkpoints existing consumer before creating a new consumer instance
* Assumes there is a valid instance of {@link PartitionGroupConsumer}
*/
private void recreateStreamConsumer(String reason) {
-_segmentLogger.info("Recreating stream consumer for topic partition {},
reason: {}", _clientId, reason);
-_currentOffset = _partitionGroupConsumer.checkpoint(_currentOffset);
-closePartitionGroupConsumer();
-_partitionGroupConsumer =
-_streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
-_partitionGroupConsumer.start(_currentOffset);
+try {
+ _segmentLogger.info("Recreating stream consumer for topic partition {},
reason: {}", _clientId, reason);
+ _currentOffset = _partitionGroupConsumer.checkpoint(_currentOffset);
+ closePartitionGroupConsumer();
Review Comment:
Is it possible that those 2 lines throw exceptions? If possible, then the
exception is not related to consumer creation.
Should we only try catch line 1690 and 1691?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]
suddendust commented on code in PR #12522:
URL: https://github.com/apache/pinot/pull/12522#discussion_r1508009397
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##
@@ -1661,22 +1666,35 @@ private void makeStreamConsumer(String reason) {
closePartitionGroupConsumer();
}
_segmentLogger.info("Creating new stream consumer for topic partition {} ,
reason: {}", _clientId, reason);
-_partitionGroupConsumer =
-_streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
-_partitionGroupConsumer.start(_currentOffset);
+try {
+ _partitionGroupConsumer =
+ _streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
+ _partitionGroupConsumer.start(_currentOffset);
+} catch (Exception e) {
+ _segmentLogger.error("Faced exception while trying to recreate stream
consumer for topic partition {} reason {}",
+ _clientId, reason, e);
+ _serverMetrics.addMeteredTableValue(_clientId,
ServerMeter.STREAM_CONSUMER_CREATE_EXCEPTIONS, 1L);
+ throw e;
+}
}
/**
* Checkpoints existing consumer before creating a new consumer instance
* Assumes there is a valid instance of {@link PartitionGroupConsumer}
*/
private void recreateStreamConsumer(String reason) {
-_segmentLogger.info("Recreating stream consumer for topic partition {},
reason: {}", _clientId, reason);
-_currentOffset = _partitionGroupConsumer.checkpoint(_currentOffset);
-closePartitionGroupConsumer();
-_partitionGroupConsumer =
-_streamConsumerFactory.createPartitionGroupConsumer(_clientId,
_partitionGroupConsumptionStatus);
-_partitionGroupConsumer.start(_currentOffset);
+try {
+ _segmentLogger.info("Recreating stream consumer for topic partition {},
reason: {}", _clientId, reason);
+ _currentOffset = _partitionGroupConsumer.checkpoint(_currentOffset);
+ closePartitionGroupConsumer();
Review Comment:
If they throw an exception, the consumer wouldn't get created right? And
`_currentOffset = _partitionGroupConsumer.checkpoint(_currentOffset);` does
throw a runtime exception (at least in code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Rest Endpoint to Create ZNode [pinot]
Jackie-Jiang merged PR #12497: URL: https://github.com/apache/pinot/pull/12497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Rest Endpoint to Create ZNode (#12497)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2249be357b Rest Endpoint to Create ZNode (#12497)
2249be357b is described below
commit 2249be357b89a27b1fb74879b4235393dd4e53d0
Author: Prashant Pandey <84911643+suddend...@users.noreply.github.com>
AuthorDate: Fri Mar 1 00:12:53 2024 +0530
Rest Endpoint to Create ZNode (#12497)
---
.../api/resources/ZookeeperResource.java | 62 ++
.../helix/core/PinotHelixResourceManager.java | 4 ++
.../utils/builder/ControllerRequestURLBuilder.java | 8 +++
3 files changed, 74 insertions(+)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java
index f6f1126ec1..377424125d 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java
@@ -44,6 +44,7 @@ import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.GET;
+import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
@@ -52,6 +53,7 @@ import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.commons.lang.StringUtils;
+import org.apache.helix.AccessOption;
import org.apache.helix.zookeeper.datamodel.ZNRecord;
import org.apache.helix.zookeeper.introspect.CodehausJacksonIntrospector;
import org.apache.pinot.controller.api.access.AccessType;
@@ -256,6 +258,66 @@ public class ZookeeperResource {
}
}
+ @POST
+ @Path("/zk/create")
+ @Authorize(targetType = TargetType.CLUSTER, action =
Actions.Cluster.UPDATE_ZNODE)
+ @Authenticate(AccessType.CREATE)
+ @Produces(MediaType.APPLICATION_JSON)
+ @Consumes(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Create a node at a given path")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 204,
message = "No Content"),
+ @ApiResponse(code = 400, message = "Bad Request"), @ApiResponse(code =
500, message = "Internal server error")
+ })
+ public SuccessResponse createNode(
+ @ApiParam(value = "Zookeeper Path, must start with /", required = true)
@QueryParam("path") String path,
+ @ApiParam(value = "TTL of the node. TTL are only honoured for persistent
znodes (access option = 0x40 or 0x80),"
+ + " in which case TTL should be > 0. If access option is not 0x40 or
0x80, it will be ignored, and we can "
+ + "set it to any value, or just ignore it", defaultValue = "-1")
@QueryParam("ttl") @DefaultValue("-1")
+ int ttl, @ApiParam(value = "accessOption", defaultValue = "1")
@QueryParam("accessOption") @DefaultValue("1")
+ int accessOption, String payload) {
+
+path = validateAndNormalizeZKPath(path, false);
+
+//validate ttl range
+if ((accessOption == AccessOption.PERSISTENT_WITH_TTL
+|| accessOption == AccessOption.PERSISTENT_SEQUENTIAL_WITH_TTL) && ttl
<= 0) {
+ throw new ControllerApplicationException(LOGGER, "TTL for persistent
nodes should be > 0",
+ Response.Status.BAD_REQUEST);
+}
+
+if (StringUtils.isEmpty(payload)) {
+ throw new ControllerApplicationException(LOGGER, "Must provide payload",
Response.Status.BAD_REQUEST);
+}
+ZNRecord znRecord;
+try {
+ znRecord = MAPPER.readValue(payload, ZNRecord.class);
+} catch (Exception e) {
+ throw new ControllerApplicationException(LOGGER, "Failed to deserialize
the data", Response.Status.BAD_REQUEST,
+ e);
+}
+
+boolean result;
+try {
+ result = _pinotHelixResourceManager.createZKNode(path, znRecord,
accessOption, ttl);
+} catch (Exception e) {
+ throw new ControllerApplicationException(LOGGER, "Failed to create znode
at path: " + path,
+ Response.Status.INTERNAL_SERVER_ERROR, e);
+}
+if (result) {
+ return new SuccessResponse("Successfully created node at path: " + path);
+} else {
+ //check if node already exists
+ if (_pinotHelixResourceManager.getZKStat(path) != null) {
+throw new ControllerApplicationException(LOGGER, "ZNode already exists
at path: " + path,
+Response.Status.BAD_REQUEST);
+ } else {
+throw new ControllerApplicationException(LOGGER, "Failed to create
znode at path: " + path,
+Response.Status.INTERNAL_SERVER_ERROR);
+ }
+}
+ }
+
@GET
@Path("/zk/ls")
@Authorize(targetType = TargetType.CLUSTER, action =
Actions.Cluster.GET_ZNODE)
diff --git
a/pinot-controller/src/main/java/org/apache/p
(pinot) branch release-1.1.0-rc updated (3bae032004 -> ce94496e2e)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git discard 3bae032004 [maven-release-plugin] prepare release release-1.1.0-rc0 This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/release-1.1.0-rc (ce94496e2e) \ O -- O -- O (3bae032004) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: contrib/pinot-fmpp-maven-plugin/pom.xml| 6 ++- pinot-broker/pom.xml | 5 ++- pinot-clients/pinot-java-client/pom.xml| 5 ++- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 ++- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 ++- pinot-connectors/pinot-flink-connector/pom.xml | 6 ++- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-common/pom.xml| 5 ++- pinot-connectors/pom.xml | 6 ++- pinot-controller/pom.xml | 5 ++- pinot-core/pom.xml | 5 ++- pinot-distribution/pom.xml | 9 +++-- pinot-integration-test-base/pom.xml| 5 ++- pinot-integration-tests/pom.xml| 5 ++- pinot-minion/pom.xml | 5 ++- pinot-perf/pom.xml | 5 ++- .../pinot-batch-ingestion-common/pom.xml | 6 ++- .../pinot-batch-ingestion-hadoop/pom.xml | 6 ++- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 ++- .../pinot-batch-ingestion-spark-3/pom.xml | 6 ++- .../pinot-batch-ingestion-standalone/pom.xml | 6 ++- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 ++- .../pinot-environment/pinot-azure/pom.xml | 6 ++- pinot-plugins/pinot-environment/pom.xml| 6 ++- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +++-- pinot-plugins/pinot-file-system/pom.xml| 6 ++- .../pinot-input-format/pinot-avro-base/pom.xml | 5 ++- .../pinot-input-format/pinot-avro/pom.xml | 5 ++- .../pinot-input-format/pinot-clp-log/pom.xml | 5 ++- .../pinot-confluent-avro/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 ++- .../pinot-input-format/pinot-json/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 ++- .../pinot-input-format/pinot-parquet/pom.xml | 5 ++- .../pinot-input-format/pinot-protobuf/pom.xml | 5 ++- .../pinot-input-format/pinot-thrift/pom.xml| 5 ++- pinot-plugins/pinot-input-format/pom.xml | 6 ++- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pom.xml| 6 ++- .../pinot-minion-builtin-tasks/pom.xml | 5 ++- pinot-plugins/pinot-minion-tasks/pom.xml | 6 ++- .../pinot-segment-uploader-default/pom.xml | 6 ++- pinot-plugins/pinot-segment-uploader/pom.xml | 6 ++- .../pinot-segment-writer-file-based/pom.xml| 6 ++- pinot-plugins/pinot-segment-writer/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++- .../pinot-kafka-base/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 -- .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 ++- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++- pinot-plugins/pom.xml | 11 -- pinot-query-planner/pom.xml| 5 ++- pinot-query-runtime/pom.xml| 5 ++- pinot-segment-local/pom.xml| 5 ++- pinot-segment-spi/pom.xml | 5 ++- pinot-server/pom.xml | 5 ++- pinot-spi/pom.xml | 5 ++- pinot-tools/pom.xml| 5 ++- pom.xml| 13 --- 66 files changed, 263 insertions(+), 163 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h.
Re: [PR] Rest Endpoint to Create ZNode [pinot]
Jackie-Jiang commented on code in PR #12497:
URL: https://github.com/apache/pinot/pull/12497#discussion_r1508023342
##
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ZookeeperResource.java:
##
@@ -256,6 +258,67 @@ public SuccessResponse putData(
}
}
+ @POST
+ @Path("/zk/create")
+ @Authorize(targetType = TargetType.CLUSTER, action =
Actions.Cluster.UPDATE_ZNODE)
+ @Authenticate(AccessType.CREATE)
+ @Produces(MediaType.APPLICATION_JSON)
+ @Consumes(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Create a node at a given path")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 204,
message = "No Content"),
+ @ApiResponse(code = 400, message = "Bad Request"), @ApiResponse(code =
500, message = "Internal server error")
+ })
+ public SuccessResponse createNode(
+ @ApiParam(value = "Zookeeper Path, must start with /", required = true)
@QueryParam("path") String path,
+ @ApiParam(value = "TTL of the node. TTL are only honoured for persistent
znodes (access option = 0x40 or 0x80),"
+ + " in which case TTL should be > 0. If access option is not 0x40 or
0x80, it will be ignored, and we can "
+ + "set it to any value, or just ignore it", defaultValue = "-1")
@QueryParam("ttl") @DefaultValue("-1")
+ int ttl, @ApiParam(value = "accessOption", defaultValue = "1")
@QueryParam("accessOption") @DefaultValue("1")
+ int accessOption, String payload) {
+
+path = validateAndNormalizeZKPath(path, false);
+
+//validate ttl range
+if ((accessOption == AccessOption.PERSISTENT_WITH_TTL
+|| accessOption == AccessOption.PERSISTENT_SEQUENTIAL_WITH_TTL) && ttl
<= 0) {
+ throw new ControllerApplicationException(LOGGER, "TTL for persistent
nodes should be > 0",
+ Response.Status.BAD_REQUEST);
+}
+
+if (StringUtils.isEmpty(payload)) {
+ throw new ControllerApplicationException(LOGGER, "Must provide data
through query parameter or payload",
+ Response.Status.BAD_REQUEST);
Review Comment:
```suggestion
throw new ControllerApplicationException(LOGGER, "Must provide
payload", Response.Status.BAD_REQUEST);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) annotated tag release-1.1.0-rc0 deleted (was d236eef9b2)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to annotated tag release-1.1.0-rc0 in repository https://gitbox.apache.org/repos/asf/pinot.git *** WARNING: tag release-1.1.0-rc0 was deleted! *** tag was d236eef9b2 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Allow passing database context through `database` http header [pinot]
Jackie-Jiang commented on code in PR #12417:
URL: https://github.com/apache/pinot/pull/12417#discussion_r1508053974
##
pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotBrokerRouting.java:
##
@@ -61,8 +69,23 @@ public class PinotBrokerRouting {
@ApiResponse(code = 200, message = "Success"),
@ApiResponse(code = 500, message = "Internal server error")
})
- public String buildRouting(
+ public String buildRouting(@Context HttpHeaders headers,
Review Comment:
(nit, convention) We usually put this as the last argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Adding record reader config/context param to record transformer [pinot]
klsince commented on code in PR #12520:
URL: https://github.com/apache/pinot/pull/12520#discussion_r1508078762
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/RecordTransformer.java:
##
@@ -43,4 +44,15 @@ default boolean isNoOp() {
*/
@Nullable
GenericRow transform(GenericRow record);
+
+ /**
+ * Transforms a record based on some custom rules using record reader
context.
+ * @param record Record to transform
+ * @return Transformed record, or {@code null} if the record does not follow
certain rules.
+ */
+ @Nullable
+ default GenericRow transformUsingRecordReaderContext(GenericRow record,
Review Comment:
just call it `transform(record, config)`, as it's easy to tell them apart by
param list.
I'd suggest to pass in RecordReaderConfig (the interface), and make
RecordReaderFileConfig implement RecordReaderConfig (which is an empty
interface right now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Adding record reader config/context param to record transformer [pinot]
swaminathanmanish commented on code in PR #12520:
URL: https://github.com/apache/pinot/pull/12520#discussion_r1508091484
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/RecordTransformer.java:
##
@@ -43,4 +44,15 @@ default boolean isNoOp() {
*/
@Nullable
GenericRow transform(GenericRow record);
+
+ /**
+ * Transforms a record based on some custom rules using record reader
context.
+ * @param record Record to transform
+ * @return Transformed record, or {@code null} if the record does not follow
certain rules.
+ */
+ @Nullable
+ default GenericRow transformUsingRecordReaderContext(GenericRow record,
Review Comment:
Yes this makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare release release-1.1.0-rc0
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new 56fc5a6dfc [maven-release-plugin] prepare release release-1.1.0-rc0 56fc5a6dfc is described below commit 56fc5a6dfc1f25020edd2a648e841038f8a77197 Author: Vivek Iyer Vaidyanathan AuthorDate: Thu Feb 29 11:28:01 2024 -0800 [maven-release-plugin] prepare release release-1.1.0-rc0 --- contrib/pinot-fmpp-maven-plugin/pom.xml| 6 +-- pinot-broker/pom.xml | 5 +-- pinot-clients/pinot-java-client/pom.xml| 5 +-- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 +-- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 +-- pinot-connectors/pinot-flink-connector/pom.xml | 6 +-- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-common/pom.xml| 5 +-- pinot-connectors/pom.xml | 6 +-- pinot-controller/pom.xml | 5 +-- pinot-core/pom.xml | 5 +-- pinot-distribution/pom.xml | 9 ++--- pinot-integration-test-base/pom.xml| 5 +-- pinot-integration-tests/pom.xml| 5 +-- pinot-minion/pom.xml | 5 +-- pinot-perf/pom.xml | 5 +-- .../pinot-batch-ingestion-common/pom.xml | 6 +-- .../pinot-batch-ingestion-hadoop/pom.xml | 6 +-- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 +-- .../pinot-batch-ingestion-spark-3/pom.xml | 6 +-- .../pinot-batch-ingestion-standalone/pom.xml | 6 +-- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 +-- .../pinot-environment/pinot-azure/pom.xml | 6 +-- pinot-plugins/pinot-environment/pom.xml| 6 +-- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 +-- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 ++--- pinot-plugins/pinot-file-system/pom.xml| 6 +-- .../pinot-input-format/pinot-avro-base/pom.xml | 5 +-- .../pinot-input-format/pinot-avro/pom.xml | 5 +-- .../pinot-input-format/pinot-clp-log/pom.xml | 5 +-- .../pinot-confluent-avro/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 +-- .../pinot-input-format/pinot-json/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 +-- .../pinot-input-format/pinot-parquet/pom.xml | 5 +-- .../pinot-input-format/pinot-protobuf/pom.xml | 5 +-- .../pinot-input-format/pinot-thrift/pom.xml| 5 +-- pinot-plugins/pinot-input-format/pom.xml | 6 +-- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pom.xml| 6 +-- .../pinot-minion-builtin-tasks/pom.xml | 5 +-- pinot-plugins/pinot-minion-tasks/pom.xml | 6 +-- .../pinot-segment-uploader-default/pom.xml | 6 +-- pinot-plugins/pinot-segment-uploader/pom.xml | 6 +-- .../pinot-segment-writer-file-based/pom.xml| 6 +-- pinot-plugins/pinot-segment-writer/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 +-- .../pinot-kafka-base/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 ++ .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 +-- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 +-- pinot-plugins/pom.xml | 11 ++ pinot-query-planner/pom.xml| 5 +-- pinot-query-runtime/pom.xml| 5 +-- pinot-segment-local/pom.xml| 5 +-- pinot-segment-spi/pom.xml | 5 +-- pinot-server/pom.xml | 5 +-- pinot-spi/pom.xml | 5 +-- pinot-tools/pom.xml| 5 +-- pom.xml| 13 +++ 66 files changed, 163 insertions(+), 263 deletions(-) diff --git a/contrib/pinot-fmpp-maven-plugin/pom.xml b/contrib/pinot-fmpp-maven-plugin/pom.xml index 94df4afc49..94fc21d142 100644 --- a/contrib/pinot-fmpp-maven-plugin/pom.xml +++ b/contrib/pinot-fmpp-maven-plugin/pom.xml @@ -19,14 +19,12 @@ under the License. --> -http://maven.apache.org/POM/4.0.0"; -
(pinot) annotated tag release-1.1.0-rc0 updated (56fc5a6dfc -> 6ad905acf2)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to annotated tag release-1.1.0-rc0 in repository https://gitbox.apache.org/repos/asf/pinot.git *** WARNING: tag release-1.1.0-rc0 was modified! *** from 56fc5a6dfc (commit) to 6ad905acf2 (tag) tagging 56fc5a6dfc1f25020edd2a648e841038f8a77197 (commit) by Vivek Iyer Vaidyanathan on Thu Feb 29 11:28:15 2024 -0800 - Log - [maven-release-plugin] copy for tag release-1.1.0-rc0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new e75703bda3 [maven-release-plugin] prepare for next development iteration e75703bda3 is described below commit e75703bda3f1df8991ef7c7ed10113d32b618dbd Author: Vivek Iyer Vaidyanathan AuthorDate: Thu Feb 29 11:28:18 2024 -0800 [maven-release-plugin] prepare for next development iteration --- contrib/pinot-fmpp-maven-plugin/pom.xml | 2 +- pinot-broker/pom.xml | 2 +- pinot-clients/pinot-java-client/pom.xml | 2 +- pinot-clients/pinot-jdbc-client/pom.xml | 2 +- pinot-clients/pom.xml | 2 +- pinot-common/pom.xml | 2 +- pinot-compatibility-verifier/pom.xml | 2 +- pinot-connectors/pinot-flink-connector/pom.xml| 2 +- pinot-connectors/pinot-spark-2-connector/pom.xml | 2 +- pinot-connectors/pinot-spark-3-connector/pom.xml | 2 +- pinot-connectors/pinot-spark-common/pom.xml | 2 +- pinot-connectors/pom.xml | 2 +- pinot-controller/pom.xml | 2 +- pinot-core/pom.xml| 2 +- pinot-distribution/pom.xml| 2 +- pinot-integration-test-base/pom.xml | 2 +- pinot-integration-tests/pom.xml | 2 +- pinot-minion/pom.xml | 2 +- pinot-perf/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-common/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pom.xml | 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-spark-3/pom.xml | 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-standalone/pom.xml| 2 +- pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +- pinot-plugins/pinot-environment/pinot-azure/pom.xml | 2 +- pinot-plugins/pinot-environment/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-adls/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 2 +- pinot-plugins/pinot-file-system/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-clp-log/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-csv/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-json/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-orc/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-parquet/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-thrift/pom.xml | 2 +- pinot-plugins/pinot-input-format/pom.xml | 2 +- pinot-plugins/pinot-metrics/pinot-dropwizard/pom.xml | 2 +- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 2 +- pinot-plugins/pinot-metrics/pom.xml | 2 +- pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/pom.xml | 2 +- pinot-plugins/pinot-minion-tasks/pom.xml | 2 +- .../pinot-segment-uploader/pinot-segment-uploader-default/pom.xml | 2 +- pinot-plugins/pinot-segment-uploader/pom.xml | 2 +- .../pinot-segment-writer/pinot-segment-writer-file-based/pom.xml | 2 +- pinot-plugins/pinot-segment-writer/pom.xml| 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kinesis/pom.xml| 2 +- pinot-plugins/pinot-stream-ingestion/pinot-pulsar/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pom.xml
Re: [PR] Adding record reader config/context param to record transformer [pinot]
klsince commented on code in PR #12520:
URL: https://github.com/apache/pinot/pull/12520#discussion_r1508106925
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/RecordTransformer.java:
##
@@ -43,4 +43,15 @@ default boolean isNoOp() {
*/
@Nullable
GenericRow transform(GenericRow record);
+
+ /**
+ * Transforms a record based on some custom rules using record reader
context.
+ * @param record Record to transform
+ * @return Transformed record, or {@code null} if the record does not follow
certain rules.
+ */
+ @Nullable
+ default GenericRow transform(GenericRow record,
Review Comment:
nit: format? looks like the two params can be put on one line :p
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Adding record reader config/context param to record transformer [pinot]
klsince merged PR #12520: URL: https://github.com/apache/pinot/pull/12520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (2249be357b -> f0fcbd8701)
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 2249be357b Rest Endpoint to Create ZNode (#12497) add f0fcbd8701 Adding record reader config/context param to record transformer (#12520) No new revisions were added by this update. Summary of changes: .../pinot/core/segment/processing/mapper/SegmentMapper.java | 11 +-- .../local/recordtransformer/CompositeTransformer.java | 13 ++--- .../segment/local/recordtransformer/RecordTransformer.java | 12 +++- .../pinot/spi/data/readers/RecordReaderFileConfig.java | 2 +- 4 files changed, 27 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] adding support of mega_bytes when configuring broker response size [pinot]
Zhenyun20023 commented on PR #12510: URL: https://github.com/apache/pinot/pull/12510#issuecomment-1972037243 @Jackie-Jiang , I made changes per your suggestions. Can you take a look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Modify consumingSegmentsInfo endpoint to indicate how many servers failed [pinot]
codecov-commenter commented on PR #12523: URL: https://github.com/apache/pinot/pull/12523#issuecomment-1972090121 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12523?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 61.68%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`5593975`)](https://app.codecov.io/gh/apache/pinot/pull/12523?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 33 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12523 +/- ## - Coverage 61.75% 61.68% -0.07% Complexity 207 207 Files 2436 2450 +14 Lines133233 133518 +285 Branches 2063620688 +52 + Hits 8227482363 +89 - Misses4491145074 +163 - Partials 6048 6081 +33 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.64% <100.00%> (-0.06%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.56% <100.00%> (-0.06%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.66% <100.00%> (-0.08%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.54% <100.00%> (+33.81%)` | :arrow_up: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.68% <100.00%> (-0.07%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.68% <100.00%> (-0.07%)` | :arrow_down: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.84% <ø> (-0.05%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12523/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.68% <100.00%> (-0.05%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by
Re: [I] Provide pinot schema when initializing StreamMessageDecoder [pinot]
Jackie-Jiang commented on issue #12521: URL: https://github.com/apache/pinot/issues/12521#issuecomment-1972111556 Yes. I think we can modify the javadoc, and it should apply to all decoders. It doesn't make sense to have a decoder that decodes nothing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] add insecure mode when Pinot uses TLS connections [pinot]
zhtaoxiang opened a new pull request, #12525: URL: https://github.com/apache/pinot/pull/12525 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Revert " Adding record reader config/context param to record transformer (#12520)" [pinot]
klsince opened a new pull request, #12526: URL: https://github.com/apache/pinot/pull/12526 This reverts commit f0fcbd8701e025495a136e644262c83607e1eaa7. The interface RecordRecordConfig was mistaken as the TransformerContext, that's supposed to be added and passed to transform() to access context like where the record comes from. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Revert " Adding record reader config/context param to record transformer (#12520)" [pinot]
Jackie-Jiang merged PR #12526: URL: https://github.com/apache/pinot/pull/12526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (f0fcbd8701 -> 8972bc6066)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from f0fcbd8701 Adding record reader config/context param to record transformer (#12520) add 8972bc6066 Revert " Adding record reader config/context param to record transformer (#12520)" (#12526) No new revisions were added by this update. Summary of changes: .../pinot/core/segment/processing/mapper/SegmentMapper.java | 11 ++- .../local/recordtransformer/CompositeTransformer.java | 13 +++-- .../segment/local/recordtransformer/RecordTransformer.java | 12 +--- .../pinot/spi/data/readers/RecordReaderFileConfig.java | 2 +- 4 files changed, 11 insertions(+), 27 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]
Jackie-Jiang commented on code in PR #12227:
URL: https://github.com/apache/pinot/pull/12227#discussion_r1508308471
##
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java:
##
@@ -263,11 +264,14 @@ public void aggregate(int length, AggregationResultHolder
aggregationResultHolde
// For dictionary-encoded expression, store dictionary ids into the dictId
map
Dictionary dictionary = blockValSet.getDictionary();
if (dictionary != null) {
- int[] dictIds = blockValSet.getDictionaryIdsSV();
+
Int2IntOpenHashMap dictIdValueMap =
getDictIdCountMap(aggregationResultHolder, dictionary);
- for (int i = 0; i < length; i++) {
-dictIdValueMap.merge(dictIds[i], 1, Integer::sum);
- }
+ int[] dicts = blockValSet.getDictionaryIdsSV();
Review Comment:
lol, you really hate the name `dictIds`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]
Jackie-Jiang commented on code in PR #12475:
URL: https://github.com/apache/pinot/pull/12475#discussion_r1508311185
##
pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.validate;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlCastFunction;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.util.Static;
+
+
+public class BytesCastVisitor extends SqlBasicVisitor {
+
+ private final SqlValidator _originalValidator;
+
+ public BytesCastVisitor(SqlValidator originalValidator) {
+_originalValidator = originalValidator;
+ }
+
+ @SuppressWarnings("checkstyle:WhitespaceAround")
+ @Override
+ public Void visit(SqlCall call) {
+if (call.getOperator() instanceof SqlCastFunction) {
+ List operands = call.getOperandList();
+ SqlNode sqlNode = operands.get(1);
+ Preconditions.checkState(sqlNode instanceof SqlDataTypeSpec);
+ RelDataType toType = ((SqlDataTypeSpec)
sqlNode).deriveType(_originalValidator);
+ if (!SqlTypeUtil.isBinary(toType)) {
+return super.visit(call);
Review Comment:
I see, make sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]
Jackie-Jiang commented on code in PR #12475:
URL: https://github.com/apache/pinot/pull/12475#discussion_r1508312558
##
pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.validate;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlCastFunction;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.util.Static;
+
+
+public class BytesCastVisitor extends SqlBasicVisitor {
+
+ private final SqlValidator _originalValidator;
+
+ public BytesCastVisitor(SqlValidator originalValidator) {
+_originalValidator = originalValidator;
+ }
+
+ @SuppressWarnings("checkstyle:WhitespaceAround")
Review Comment:
^^ Should we remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Record enricher [pinot]
Jackie-Jiang commented on PR #12243: URL: https://github.com/apache/pinot/pull/12243#issuecomment-1972155063 Yes. I prefer keeping one single interface for enrichment/transformation on records, unless there are fundamental differences between them. @saurabhd336 what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Egalpin/skip indexes minor changes [pinot]
egalpin commented on PR #12514: URL: https://github.com/apache/pinot/pull/12514#issuecomment-1972157886 @Jackie-Jiang all set, I believe the first tests that failed were due to noisy remote test environment. I tried to make a change to account for the failures in https://github.com/apache/pinot/pull/12514/commits/733d17973ac677098527db527e2b7387b6ff9a9f, which was simply misguided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] add insecure mode when Pinot uses TLS connections [pinot]
codecov-commenter commented on PR #12525: URL: https://github.com/apache/pinot/pull/12525#issuecomment-1972164101 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `3.44828%` with `28 lines` in your changes are missing coverage. Please review. > Project coverage is 34.86%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`1903f78`)](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...ache/pinot/common/utils/tls/RenewableTlsUtils.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvdGxzL1JlbmV3YWJsZVRsc1V0aWxzLmphdmE=) | 16.66% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...ache/pinot/common/utils/tls/PinotInsecureMode.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvdGxzL1Bpbm90SW5zZWN1cmVNb2RlLmphdmE=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...va/org/apache/pinot/common/utils/tls/TlsUtils.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvdGxzL1Rsc1V0aWxzLmphdmE=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...e/pinot/broker/broker/helix/BaseBrokerStarter.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jhc2VCcm9rZXJTdGFydGVyLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...apache/pinot/controller/BaseControllerStarter.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9CYXNlQ29udHJvbGxlclN0YXJ0ZXIuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...che/pinot/core/transport/grpc/GrpcQueryServer.java](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS90cmFuc3BvcnQvZ3JwYy9HcnBjUXVlcnlTZXJ2ZXIuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12525?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...org/apache/pinot/cor
Re: [I] Auto-tuning Pinot real-time segment size based on actual stream data consumption [pinot]
chenboat commented on issue #12513: URL: https://github.com/apache/pinot/issues/12513#issuecomment-1972172243 My proposal is to use number of *bytes* consumed in the previous segment instead of the number of *rows* consumed to determine the current segment consumption target. In particular, (1) The number of bytes consumed is recorded as part of the zkMetadata per segments (2) When a new segment is created, there is a new mode/config to allow the segment to consume until the byte limit. (3) When one server replica reaches that limit, it will commit the segment. (4) The rest of the replicas will download and replace their current segments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add minionInstanceTag config to minion-tasks for resource isolation [pinot]
Jackie-Jiang commented on code in PR #12459:
URL: https://github.com/apache/pinot/pull/12459#discussion_r1508329145
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java:
##
@@ -534,66 +535,88 @@ private synchronized Map
scheduleTasks(List tableNamesWi
* Returns the task name, or {@code null} if no task is scheduled.
*/
@Nullable
- private String scheduleTask(PinotTaskGenerator taskGenerator,
List enabledTableConfigs,
+ private List scheduleTask(PinotTaskGenerator taskGenerator,
List enabledTableConfigs,
boolean isLeader) {
LOGGER.info("Trying to schedule task type: {}, isLeader: {}",
taskGenerator.getTaskType(), isLeader);
-List pinotTaskConfigs;
-try {
- /* TODO taskGenerator may skip generating tasks for some of the tables
being passed to it.
-In that case, we should not be storing success timestamps for those
table. Same with exceptions that should
-only be associated with the table for which it was raised and not
every eligible table. We can have the
-generateTasks() return a list of TaskGeneratorMostRecentRunInfo for
each table
- */
- pinotTaskConfigs = taskGenerator.generateTasks(enabledTableConfigs);
- long successRunTimestamp = System.currentTimeMillis();
- for (TableConfig tableConfig : enabledTableConfigs) {
-
_taskManagerStatusCache.saveTaskGeneratorInfo(tableConfig.getTableName(),
taskGenerator.getTaskType(),
+Map> pinotMinionInstanceToTaskConfigs = new
HashMap<>();
+String taskType = taskGenerator.getTaskType();
+for (TableConfig tableConfig : enabledTableConfigs) {
+ String tableName = tableConfig.getTableName();
+ try {
+String minionInstanceTag =
taskGenerator.getMinionInstanceTag(tableConfig);
+List pinotTaskConfig =
taskGenerator.generateTasks(List.of(tableConfig));
Review Comment:
This changes the task generation logic to processing one table at a time,
which means we cannot apply constraints across tables (e.g. limit total number
of tasks for a given task type). One way to solve this is to add an API `void
generateTasks(TableConfig tableConfig, List taskConfigs)`,
where we also pass in the already generated task configs. When new task config
is generated, we directly append it to this list.
##
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java:
##
@@ -667,13 +667,12 @@ public Map getCronSchedulerJobDetails(
@Produces(MediaType.APPLICATION_JSON)
@Authenticate(AccessType.UPDATE)
@ApiOperation("Schedule tasks and return a map from task type to task name
scheduled")
- public Map scheduleTasks(@ApiParam(value = "Task type")
@QueryParam("taskType") String taskType,
+ public Map> scheduleTasks(@ApiParam(value = "Task
type") @QueryParam("taskType") String taskType,
Review Comment:
This is backward incompatible. I guess we can return a comma separated
string instead so that the response is still compatible
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java:
##
@@ -534,66 +535,88 @@ private synchronized Map
scheduleTasks(List tableNamesWi
* Returns the task name, or {@code null} if no task is scheduled.
*/
@Nullable
- private String scheduleTask(PinotTaskGenerator taskGenerator,
List enabledTableConfigs,
+ private List scheduleTask(PinotTaskGenerator taskGenerator,
List enabledTableConfigs,
boolean isLeader) {
LOGGER.info("Trying to schedule task type: {}, isLeader: {}",
taskGenerator.getTaskType(), isLeader);
-List pinotTaskConfigs;
-try {
- /* TODO taskGenerator may skip generating tasks for some of the tables
being passed to it.
-In that case, we should not be storing success timestamps for those
table. Same with exceptions that should
-only be associated with the table for which it was raised and not
every eligible table. We can have the
-generateTasks() return a list of TaskGeneratorMostRecentRunInfo for
each table
- */
- pinotTaskConfigs = taskGenerator.generateTasks(enabledTableConfigs);
- long successRunTimestamp = System.currentTimeMillis();
- for (TableConfig tableConfig : enabledTableConfigs) {
-
_taskManagerStatusCache.saveTaskGeneratorInfo(tableConfig.getTableName(),
taskGenerator.getTaskType(),
+Map> pinotMinionInstanceToTaskConfigs = new
HashMap<>();
+String taskType = taskGenerator.getTaskType();
+for (TableConfig tableConfig : enabledTableConfigs) {
+ String tableName = tableConfig.getTableName();
+ try {
+String minionInstanceTag =
taskGenerator.getMinionInstanceTag(tableConfig);
+List pinotTaskConfig =
taskGenerator.generateTasks(List.of(tableConfig));
+List presentTaskConfig =
pinotMinionInstanceToTaskConfigs.get(minionInstanceTag);
+if (presentTaskC
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Jackie-Jiang commented on issue #12503: URL: https://github.com/apache/pinot/issues/12503#issuecomment-1972254102 Is the problem resolved automatically? We'll need more context in order to debug it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Consistency in API response for live broker [pinot]
Jackie-Jiang commented on code in PR #12201:
URL: https://github.com/apache/pinot/pull/12201#discussion_r1508366878
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##
@@ -3921,7 +3921,16 @@ public TableStats getTableStats(String
tableNameWithType) {
* Returns map of tableName to list of live brokers
* @return Map of tableName to list of ONLINE brokers serving the table
*/
- public Map> getTableToLiveBrokersMapping() {
+ public Map> getTableToLiveBrokersMapping() throws
TableNotFoundException {
+return getTableToLiveBrokersMapping(null);
+ }
+
+ /**
+ * Returns map of arg tableName to list of live brokers
+ * @return Map of arg tableName to list of ONLINE brokers serving the table
+ */
+ public Map>
getTableToLiveBrokersMapping(@Nullable Object nullableTableName)
Review Comment:
```suggestion
public Map>
getTableToLiveBrokersMapping(@Nullable String tableName)
```
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##
@@ -3933,6 +3942,29 @@ public Map>
getTableToLiveBrokersMapping() {
Map> result = new HashMap<>();
ZNRecord znRecord = ev.getRecord();
+
+if (nullableTableName != null && !nullableTableName.toString().isEmpty()) {
Review Comment:
```suggestion
if (StringUtils.isEmpty(tableName)) {
```
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##
@@ -3933,6 +3942,29 @@ public Map>
getTableToLiveBrokersMapping() {
Map> result = new HashMap<>();
ZNRecord znRecord = ev.getRecord();
+
+if (nullableTableName != null && !nullableTableName.toString().isEmpty()) {
+ List tableNameWithType =
getExistingTableNamesWithType(nullableTableName.toString(), null);
+ if (tableNameWithType.isEmpty()) {
+throw new ControllerApplicationException(LOGGER,
String.format("Table=%s not found", nullableTableName),
+Response.Status.NOT_FOUND);
+ }
+ tableNameWithType.forEach(tableName -> {
+Map brokersToState = znRecord.getMapField(tableName);
+List hosts = new ArrayList<>();
+for (Map.Entry brokerEntry :
brokersToState.entrySet()) {
+ if ("ONLINE".equalsIgnoreCase(brokerEntry.getValue())
+ && instanceConfigMap.containsKey(brokerEntry.getKey())) {
+InstanceConfig instanceConfig =
instanceConfigMap.get(brokerEntry.getKey());
+hosts.add(new InstanceInfo(instanceConfig.getInstanceName(),
instanceConfig.getHostName(),
+Integer.parseInt(instanceConfig.getPort(;
+ }
+}
Review Comment:
Consider extracting a common method
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##
@@ -3921,7 +3921,16 @@ public TableStats getTableStats(String
tableNameWithType) {
* Returns map of tableName to list of live brokers
* @return Map of tableName to list of ONLINE brokers serving the table
*/
- public Map> getTableToLiveBrokersMapping() {
+ public Map> getTableToLiveBrokersMapping() throws
TableNotFoundException {
Review Comment:
(minor) reformat this change
##
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableInstances.java:
##
@@ -165,9 +166,11 @@ public List getLiveBrokersForTable(
@ApiResponses(value = {
@ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 500,
message = "Internal server error")
})
- public Map> getLiveBrokers() {
+ public Map> getLiveBrokers(
+ @ApiParam(value = "Table name (with or without type)", required =
false) @DefaultValue("")
Review Comment:
Are we going to ask for liver brokers on multiple tables (e.g. for JOIN
queries)? If so, suggest taking `tableName` as a list. Search for
`allowMultiple = true` to find examples of using list as api param
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] adding support of mega_bytes when configuring broker response size [pinot]
Jackie-Jiang commented on code in PR #12510:
URL: https://github.com/apache/pinot/pull/12510#discussion_r1508376156
##
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##
@@ -1773,17 +1774,23 @@ private void setMaxServerResponseSizeBytes(int
numServers, Map q
}
// BrokerConfig
-Long maxServerResponseSizeBrokerConfig =
-_config.getProperty(Broker.CONFIG_OF_MAX_SERVER_RESPONSE_SIZE_BYTES,
Long.class);
-if (maxServerResponseSizeBrokerConfig != null) {
- queryOptions.put(QueryOptionKey.MAX_SERVER_RESPONSE_SIZE_BYTES,
Long.toString(maxServerResponseSizeBrokerConfig));
+String strResponseSize =
_config.getProperty(Broker.CONFIG_OF_MAX_SERVER_RESPONSE_SIZE_BYTES);
+if (strResponseSize != null) {
+ Long maxServerResponseSizeBrokerConfig =
DataSizeUtils.toBytes(strResponseSize);
Review Comment:
This method returns primitive `long`, no need to do another null check
##
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##
@@ -329,12 +329,14 @@ public static class Broker {
// Broker config indicating the maximum serialized response size across
all servers for a query. This value is
// equally divided across all servers processing the query.
+// The value can be in human readable format (e.g. '150K', '150KB',
'0.15MB') or in bytes (e.g. '15').
public static final String CONFIG_OF_MAX_QUERY_RESPONSE_SIZE_BYTES =
"pinot.broker.max.query.response.size.bytes";
Review Comment:
Ideally we want to remove `.bytes` suffix here. Since this feature is not
released yet (added after `1.0`), I guess we can change it? cc @vvivekiyer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Added /uptime and /start-time for all components. Addressed comments and fixed tests from pull request 12389. [pinot]
suyashpatel98 commented on PR #12512: URL: https://github.com/apache/pinot/pull/12512#issuecomment-1972399286 > Do we want uptime in seconds, or in a more human readable format? @Jackie-Jiang Xiang in the old PR [here](https://github.com/apache/pinot/pull/12389) was inclined towards returning just a number. I am okay either way -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] enable Netty native transports in ClusterTest [pinot]
sullis opened a new pull request, #12527: URL: https://github.com/apache/pinot/pull/12527 Motiviation: by default NettyConfig does not use native transports. Modifications: - update Netty properties in ClusterTest.java Result: This change enables Netty native transports for all integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] maven: no transfer progress [pinot]
sullis opened a new pull request, #12528: URL: https://github.com/apache/pinot/pull/12528 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] enable Netty native transports in ClusterTest [pinot]
codecov-commenter commented on PR #12527: URL: https://github.com/apache/pinot/pull/12527#issuecomment-1972467793 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12527?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`ed3030b`)](https://app.codecov.io/gh/apache/pinot/pull/12527?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12527 +/- ## = - Coverage 61.75%0.00% -61.75% + Complexity 2076 -201 = Files 2436 2374 -62 Lines133233 129802 -3431 Branches 2063620134 -502 = - Hits 822746-82268 - Misses44911 129796+84885 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-61.71%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-61.75%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12527?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_t
Re: [PR] maven no transfer progress [pinot]
codecov-commenter commented on PR #12528: URL: https://github.com/apache/pinot/pull/12528#issuecomment-1972468907 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12528?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`df3f9a1`)](https://app.codecov.io/gh/apache/pinot/pull/12528?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12528 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2374 -62 Lines133233 129800 -3433 Branches 2063620134 -502 = - Hits 822740-82274 - Misses44911 129800+84889 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12528/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12528?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=r
[PR] Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [pinot]
swaminathanmanish opened a new pull request, #12529: URL: https://github.com/apache/pinot/pull/12529 **Whats in the PR:** Allow custom records to be inited/closed i.e managed within SegmentProcessorFramework **Why its needed:** We have custom readers (eg: where record needs to be enriched based on file path) that are passed to the SegmentProcessorFramework. We want the ability to init & close these readers after use, within the framework to manage the lifecycle of the reader. Otherwise we would be holding up memory for these readers through out segment creation, which can result in unnecessary use of memory. Also these readers are used one at a time, so they can be inited/closed when needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [pinot]
swaminathanmanish commented on PR #12529: URL: https://github.com/apache/pinot/pull/12529#issuecomment-1972548026 @snleee - Please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] [partial-upsert] Bug in Handling Comparison Column Ties When Replacing Segment [pinot]
ankitsultana closed issue #12398: [partial-upsert] Bug in Handling Comparison Column Ties When Replacing Segment URL: https://github.com/apache/pinot/issues/12398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] [partial-upsert] Bug in Handling Comparison Column Ties When Replacing Segment [pinot]
ankitsultana commented on issue #12398: URL: https://github.com/apache/pinot/issues/12398#issuecomment-1972554920 Closing this since this was resolved by #12395. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [pinot]
codecov-commenter commented on PR #12529: URL: https://github.com/apache/pinot/pull/12529#issuecomment-1972579030 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12529?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 34.90%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`d5c0829`)](https://app.codecov.io/gh/apache/pinot/pull/12529?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12529 +/- ## = - Coverage 61.75% 34.90% -26.85% = Files 2436 2374 -62 Lines133233 129802 -3431 Branches 2063620135 -501 = - Hits 8227445309-36965 - Misses4491181214+36303 + Partials 6048 3279 -2769 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.90% <100.00%> (-26.72%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.89% <100.00%> (-26.86%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.70% <100.00%> (+18.97%)` | :arrow_up: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.90% <100.00%> (-26.85%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-15.00%)` | :arrow_down: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-0.15%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12529/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12529?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=a
Re: [PR] Allow passing database context through `database` http header [pinot]
shounakmk219 commented on PR #12417: URL: https://github.com/apache/pinot/pull/12417#issuecomment-1972579675 @Jackie-Jiang I have reverted the v2 APIs and the request filter approach to achieve table translation. We can talk around what's the best way to handle the translation separately in the design [doc](https://docs.google.com/document/d/1l1dco4lRhB28y-yKMVKlOMZ0sdFKObjJfhAjG6cmktU/edit?usp=sharing) . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Prepare for next development iteration [pinot]
vvivekiyer opened a new pull request, #12530: URL: https://github.com/apache/pinot/pull/12530 Updating version for next release development iteration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] reformat ClusterTest.java [pinot]
sullis opened a new pull request, #12531: URL: https://github.com/apache/pinot/pull/12531 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) annotated tag release-1.1.0-rc0 deleted (was 6ad905acf2)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to annotated tag release-1.1.0-rc0 in repository https://gitbox.apache.org/repos/asf/pinot.git *** WARNING: tag release-1.1.0-rc0 was deleted! *** tag was 6ad905acf2 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch release-1.1.0-rc updated (e75703bda3 -> ce94496e2e)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git discard e75703bda3 [maven-release-plugin] prepare for next development iteration discard 56fc5a6dfc [maven-release-plugin] prepare release release-1.1.0-rc0 This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/release-1.1.0-rc (ce94496e2e) \ O -- O -- O (e75703bda3) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: contrib/pinot-fmpp-maven-plugin/pom.xml| 6 ++- pinot-broker/pom.xml | 5 ++- pinot-clients/pinot-java-client/pom.xml| 5 ++- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 ++- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 ++- pinot-connectors/pinot-flink-connector/pom.xml | 6 ++- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-common/pom.xml| 5 ++- pinot-connectors/pom.xml | 6 ++- pinot-controller/pom.xml | 5 ++- pinot-core/pom.xml | 5 ++- pinot-distribution/pom.xml | 9 +++-- pinot-integration-test-base/pom.xml| 5 ++- pinot-integration-tests/pom.xml| 5 ++- pinot-minion/pom.xml | 5 ++- pinot-perf/pom.xml | 5 ++- .../pinot-batch-ingestion-common/pom.xml | 6 ++- .../pinot-batch-ingestion-hadoop/pom.xml | 6 ++- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 ++- .../pinot-batch-ingestion-spark-3/pom.xml | 6 ++- .../pinot-batch-ingestion-standalone/pom.xml | 6 ++- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 ++- .../pinot-environment/pinot-azure/pom.xml | 6 ++- pinot-plugins/pinot-environment/pom.xml| 6 ++- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +++-- pinot-plugins/pinot-file-system/pom.xml| 6 ++- .../pinot-input-format/pinot-avro-base/pom.xml | 5 ++- .../pinot-input-format/pinot-avro/pom.xml | 5 ++- .../pinot-input-format/pinot-clp-log/pom.xml | 5 ++- .../pinot-confluent-avro/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 ++- .../pinot-input-format/pinot-json/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 ++- .../pinot-input-format/pinot-parquet/pom.xml | 5 ++- .../pinot-input-format/pinot-protobuf/pom.xml | 5 ++- .../pinot-input-format/pinot-thrift/pom.xml| 5 ++- pinot-plugins/pinot-input-format/pom.xml | 6 ++- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pom.xml| 6 ++- .../pinot-minion-builtin-tasks/pom.xml | 5 ++- pinot-plugins/pinot-minion-tasks/pom.xml | 6 ++- .../pinot-segment-uploader-default/pom.xml | 6 ++- pinot-plugins/pinot-segment-uploader/pom.xml | 6 ++- .../pinot-segment-writer-file-based/pom.xml| 6 ++- pinot-plugins/pinot-segment-writer/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++- .../pinot-kafka-base/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 -- .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 ++- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++- pinot-plugins/pom.xml | 11 -- pinot-query-planner/pom.xml| 5 ++- pinot-query-runtime/pom.xml| 5 ++- pinot-segment-local/pom.xml| 5 ++- pinot-segment-spi/pom.xml | 5 ++- pinot-server/pom.xml | 5 ++- pinot-spi/pom.xml | 5 ++- pinot-tools/pom.xml| 5 ++- pom.xml| 11 +++--- 66 files changed, 262 insertions(+), 162 deletions(-) - To unsubscribe, e-mail
Re: [PR] Prepare for next development iteration [pinot]
codecov-commenter commented on PR #12530: URL: https://github.com/apache/pinot/pull/12530#issuecomment-1972656712 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12530?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`4ce7e4e`)](https://app.codecov.io/gh/apache/pinot/pull/12530?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12530 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2374 -62 Lines133233 129800 -3433 Branches 2063620134 -502 = - Hits 822740-82274 - Misses44911 129800+84889 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12530/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12530?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=r
Re: [PR] Add the support to extract json array elements from json index for the transform function jsonExtractIndex [pinot]
itschrispeck commented on code in PR #12466:
URL: https://github.com/apache/pinot/pull/12466#discussion_r1508495654
##
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractIndexTransformFunction.java:
##
@@ -90,9 +90,13 @@ public void init(List arguments,
Map c
}
String resultsType = ((LiteralTransformFunction)
thirdArgument).getStringLiteral().toUpperCase();
boolean isSingleValue = !resultsType.endsWith("_ARRAY");
+// TODO: will support mv; the underlying
jsonIndexReader.getMatchingDocsMap already works for the json path [*]
if (!isSingleValue) {
throw new IllegalArgumentException("jsonExtractIndex only supports
single value type");
}
+if (isSingleValue && inputJsonPath.contains("[*]")) {
+ throw new IllegalArgumentException("json path should not use [*] if the
return type is single value");
Review Comment:
nit: something like `[*] syntax is unsupported as json_extract_index does
not support returning array types` is clearer to a user
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java:
##
@@ -370,6 +312,90 @@ private int[] getDictIdRangeForKey(String key) {
return new int[]{minDictId, maxDictId};
}
+ // If key doesn't contain the array index, return
+ // Elif the key, i.e. the json path provided by user doesn't match any data,
return
+ // Else, return the json path that is generated by replacing array index
with . on the original key
+ // and the associated flattenDocId bitmap
Review Comment:
nit: javadoc formatting
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java:
##
@@ -301,12 +272,22 @@ public Map
getMatchingDocsMap(String key) {
Map matchingDocsMap = new HashMap<>();
_readLock.lock();
try {
+ Pair result = processArrayIndex(key);
+ key = result.getLeft();
+ RoaringBitmap arrayIndexFlattenDocIds = result.getRight();
+ if (arrayIndexFlattenDocIds != null &&
arrayIndexFlattenDocIds.isEmpty()) {
+return matchingDocsMap;
+ }
for (Map.Entry entry :
_postingListMap.entrySet()) {
if (!entry.getKey().startsWith(key +
BaseJsonIndexCreator.KEY_VALUE_SEPARATOR)) {
continue;
}
-MutableRoaringBitmap flattenedDocIds =
entry.getValue().toMutableRoaringBitmap();
-PeekableIntIterator it = flattenedDocIds.getIntIterator();
+RoaringBitmap kvPairFlattenedDocIds = entry.getValue();
+PeekableIntIterator it = arrayIndexFlattenDocIds == null ?
kvPairFlattenedDocIds.getIntIterator()
+: intersect(arrayIndexFlattenDocIds.clone(),
kvPairFlattenedDocIds).getIntIterator();
Review Comment:
`RoaringBitmap.and(arrayIndexFlattenDocIds,
flattenedDocIds).getIntIterator()`
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java:
##
@@ -342,6 +323,51 @@ public String[] getValuesForKeyAndDocs(int[] docIds,
Map
return values;
}
+ private Pair processArrayIndex(String key) {
Review Comment:
`processArrayIndex` -> `getKeyAndFlattenedDocIds`?
Can you add a comment here about input/output (already done in the immutable
reader)
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java:
##
@@ -309,12 +242,21 @@ private int getDocId(int flattenedDocId) {
@Override
public Map getMatchingDocsMap(String key) {
Map matchingDocsMap = new HashMap<>();
+Pair result = processArrayIndex(key);
+key = result.getLeft();
+MutableRoaringBitmap arrayIndexFlattenDocIds = result.getRight();
+if (arrayIndexFlattenDocIds != null && arrayIndexFlattenDocIds.isEmpty()) {
+ return matchingDocsMap;
+}
int[] dictIds = getDictIdRangeForKey(key);
-
for (int dictId = dictIds[0]; dictId < dictIds[1]; dictId++) {
// get docIds from posting list, convert these to the actual docIds
ImmutableRoaringBitmap flattenedDocIds =
_invertedIndex.getDocIds(dictId);
- PeekableIntIterator it = flattenedDocIds.getIntIterator();
+ PeekableIntIterator it = arrayIndexFlattenDocIds == null ?
flattenedDocIds.getIntIterator()
+ : intersect(arrayIndexFlattenDocIds.clone(), flattenedDocIds);
Review Comment:
`RoaringBitmap.and(arrayIndexFlattenDocIds,
flattenedDocIds).getIntIterator()`
##
pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java:
##
@@ -158,6 +159,11 @@ public void setUp()
+ "\"stringVal\":\"%s\"}", RANDOM.nextInt(), RANDOM.nextLong(),
RANDOM.nextFloat(), RANDOM.nextDouble(),
BigDecimal.valueOf(RANDOM.nextDouble()).multiply(BigDecimal.valueOf(RANDOM.nextInt())),
df.fo
Re: [PR] reformat ClusterTest.java [pinot]
codecov-commenter commented on PR #12531: URL: https://github.com/apache/pinot/pull/12531#issuecomment-1972679095 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12531?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 61.66%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`36352c5`)](https://app.codecov.io/gh/apache/pinot/pull/12531?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 34 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12531 +/- ## - Coverage 61.75% 61.66% -0.10% Complexity 207 207 Files 2436 2450 +14 Lines133233 133550 +317 Branches 2063620692 +56 + Hits 8227482349 +75 - Misses4491145096 +185 - Partials 6048 6105 +57 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.66% <ø> (-0.05%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.65% <ø> (-0.10%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.66% <ø> (-0.10%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.65% <ø> (-0.09%)` | :arrow_down: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.84% <ø> (-0.06%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12531/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.66% <ø> (-0.07%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12531?src=pr&el=continue&utm_medium
(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare release release-1.1.0-rc0
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new c72b56a55c [maven-release-plugin] prepare release release-1.1.0-rc0 c72b56a55c is described below commit c72b56a55c5faf095518ee8799d2f756efb383f2 Author: Vivek Iyer Vaidyanathan AuthorDate: Thu Feb 29 23:46:07 2024 -0800 [maven-release-plugin] prepare release release-1.1.0-rc0 --- contrib/pinot-fmpp-maven-plugin/pom.xml| 6 +-- pinot-broker/pom.xml | 5 +-- pinot-clients/pinot-java-client/pom.xml| 5 +-- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 +-- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 +-- pinot-connectors/pinot-flink-connector/pom.xml | 6 +-- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 +-- pinot-connectors/pinot-spark-common/pom.xml| 5 +-- pinot-connectors/pom.xml | 6 +-- pinot-controller/pom.xml | 5 +-- pinot-core/pom.xml | 5 +-- pinot-distribution/pom.xml | 9 ++--- pinot-integration-test-base/pom.xml| 5 +-- pinot-integration-tests/pom.xml| 5 +-- pinot-minion/pom.xml | 5 +-- pinot-perf/pom.xml | 5 +-- .../pinot-batch-ingestion-common/pom.xml | 6 +-- .../pinot-batch-ingestion-hadoop/pom.xml | 6 +-- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 +-- .../pinot-batch-ingestion-spark-3/pom.xml | 6 +-- .../pinot-batch-ingestion-standalone/pom.xml | 6 +-- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 +-- .../pinot-environment/pinot-azure/pom.xml | 6 +-- pinot-plugins/pinot-environment/pom.xml| 6 +-- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 +-- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 +-- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 ++--- pinot-plugins/pinot-file-system/pom.xml| 6 +-- .../pinot-input-format/pinot-avro-base/pom.xml | 5 +-- .../pinot-input-format/pinot-avro/pom.xml | 5 +-- .../pinot-input-format/pinot-clp-log/pom.xml | 5 +-- .../pinot-confluent-avro/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 +-- .../pinot-input-format/pinot-json/pom.xml | 5 +-- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 +-- .../pinot-input-format/pinot-parquet/pom.xml | 5 +-- .../pinot-input-format/pinot-protobuf/pom.xml | 5 +-- .../pinot-input-format/pinot-thrift/pom.xml| 5 +-- pinot-plugins/pinot-input-format/pom.xml | 6 +-- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 +-- pinot-plugins/pinot-metrics/pom.xml| 6 +-- .../pinot-minion-builtin-tasks/pom.xml | 5 +-- pinot-plugins/pinot-minion-tasks/pom.xml | 6 +-- .../pinot-segment-uploader-default/pom.xml | 6 +-- pinot-plugins/pinot-segment-uploader/pom.xml | 6 +-- .../pinot-segment-writer-file-based/pom.xml| 6 +-- pinot-plugins/pinot-segment-writer/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 +-- .../pinot-kafka-base/pom.xml | 6 +-- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 ++ .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 +-- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 +-- pinot-plugins/pom.xml | 11 ++ pinot-query-planner/pom.xml| 5 +-- pinot-query-runtime/pom.xml| 5 +-- pinot-segment-local/pom.xml| 5 +-- pinot-segment-spi/pom.xml | 5 +-- pinot-server/pom.xml | 5 +-- pinot-spi/pom.xml | 5 +-- pinot-tools/pom.xml| 5 +-- pom.xml| 13 +++ 66 files changed, 163 insertions(+), 263 deletions(-) diff --git a/contrib/pinot-fmpp-maven-plugin/pom.xml b/contrib/pinot-fmpp-maven-plugin/pom.xml index 94df4afc49..94fc21d142 100644 --- a/contrib/pinot-fmpp-maven-plugin/pom.xml +++ b/contrib/pinot-fmpp-maven-plugin/pom.xml @@ -19,14 +19,12 @@ under the License. --> -http://maven.apache.org/POM/4.0.0"; -
(pinot) branch release-1.1.0-rc updated (c72b56a55c -> ce94496e2e)
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git discard c72b56a55c [maven-release-plugin] prepare release release-1.1.0-rc0 This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/release-1.1.0-rc (ce94496e2e) \ O -- O -- O (c72b56a55c) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: contrib/pinot-fmpp-maven-plugin/pom.xml| 6 ++- pinot-broker/pom.xml | 5 ++- pinot-clients/pinot-java-client/pom.xml| 5 ++- pinot-clients/pinot-jdbc-client/pom.xml| 8 ++-- pinot-clients/pom.xml | 6 ++- pinot-common/pom.xml | 43 +++--- pinot-compatibility-verifier/pom.xml | 5 ++- pinot-connectors/pinot-flink-connector/pom.xml | 6 ++- pinot-connectors/pinot-spark-2-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-3-connector/pom.xml | 5 ++- pinot-connectors/pinot-spark-common/pom.xml| 5 ++- pinot-connectors/pom.xml | 6 ++- pinot-controller/pom.xml | 5 ++- pinot-core/pom.xml | 5 ++- pinot-distribution/pom.xml | 9 +++-- pinot-integration-test-base/pom.xml| 5 ++- pinot-integration-tests/pom.xml| 5 ++- pinot-minion/pom.xml | 5 ++- pinot-perf/pom.xml | 5 ++- .../pinot-batch-ingestion-common/pom.xml | 6 ++- .../pinot-batch-ingestion-hadoop/pom.xml | 6 ++- .../pinot-batch-ingestion-spark-2.4/pom.xml| 6 ++- .../pinot-batch-ingestion-spark-3/pom.xml | 6 ++- .../pinot-batch-ingestion-standalone/pom.xml | 6 ++- pinot-plugins/pinot-batch-ingestion/pom.xml| 6 ++- .../pinot-environment/pinot-azure/pom.xml | 6 ++- pinot-plugins/pinot-environment/pom.xml| 6 ++- pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 ++- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +++-- pinot-plugins/pinot-file-system/pom.xml| 6 ++- .../pinot-input-format/pinot-avro-base/pom.xml | 5 ++- .../pinot-input-format/pinot-avro/pom.xml | 5 ++- .../pinot-input-format/pinot-clp-log/pom.xml | 5 ++- .../pinot-confluent-avro/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 ++- .../pinot-input-format/pinot-json/pom.xml | 5 ++- pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 ++- .../pinot-input-format/pinot-parquet/pom.xml | 5 ++- .../pinot-input-format/pinot-protobuf/pom.xml | 5 ++- .../pinot-input-format/pinot-thrift/pom.xml| 5 ++- pinot-plugins/pinot-input-format/pom.xml | 6 ++- .../pinot-metrics/pinot-dropwizard/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 ++- pinot-plugins/pinot-metrics/pom.xml| 6 ++- .../pinot-minion-builtin-tasks/pom.xml | 5 ++- pinot-plugins/pinot-minion-tasks/pom.xml | 6 ++- .../pinot-segment-uploader-default/pom.xml | 6 ++- pinot-plugins/pinot-segment-uploader/pom.xml | 6 ++- .../pinot-segment-writer-file-based/pom.xml| 6 ++- pinot-plugins/pinot-segment-writer/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++- .../pinot-kafka-base/pom.xml | 6 ++- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 -- .../pinot-stream-ingestion/pinot-pulsar/pom.xml| 6 ++- pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++- pinot-plugins/pom.xml | 11 -- pinot-query-planner/pom.xml| 5 ++- pinot-query-runtime/pom.xml| 5 ++- pinot-segment-local/pom.xml| 5 ++- pinot-segment-spi/pom.xml | 5 ++- pinot-server/pom.xml | 5 ++- pinot-spi/pom.xml | 5 ++- pinot-tools/pom.xml| 5 ++- pom.xml| 13 --- 66 files changed, 263 insertions(+), 163 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h.
