This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/maven/calcite.version-1.41.0
in repository https://gitbox.apache.org/repos/asf/pinot.git
omit 6f1a86c1f80 Bump calcite.version from 1.40.0 to 1.41.0
add ff41d7dc9e3 Bump com.jayway.jsonpath:json-path from 2.9.0 to 2.10.0
(#17131)
add 62ec58c8233 Bump software.amazon.awssdk:bom from 2.37.2 to 2.37.3
(#17132)
add 490d392ae06 Bump org.webjars:swagger-ui from 5.30.0 to 5.30.1 (#17129)
add 5497d4aa44c IP Address Functions Implemented (#17127)
add eaaf1ac519a Adding a Test for String to MAP transformation using
jsonStringToMap (#17107)
add fa7e32bbb5d Set the upload URI and pinot controller URI to the lead
controller for minion tasks when possible (#17125)
add ee47213d30b fix authorization issue caused by duplicate auth header
(#17134)
add 13da6e3935d column major segment build for columnar datasource (#16727)
add 7c8f54cfce7 v1 queries populate tablesQueried for empty responses
(#17133)
add fe46c3038ab [Feature] Develop LaunchBackfillIngestionJob to support
complete backfill (#16890)
add 273e0ce0766 Safe Format Change in Apache Pinot: From Colon to Pipe
Notation (#17079)
add 80ad4f7ecb0 sort servers by their assigned segments before the greedy
selection (#17138)
add c63e3ccdf5d [Feature] Add segment reload failure tracking with
in-memory status cache (#17099)
add 8ae486ad17f Bump software.amazon.awssdk:bom from 2.37.3 to 2.37.4
(#17135)
add 189bbdfcfff Bump com.gradle:develocity-maven-extension from 2.2 to
2.2.1 (#17136)
add e47214ef4c6 feat(array): add arrayPushFront/arrayPushBack scalar
functions for all types; add unit and integration tests; update UDF snapshot
(#17140)
add ddeecef5db5 Partial Upserts - Add a metric to detect when server is
ahead of zk committed offset (#17139)
add 84505f8070b funnel: fix deserialize offset for extra fields; add
round-trip unit test\n\nFixes IllegalArgumentException when Jackson is given
invalid offset/len. The JSON portion begins after FunnelStepEvent
(SIZE_IN_BYTES). (#17141)
add 3ba94a35664 Bump com.azure:azure-sdk-bom from 1.3.0 to 1.3.2 (#17151)
add af33b1002dd Bump org.checkerframework:checker-qual from 3.51.1 to
3.52.0 (#17149)
add 19cd973e566 Bump
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 (#17147)
add d6778b5432e Bump software.amazon.awssdk:bom from 2.37.4 to 2.37.5
(#17148)
add 894efd3785f Respect column aliases in empty query responses (#17143)
add 9c709dc783b Partial Upsert - Disable Reload on consuming segment and
Force Commit on Partial Upsert Tables (#17144)
add bd50c0fb788 Support MAP type in derived column creation during segment
reload (#17113)
add 4c8ac191d29 Fix Minion task metrics collection gaps (#17128)
add ec4bd2d8705 Fix missing serializeIntermediateResult and
deserializeIntermediateResult methods for SumArrayLongAggregationFunction and
SumArrayDoubleAggregationFunction (#17158)
add 772d551b77c Bump commons-codec:commons-codec from 1.19.0 to 1.20.0
(#17157)
add ccc41ea8e54 Make dedup table use StrictRealtimeSegmentAssignment with
support of multi tiers (#17154)
add cf6a9204f8b feat(common): add ARRAYHASANY scalar UDF for arrays
(int/long/float/double/string)\n\n- New PinotScalarFunction:
ArrayHasAnyScalarFunction (ARRAYHASANY)\n- Type-dispatched implementation via
FunctionInfo map\n- Integration tests: ArrayTest adds literals, column filters,
and same-column self-overlap checks\n- Minor planner wiring update in
PinotOperatorTable (#17156)
add ebe2f2018b9 Bump software.amazon.awssdk:bom from 2.37.5 to 2.38.1
(#17164)
add 575d89f0c47 feat(agg): add ListAggMv (multi-value list aggregation)
and distinct variant; wire enum/factory; add unit and query tests.\n\nAlso
enables SQL syntax: listAggMv(col, 'sep'[, true]). (#17155)
add 1a42f603a6f Bump com.nimbusds:nimbus-jose-jwt from 10.5 to 10.6
(#17165)
add 3d62284db31 [logical] Created QuickStart for logical tables (#17166)
add 8de6fa9b9a2 Enhance ArrayAgg to support aggregation on multi-value
columns (#17153)
add b8470586540 Bump net.openhft:chronicle-bom from 2.27ea86 to 2.27ea87
(#17173)
add 46ba301ef5a Bump software.amazon.awssdk:bom from 2.38.1 to 2.38.2
(#17174)
add 368f148ecf9 Bump com.google.errorprone:error_prone_annotations from
2.43.0 to 2.44.0 (#17172)
add 8403dcbff6e Bump org.webjars:swagger-ui from 5.30.1 to 5.30.2 (#17176)
add ba014509414 Adding a new property minNumSegmentsPerTask for
UpsertCompactMerge task (#17104)
add a5a9cdb253d Add test demonstrating MultiStageReplicaGroupSelector bug
when instance partitions are out of sync with ideal state (#17180)
add 7d38ce4844a Add interfaces to get partition columns and partition
values (#17171)
add 5b4d48b914b Misc Fixes and Addition of End-to-End Ingestion Delay
Metric (#17163)
add 93a15a576d4 Bump commons-io:commons-io from 2.20.0 to 2.21.0 (#17175)
add 31d6a382a6c Bump com.gradle:common-custom-user-data-maven-extension
(#17187)
add 1f982956bd6 Bump software.amazon.awssdk:bom from 2.38.2 to 2.38.3
(#17188)
add fb0d8d69d9c Adding Raw bitmap inverted index creator and reader
(#17060)
add c6d4f1c5aab Bump software.amazon.awssdk:bom from 2.38.3 to 2.38.4
(#17196)
add 547115ef925 Add consuming segment check for retention check (#17117)
add b32dfedf93d [logical] Adding logical tables panel in Query Console
Controller UI (#17199)
add 45bf79241bf [bugfix] Returning error block for OpChain initialization
errors (#17161)
add 5172ada0e4f segment-local: NoDict hybrid cardinality (exact ≤ 2,048 →
HLL++ +10% cap) (#17186)
add 18d37ce6955 Bump net.openhft:chronicle-bom from 2.27ea87 to 2.27ea88
(#17201)
add 27f607a6db0 Bump software.amazon.awssdk:bom from 2.38.4 to 2.38.5
(#17202)
add e0f91462545 Bump commons-cli:commons-cli from 1.10.0 to 1.11.0 (#17203)
add 3224ce6b796 [server] Add dynamic cluster config support to
ServerReloadJobStatusCache (#17159)
add 43f92c4f043 Add error message for leaf stage errors (#17207)
add c0c518c57b0 [bugfix] Fixing flaky SSBQueryIntegrationTest (#17205)
add 844d4ab8f9f [timeseries] Introducing M3ql JavaCC parser to replace
custom tokenizer implementation (#17192)
add bf923702148 Bump software.amazon.awssdk:bom from 2.38.5 to 2.38.6
(#17211)
add 2c27fb33f11 Bump net.openhft:chronicle-bom from 2.27ea88 to 2.27ea90
(#17212)
add 47fa39c26cd Bump org.codehaus.mojo:javacc-maven-plugin from 3.0.1 to
3.1.1 (#17210)
add a790681713f Bump js-yaml from 4.1.0 to 4.1.1 in
/pinot-controller/src/main/resources (#17214)
add 62b20d9decc Fix collision in trace (#17206)
add 80062305e3a Fix Segment Partition Tracking for Multi-Topic Ingestion
(#17217)
add a23c610309b Fixes all race conditions from stream consumer (#17089)
add 852305a4cd0 Make Scalar Functions Advertise Determinism and Support
RAND function with Runtime-Only and Seeded Mode (#17208)
add 560858552e4 Add APIs for primitive data types for stats collectors,
index creators, column readers (#17183)
add f747903a6e2 Bump org.apache.pulsar:pulsar-bom from 4.0.7 to 4.0.8
(#17232)
add 8ab8581cf2e Bump org.asynchttpclient:async-http-client from 3.0.3 to
3.0.4 (#17231)
add 32c670814a9 Bump com.gradle:common-custom-user-data-maven-extension
(#17230)
add a468f938a24 Bump com.squareup.okio:okio-bom from 3.16.2 to 3.16.4
(#17229)
add 3451acad535 Bump software.amazon.awssdk:bom from 2.38.6 to 2.38.8
(#17228)
add 096d68c06aa Bump commons-validator:commons-validator from 1.10.0 to
1.10.1 (#17227)
add 72dc56f16da Bump org.apache.commons:commons-lang3 from 3.19.0 to
3.20.0 (#17226)
add 7a77329003b Bump io.grpc:grpc-bom from 1.76.0 to 1.77.0 (#17225)
add af01b931c9a Bump net.alchim31.maven:scala-maven-plugin from 4.9.6 to
4.9.7 (#17223)
add 0c954d4ab0a Bump immutables.version from 2.11.6 to 2.11.7 (#17222)
add c959711235c Bump calcite.version from 1.40.0 to 1.41.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6f1a86c1f80)
\
N -- N -- N refs/heads/dependabot/maven/calcite.version-1.41.0
(c959711235c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.mvn/extensions.xml | 4 +-
.../broker/ZkBasicAuthAccessControlFactory.java | 6 +-
.../BaseSingleStageBrokerRequestHandler.java | 1 +
.../broker/requesthandler/EmptyResponseUtils.java | 27 +-
.../requesthandler/EmptyResponseUtilsTest.java | 55 ++
.../MultiStageReplicaGroupSelectorTest.java | 34 +
.../pinot/client/admin/PinotAdminClient.java | 13 +
.../pinot/client/admin/PinotAdminTransport.java | 10 +-
.../pinot/client/admin/PinotSegmentApiClient.java | 76 ++
.../apache/pinot/common/function/FunctionInfo.java | 14 +-
.../pinot/common/function/FunctionRegistry.java | 14 +-
.../pinot/common/function/FunctionUtils.java | 3 +
.../function/scalar/ArithmeticFunctions.java | 31 +
.../common/function/scalar/ArrayFunctions.java | 50 ++
.../common/function/scalar/IpAddressFunctions.java | 52 ++
.../scalar/array/ArraysOverlapScalarFunction.java | 181 ++++
.../common/messages/SegmentReloadMessage.java | 16 +
.../apache/pinot/common/metrics/ServerGauge.java | 1 +
.../apache/pinot/common/metrics/ServerMeter.java | 4 +
.../apache/pinot/common/utils/PinotDataType.java | 13 +
.../rewriter/CompileTimeFunctionsInvoker.java | 2 +-
.../function/scalar/IpAddressFunctionsTest.java | 287 +++++++
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 11 +
.../apache/pinot/controller/ControllerConf.java | 4 +
.../access/ZkBasicAuthAccessControlFactory.java | 6 +-
.../api/dto/PinotTableReloadStatusResponse.java | 10 +
...va => BaseStrictRealtimeSegmentAssignment.java} | 62 +-
.../MultiTierStrictRealtimeSegmentAssignment.java | 86 ++
.../segment/RealtimeSegmentAssignment.java | 7 +-
.../segment/SegmentAssignmentFactory.java | 6 +-
.../assignment/segment/SegmentAssignmentUtils.java | 10 +-
.../SingleTierStrictRealtimeSegmentAssignment.java | 77 ++
.../helix/core/minion/ClusterInfoAccessor.java | 98 +++
.../core/minion/PinotHelixTaskResourceManager.java | 57 +-
.../helix/core/minion/TaskMetricsEmitter.java | 90 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 22 +-
.../helix/core/rebalance/TableRebalancer.java | 6 +-
.../retention/strategy/TimeRetentionStrategy.java | 7 +
.../services/PinotTableReloadStatusReporter.java | 24 +-
.../app/components/Query/QuerySideBar.tsx | 14 +-
.../src/main/resources/app/pages/Query.tsx | 7 +
.../src/main/resources/app/requests/index.ts | 3 +
.../main/resources/app/utils/PinotMethodUtils.ts | 20 +-
.../src/main/resources/package-lock.json | 12 +-
.../segment/SegmentAssignmentUtilsTest.java | 70 +-
.../StrictRealtimeSegmentAssignmentTest.java | 225 ++++-
.../helix/core/minion/TaskMetricsEmitterTest.java | 149 ++++
.../minion/generator/TaskGeneratorUtilsTest.java | 2 +
.../PinotLLCRealtimeSegmentManagerTest.java | 106 ++-
.../helix/core/retention/RetentionManagerTest.java | 1 +
.../strategy/TimeRetentionStrategyTest.java | 71 ++
.../apache/pinot/core/common/MinionConstants.java | 12 +
.../core/data/manager/BaseTableDataManager.java | 39 +-
.../core/data/manager/InstanceDataManager.java | 9 +-
.../provider/DefaultTableDataManagerProvider.java | 13 +-
.../manager/provider/TableDataManagerProvider.java | 18 +-
.../manager/realtime/IngestionDelayTracker.java | 114 ++-
.../realtime/RealtimeSegmentDataManager.java | 82 +-
.../realtime/RealtimeSegmentMetadataUtils.java | 65 ++
.../manager/realtime/RealtimeTableDataManager.java | 60 +-
.../core/operator/filter/FilterOperatorUtils.java | 7 +-
.../RawValueInvertedIndexFilterOperator.java | 193 +++++
.../function/AggregationFunctionFactory.java | 9 +-
.../array/ArrayAggDistinctDoubleFunction.java | 25 +-
.../array/ArrayAggDistinctFloatFunction.java | 25 +-
.../array/ArrayAggDistinctIntFunction.java | 25 +-
.../array/ArrayAggDistinctLongFunction.java | 25 +-
.../array/ArrayAggDistinctStringFunction.java | 21 +-
.../function/array/ArrayAggDoubleFunction.java | 25 +-
.../function/array/ArrayAggFloatFunction.java | 25 +-
.../function/array/ArrayAggIntFunction.java | 25 +-
.../function/array/ArrayAggLongFunction.java | 25 +-
.../function/array/ArrayAggStringFunction.java | 21 +-
.../function/array/BaseArrayAggDoubleFunction.java | 58 +-
.../function/array/BaseArrayAggFloatFunction.java | 58 +-
.../function/array/BaseArrayAggIntFunction.java | 58 +-
.../function/array/BaseArrayAggLongFunction.java | 58 +-
.../function/array/BaseArrayAggStringFunction.java | 58 +-
.../function/array/ListAggFunction.java | 76 +-
.../array/SumArrayDoubleAggregationFunction.java | 13 +
.../array/SumArrayLongAggregationFunction.java | 13 +
.../funnel/FunnelStepEventWithExtraFields.java | 3 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 12 +-
.../partitioner/ColumnValuePartitioner.java | 14 +
.../processing/partitioner/NoOpPartitioner.java | 10 +
.../processing/partitioner/Partitioner.java | 21 +
.../partitioner/RoundRobinPartitioner.java | 14 +
.../partitioner/TableConfigPartitioner.java | 16 +-
.../partitioner/TransformFunctionPartitioner.java | 11 +
.../processing/timehandler/EpochTimeHandler.java | 45 +
.../processing/timehandler/NoOpTimeHandler.java | 15 +
.../processing/timehandler/TimeHandler.java | 26 +
.../pinot/core/util/trace/BuiltInTracer.java | 4 +-
.../apache/pinot/core/util/trace/TraceContext.java | 45 +-
.../server/access/ZkBasicAuthAccessFactory.java | 6 +-
.../core/data/function/ArrayFunctionsTest.java | 16 +
.../BaseTableDataManagerAcquireSegmentTest.java | 3 +-
...ableDataManagerEnqueueSegmentToReplaceTest.java | 15 +-
.../data/manager/BaseTableDataManagerTest.java | 5 +-
.../offline/DimensionTableDataManagerTest.java | 5 +-
.../realtime/IngestionDelayTrackerTest.java | 67 +-
.../realtime/RealtimeSegmentDataManagerTest.java | 7 +-
.../realtime/RealtimeTableDataManagerTest.java | 66 ++
.../ScalarTransformFunctionWrapperTest.java | 52 ++
.../function/ArrayAggMvFunctionTest.java | 299 +++++++
.../aggregation/function/ListAggFunctionTest.java | 153 ++++
.../FunnelStepEventWithExtraFieldsTest.java} | 32 +-
.../executor/QueryExecutorExceptionsTest.java | 4 +-
.../core/query/executor/QueryExecutorTest.java | 4 +-
.../pinot/core/util/trace/TraceContextTest.java | 57 +-
...QueriesTest.java => ArrayAggMvQueriesTest.java} | 138 ++-
.../pinot/queries/ExplainPlanQueriesTest.java | 4 +-
.../apache/pinot/queries/ListAggQueriesTest.java | 193 +++++
.../queries/SegmentWithNullValueVectorTest.java | 4 +-
.../src/test/resources/TableIndexingTest.csv | 46 +-
.../tests/BaseClusterIntegrationTest.java | 13 +-
.../tests/MultiStageEngineIntegrationTest.java | 20 +
.../integration/tests/SSBQueryIntegrationTest.java | 1 +
.../pinot/integration/tests/custom/ArrayTest.java | 221 ++++-
.../FailureInjectingTableDataManagerProvider.java | 33 +-
.../resources/udf-test-results/all-functions.yaml | 40 +
.../perf/BenchmarkDimensionTableOverhead.java | 16 +-
.../pinot/plugin/inputformat/avro/AvroUtils.java | 4 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 34 +-
.../pinot-input-format/pinot-parquet/pom.xml | 5 +
.../inputformat/parquet/ParquetMapUDFTest.java | 187 +++++
.../inputformat/protobuf/ProtoBufSchemaUtils.java | 4 +-
.../protobuf/ProtoBufSchemaUtilsTest.java | 2 +-
.../src/test/resources/complex_type_schema.json | 2 +-
.../pinot/plugin/minion/tasks/MinionTaskUtils.java | 3 +-
.../mergerollup/MergeRollupTaskGenerator.java | 3 +-
.../minion/tasks/purge/PurgeTaskGenerator.java | 3 +-
.../RealtimeToOfflineSegmentsTaskGenerator.java | 3 +-
.../RefreshSegmentTaskGenerator.java | 3 +-
.../SegmentGenerationAndPushTaskGenerator.java | 3 +-
.../UpsertCompactionTaskGenerator.java | 3 +-
.../UpsertCompactMergeTaskGenerator.java | 42 +-
.../stream/kafka20/KafkaConsumerFactory.java | 9 +
.../SynchronizedKafkaStreamMetadataProvider.java} | 23 +-
.../stream/kafka30/KafkaConsumerFactory.java | 9 +
.../SynchronizedKafkaStreamMetadataProvider.java} | 23 +-
.../pinot-timeseries-m3ql/pom.xml | 26 +
.../src/main/codegen/templates/M3qlParser.jj | 262 ++++++
.../pinot/tsdb/m3ql/M3TimeSeriesPlanner.java | 11 +-
.../apache/pinot/tsdb/m3ql/parser/Tokenizer.java | 119 ---
.../pinot/tsdb/m3ql/parser/M3qlParserTest.java | 167 ++++
.../rel/rules/PinotEvaluateLiteralRule.java | 2 +-
.../pinot/query/QueryPlannerRuleOptionsTest.java | 7 +
.../apache/pinot/query/runtime/QueryRunner.java | 18 +-
.../pinot/query/runtime/operator/LeafOperator.java | 4 +
pinot-segment-local/pom.xml | 5 +
.../local/data/manager/TableDataManager.java | 22 +-
.../creator/ColumnarSegmentCreationDataSource.java | 95 +++
.../creator/impl/SegmentColumnarIndexCreator.java | 48 ++
.../creator/impl/SegmentDictionaryCreator.java | 94 +++
.../impl/SegmentIndexCreationDriverImpl.java | 169 +++-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 4 +
.../inv/RawValueBitmapInvertedIndexCreator.java | 325 ++++++++
.../stats/AbstractColumnStatisticsCollector.java | 34 +
.../ColumnarSegmentPreIndexStatsContainer.java | 178 ++++
.../stats/DoubleColumnPreIndexStatsCollector.java | 48 +-
.../stats/FloatColumnPreIndexStatsCollector.java | 48 +-
.../stats/IntColumnPreIndexStatsCollector.java | 48 +-
.../stats/LongColumnPreIndexStatsCollector.java | 48 +-
.../stats/MapColumnPreIndexStatsCollector.java | 98 ++-
.../stats/NoDictColumnStatisticsCollector.java | 45 +-
.../stats/SegmentPreIndexStatsCollectorImpl.java | 32 +-
.../creator/impl/stats/StatsCollectorUtil.java | 65 ++
.../index/dictionary/DictionaryIndexType.java | 7 +-
.../segment/index/inverted/InvertedIndexType.java | 19 +-
.../ColumnMinMaxValueGenerator.java | 4 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 43 +-
.../loader/invertedindex/InvertedIndexHandler.java | 90 +-
.../readers/RawValueBitmapInvertedIndexReader.java | 162 ++++
.../segment/readers/DefaultValueColumnReader.java | 396 +++++++++
.../segment/readers/PinotSegmentColumnReader.java | 118 +++
.../readers/PinotSegmentColumnReaderFactory.java | 186 +++++
.../readers/PinotSegmentColumnReaderImpl.java | 367 ++++++++
.../MutableArc.java => ReloadJobStatus.java} | 56 +-
.../local/utils/ServerReloadJobStatusCache.java | 174 ++++
.../utils/ServerReloadJobStatusCacheConfig.java | 49 +-
.../segment/local/utils/TableConfigUtils.java | 81 +-
.../ExpressionTransformerTest.java | 34 +-
.../creator/impl/ColumnReaderInterfaceTest.java | 129 +++
.../impl/ColumnarRowMajorEquivalenceTest.java | 85 ++
.../creator/impl/ColumnarSchemaEvolutionTest.java | 55 ++
.../impl/ColumnarSegmentBuildingTestBase.java | 578 +++++++++++++
.../AbstractColumnStatisticsCollectorTest.java | 708 ++++++++++++++++
.../stats/MapColumnPreIndexStatsCollectorTest.java | 139 +++
.../stats/NoDictColumnStatisticsCollectorTest.java | 8 +-
.../inv/RawValueBitmapInvertedIndexTest.java | 302 +++++++
.../index/loader/SegmentPreProcessorTest.java | 18 +-
.../readers/DefaultValueColumnReaderTest.java | 613 ++++++++++++++
.../readers/PinotSegmentColumnReaderImplTest.java | 927 +++++++++++++++++++++
.../utils/ServerReloadJobStatusCacheTest.java | 312 +++++++
.../segment/local/utils/TableConfigUtilsTest.java | 180 +++-
.../pinot/segment/spi/AggregationFunctionType.java | 21 +-
.../pinot/segment/spi/creator/SegmentCreator.java | 4 +
.../pinot/segment/spi/index/IndexCreator.java | 85 ++
.../creator/CombinedInvertedIndexCreator.java | 113 +++
.../DictionaryBasedInvertedIndexCreator.java | 65 ++
.../spi/index/creator/ForwardIndexCreator.java | 126 +++
.../creator/RawValueBasedInvertedIndexCreator.java | 30 +
.../pinot/server/api/AdminApiApplication.java | 4 +
.../api/resources/ControllerJobStatusResource.java | 29 +-
.../pinot/server/api/resources/DebugResource.java | 26 +-
.../pinot/server/api/resources/TablesResource.java | 24 +-
.../pinot/server/starter/ServerInstance.java | 10 +-
.../server/starter/helix/BaseServerStarter.java | 13 +-
.../FreshnessBasedConsumptionStatusChecker.java | 26 +-
.../starter/helix/HelixInstanceDataManager.java | 39 +-
.../IngestionBasedConsumptionStatusChecker.java | 6 +-
.../helix/OffsetBasedConsumptionStatusChecker.java | 4 +-
.../helix/SegmentMessageHandlerFactory.java | 8 +-
.../starter/helix/SegmentReloadStatusValue.java | 10 +
.../apache/pinot/server/api/AccessControlTest.java | 5 +-
.../apache/pinot/server/api/BaseResourceTest.java | 19 +-
...FreshnessBasedConsumptionStatusCheckerTest.java | 156 +++-
.../OffsetBasedConsumptionStatusCheckerTest.java | 18 +-
.../pinot/spi/annotations/ScalarFunction.java | 5 +
.../pinot/spi/data/readers/ColumnReader.java | 295 +++++++
.../spi/data/readers/ColumnReaderFactory.java | 88 ++
.../pinot/spi/stream/StreamConsumerFactory.java | 4 +
.../pinot/spi/stream/StreamMetadataProvider.java | 3 +
.../java/org/apache/pinot/spi/trace/Tracer.java | 7 +-
.../java/org/apache/pinot/spi/trace/Tracing.java | 2 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 4 +-
.../org/apache/pinot/spi/utils/JsonUtilsTest.java | 8 +-
pinot-tools/pom.xml | 4 +
...art.java => BaseballRawEncodingQuickstart.java} | 10 +-
.../apache/pinot/tools/LogicalTableQuickstart.java | 206 +++++
.../pinot/tools/admin/PinotAdministrator.java | 2 +
.../command/LaunchBackfillIngestionJobCommand.java | 323 +++++++
.../command/LaunchDataIngestionJobCommand.java | 16 +-
.../ordersAPAC}/ingestionJobSpec.yaml | 50 +-
.../ordersAPAC_offline_table_config.json} | 8 +-
.../batch/orders/ordersAPAC/ordersAPAC_schema.json | 27 +
.../ordersAPAC/rawdata/ordersAPAC_sample.csv | 51 ++
.../ordersEU}/ingestionJobSpec.yaml | 50 +-
.../ordersEU/ordersEU_offline_table_config.json} | 8 +-
.../batch/orders/ordersEU/ordersEU_schema.json | 27 +
.../orders/ordersEU/rawdata/ordersEU_sample.csv | 51 ++
.../ordersUS}/ingestionJobSpec.yaml | 50 +-
.../ordersUS/ordersUS_offline_table_config.json} | 8 +-
.../batch/orders/ordersUS/ordersUS_schema.json | 27 +
.../orders/ordersUS/rawdata/ordersUS_sample.csv | 51 ++
.../examples/logicalTables/orders_schema.json | 27 +
.../baseballStats_offline_table_config.json | 9 +-
...s_offline_table_config_raw_inverted_index.json} | 12 +-
pom.xml | 40 +-
250 files changed, 14234 insertions(+), 1475 deletions(-)
create mode 100644
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentApiClient.java
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/array/ArraysOverlapScalarFunction.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/function/scalar/IpAddressFunctionsTest.java
rename
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/{StrictRealtimeSegmentAssignment.java
=> BaseStrictRealtimeSegmentAssignment.java} (70%)
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/MultiTierStrictRealtimeSegmentAssignment.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SingleTierStrictRealtimeSegmentAssignment.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentMetadataUtils.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/RawValueInvertedIndexFilterOperator.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/ArrayAggMvFunctionTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/ListAggFunctionTest.java
copy
pinot-core/src/{main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java
=>
test/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelStepEventWithExtraFieldsTest.java}
(54%)
copy
pinot-core/src/test/java/org/apache/pinot/queries/{ArrayAggQueriesTest.java =>
ArrayAggMvQueriesTest.java} (53%)
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/ListAggQueriesTest.java
create mode 100644
pinot-plugins/pinot-input-format/pinot-parquet/src/test/java/org/apache/pinot/plugin/inputformat/parquet/ParquetMapUDFTest.java
copy
pinot-plugins/pinot-stream-ingestion/{pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisMsgOffsetFactory.java
=>
pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/SynchronizedKafkaStreamMetadataProvider.java}
(54%)
copy
pinot-plugins/pinot-stream-ingestion/{pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisMsgOffsetFactory.java
=>
pinot-kafka-3.0/src/main/java/org/apache/pinot/plugin/stream/kafka30/SynchronizedKafkaStreamMetadataProvider.java}
(54%)
create mode 100644
pinot-plugins/pinot-timeseries-lang/pinot-timeseries-m3ql/src/main/codegen/templates/M3qlParser.jj
delete mode 100644
pinot-plugins/pinot-timeseries-lang/pinot-timeseries-m3ql/src/main/java/org/apache/pinot/tsdb/m3ql/parser/Tokenizer.java
create mode 100644
pinot-plugins/pinot-timeseries-lang/pinot-timeseries-m3ql/src/test/java/org/apache/pinot/tsdb/m3ql/parser/M3qlParserTest.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/ColumnarSegmentCreationDataSource.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/RawValueBitmapInvertedIndexCreator.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/StatsCollectorUtil.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/RawValueBitmapInvertedIndexReader.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReader.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java
copy
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/{nativefst/mutablefst/MutableArc.java
=> ReloadJobStatus.java} (51%)
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ServerReloadJobStatusCache.java
copy
pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/TableSizeInfo.java
=>
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ServerReloadJobStatusCacheConfig.java
(55%)
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnReaderInterfaceTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarRowMajorEquivalenceTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSchemaEvolutionTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTestBase.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/stats/AbstractColumnStatisticsCollectorTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/inv/RawValueBitmapInvertedIndexTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReaderTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImplTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/ServerReloadJobStatusCacheTest.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReaderFactory.java
copy
pinot-tools/src/main/java/org/apache/pinot/tools/{GeoSpatialQuickStart.java =>
BaseballRawEncodingQuickstart.java} (85%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/LogicalTableQuickstart.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/LaunchBackfillIngestionJobCommand.java
copy pinot-tools/src/main/resources/examples/batch/{sorted =>
orders/ordersAPAC}/ingestionJobSpec.yaml (62%)
copy
pinot-tools/src/main/resources/examples/batch/{baseballStats/baseballStats_offline_table_config.json
=> orders/ordersAPAC/ordersAPAC_offline_table_config.json} (78%)
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersAPAC/ordersAPAC_schema.json
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersAPAC/rawdata/ordersAPAC_sample.csv
copy pinot-tools/src/main/resources/examples/batch/{sorted =>
orders/ordersEU}/ingestionJobSpec.yaml (62%)
copy
pinot-tools/src/main/resources/examples/batch/{baseballStats/baseballStats_offline_table_config.json
=> orders/ordersEU/ordersEU_offline_table_config.json} (78%)
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersEU/ordersEU_schema.json
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersEU/rawdata/ordersEU_sample.csv
copy pinot-tools/src/main/resources/examples/batch/{sorted =>
orders/ordersUS}/ingestionJobSpec.yaml (62%)
copy
pinot-tools/src/main/resources/examples/batch/{baseballStats/baseballStats_offline_table_config.json
=> orders/ordersUS/ordersUS_offline_table_config.json} (78%)
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersUS/ordersUS_schema.json
create mode 100644
pinot-tools/src/main/resources/examples/batch/orders/ordersUS/rawdata/ordersUS_sample.csv
create mode 100644
pinot-tools/src/main/resources/examples/logicalTables/orders_schema.json
copy
pinot-tools/src/main/resources/examples/minions/batch/baseballStats/{baseballStats_offline_table_config.json
=> baseballStats_offline_table_config_raw_inverted_index.json} (88%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
