This is an automated email from the ASF dual-hosted git repository.
jlli pushed a change to branch support-spark-preprocessing
in repository https://gitbox.apache.org/repos/asf/pinot.git.
discard aa0792c Support data preprocessing in Spark framework
add cf68e22 [7187] Validate CSV Header For Configured Delimiter (#7237)
add f11b5d6 Only emit 0 for LLC_PARTITION_CONSUMING if
LLRealtimeSegmentDataManager isn't stopped (#7322)
add bfe64b5 reformat pinot-java-client and pinot-jdbc-client (#7326)
add aa5f318 add mode aggregation function (#7318)
add 94b9ca3 cleanup segment preprocessor a bit to make next RP smaller
(#7294)
add 892a0c1 Cleanup: Replace `Incubator Pinot` reference in the UI with
`Apache Pinot`. (#7339)
add f3ce66f Code style pinot-segment-spi and pinot-segment-local (#7317)
add 612803e Add an optional flag to disable swagger in Pinot servers.
(#7341)
add 52a2e40 Fix the issue with concurrent modification for segment
lineage (#7343)
add a074cde mark and sweep indices for V3 segment format (#7301)
add b71f38a Infer the data type for LiteralTransformFunction (#7332)
add 5417cbb Delete metadata properly on table deletion (#7329)
add 9116710 7090 segmentnamegenerator accept input file parameter (#7222)
add 48c1fc0 Fix TableNotFound error message in
PinotHelixResourceManager.getExistingTableNamesWithType() function. (#7340)
add 888c961 Add segmentNameGeneratorType config to IndexingConfig (#7346)
add 7e9ca6a Reformat code base with line length 120 (#7342)
add bb34406 reformat pinot-controller (#7331)
add 2fddf28 fixing code styling for pinot-broker (#7333)
add 4b8ca72 reformat pinot-server (#7334)
add 7a44f4a Manually trigger PeriodicTask (#7174)
add 58d86a6 reformat
pinot-perf/pinot-tools/pinot-integration-tests/pinot-minion (#7335)
add ce2c367 fixing code styling for pinot-core (#7325)
add f9204f7 Support data preprocessing in Spark framework
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aa0792c)
\
N -- N -- N refs/heads/support-spark-preprocessing (f9204f7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
config/checkstyle.xml | 10 +-
config/codestyle-intellij.xml | 12 +-
.../pinotdruidbenchmark/PinotThroughput.java | 1 +
.../images/pinot/etc/conf/pinot-broker-log4j2.xml | 8 +-
.../pinot/etc/conf/pinot-controller-log4j2.xml | 8 +-
.../images/pinot/etc/conf/pinot-server-log4j2.xml | 8 +-
pinot-broker/pom.xml | 3 -
.../apache/pinot/broker/api/RequestStatistics.java | 4 +-
.../broker/api/resources/PinotBrokerDebug.java | 26 +-
.../api/resources/PinotBrokerHealthCheck.java | 11 +-
.../broker/api/resources/PinotBrokerRouting.java | 15 +-
.../broker/api/resources/PinotClientRequest.java | 44 +-
.../broker/broker/helix/BaseBrokerStarter.java | 3 +-
.../broker/broker/helix/ClusterChangeMediator.java | 3 +-
.../broker/broker/helix/HelixBrokerStarter.java | 6 +-
.../HelixExternalViewBasedQueryQuotaManager.java | 22 +-
.../apache/pinot/broker/queryquota/HitCounter.java | 8 +-
.../pinot/broker/queryquota/MaxHitRateTracker.java | 8 +-
.../requesthandler/BaseBrokerRequestHandler.java | 16 +-
.../pinot/broker/routing/RoutingManager.java | 20 +-
.../instanceselector/BaseInstanceSelector.java | 11 +-
.../segmentpruner/PartitionSegmentPruner.java | 1 +
.../segmentpruner/SegmentPrunerFactory.java | 11 +-
.../routing/segmentpruner/TimeSegmentPruner.java | 19 +-
.../segmentpruner/interval/IntervalTree.java | 33 +-
.../segmentselector/RealtimeSegmentSelector.java | 23 +-
.../segmentselector/SegmentSelectorFactory.java | 1 +
...elixExternalViewBasedQueryQuotaManagerTest.java | 9 +-
.../broker/queryquota/MaxHitRateTrackerTest.java | 3 +-
.../request/PqlAndCalciteSqlCompatibilityTest.java | 51 +-
.../LiteralOnlyBrokerRequestTest.java | 18 +-
.../broker/requesthandler/QueryValidationTest.java | 26 +-
.../{IntervalST => intervalst}/IntervalTest.java | 2 +-
.../IntervalTreeTest.java | 2 +-
.../src/test/resources/SampleExternalView.json | 2652 ++++++++--------
pinot-broker/src/test/resources/log4j2.xml | 2 +-
pinot-clients/pinot-java-client/pom.xml | 3 -
.../apache/pinot/client/DynamicBrokerSelector.java | 19 +-
.../org/apache/pinot/client/ExecutionStats.java | 48 +-
.../apache/pinot/client/ExternalViewReader.java | 15 +-
.../apache/pinot/client/ResultTableResultSet.java | 1 -
.../apache/pinot/client/SimpleBrokerSelector.java | 2 +-
.../java/org/apache/pinot/client/TextTable.java | 29 +-
.../pinot/client/AggregationResultSetTest.java | 32 +-
.../pinot/client/DynamicBrokerSelectorTest.java | 47 +-
.../apache/pinot/client/ExecutionStatsTest.java | 48 +-
.../pinot/client/ExternalViewReaderTest.java | 40 +-
.../apache/pinot/client/GroupByResultSetTest.java | 33 +-
.../pinot/client/ResultTableResultSetTest.java | 37 +-
.../pinot/client/SelectionResultSetTest.java | 31 +-
.../org/apache/pinot/client/TextTableTest.java | 23 +-
.../src/test/resources/log4j2.xml | 2 +-
.../org/apache/pinot/client/aggregation.json | 15 +-
.../apache/pinot/client/aggregationGroupBy.json | 79 +-
.../org/apache/pinot/client/exception.json | 12 +-
.../org/apache/pinot/client/selection.json | 3302 +++++++++++++++++++-
pinot-clients/pinot-jdbc-client/pom.xml | 6 +-
.../pinot/client/PinotConnectionMetaData.java | 6 +-
.../java/org/apache/pinot/client/PinotDriver.java | 6 +-
.../pinot/client/PinotPreparedStatement.java | 2 +-
.../apache/pinot/client/PinotResultMetadata.java | 6 +-
.../org/apache/pinot/client/PinotStatement.java | 2 +-
.../controller/PinotControllerTransport.java | 6 +-
.../client/controller/request/TableRequest.java | 1 -
.../response/ControllerTenantBrokerResponse.java | 10 +-
.../client/controller/response/SchemaResponse.java | 12 +-
.../client/controller/response/TableResponse.java | 2 +-
.../org/apache/pinot/client/utils/Constants.java | 23 +-
.../apache/pinot/client/utils/DateTimeUtils.java | 29 +-
.../org/apache/pinot/client/utils/DriverUtils.java | 2 +
.../client/DummyPinotClientTransportFactory.java | 2 +-
.../org/apache/pinot/client/PinotDriverTest.java | 11 +-
.../pinot/client/PinotPreparedStatementTest.java | 15 +-
.../src/test/resources/result_table.json | 72 +-
.../src/test/resources/selection.json | 3302 +++++++++++++++++++-
pinot-common/pom.xml | 36 +-
.../assignment/InstanceAssignmentConfigUtils.java | 33 +-
.../common/assignment/InstancePartitions.java | 9 +-
.../common/assignment/InstancePartitionsUtils.java | 26 +-
.../pinot/common/exception/QueryException.java | 33 +-
.../pinot/common/function/DateTimeUtils.java | 4 +-
.../pinot/common/function/FunctionInvoker.java | 10 +-
.../pinot/common/function/FunctionRegistry.java | 10 +-
.../apache/pinot/common/function/TimeZoneKey.java | 8 +-
.../common/function/scalar/DateTimeFunctions.java | 6 +-
.../apache/pinot/common/lineage/LineageEntry.java | 4 +-
.../pinot/common/lineage/SegmentLineage.java | 3 +-
.../common/lineage/SegmentLineageAccessHelper.java | 25 +-
.../common/messages/RunPeriodicTaskMessage.java | 66 +
.../pinot/common/metadata/ZKMetadataProvider.java | 72 +-
.../metadata/instance/InstanceZKMetadata.java | 6 +-
.../metadata/segment/SegmentPartitionMetadata.java | 3 +-
.../common/metadata/segment/SegmentZKMetadata.java | 3 +-
.../pinot/common/metrics/AbstractMetrics.java | 40 +-
.../apache/pinot/common/metrics/BrokerMetrics.java | 3 +-
.../pinot/common/metrics/BrokerQueryPhase.java | 9 +-
.../apache/pinot/common/metrics/BrokerTimer.java | 6 +-
.../pinot/common/metrics/ControllerMetrics.java | 3 +-
.../pinot/common/metrics/PinotMetricUtils.java | 21 +-
.../apache/pinot/common/metrics/ServerMetrics.java | 3 +-
.../pinot/common/metrics/ValidationMetrics.java | 9 +-
.../apache/pinot/common/minion/MinionClient.java | 20 +-
.../common/minion/MinionTaskMetadataUtils.java | 44 +-
.../protocols/SegmentCompletionProtocol.java | 27 +-
.../response/broker/BrokerResponseNative.java | 12 +-
.../common/response/broker/SelectionResults.java | 3 +-
.../restlet/resources/SegmentConsumerInfo.java | 3 +-
.../common/restlet/resources/SegmentErrorInfo.java | 4 +-
.../restlet/resources/SegmentServerDebugInfo.java | 5 +-
.../common/restlet/resources/SegmentSizeInfo.java | 3 +-
.../resources/StartReplaceSegmentsRequest.java | 3 +-
.../restlet/resources/TableMetadataInfo.java | 6 +-
.../segment/generation/SegmentGenerationUtils.java | 11 +-
.../common/tier/TimeBasedTierSegmentSelector.java | 6 +-
.../common/utils/ClientSSLContextGenerator.java | 32 +-
.../org/apache/pinot/common/utils/DataSchema.java | 10 +-
.../common/utils/FileUploadDownloadClient.java | 125 +-
.../apache/pinot/common/utils/HLCSegmentName.java | 3 +-
.../org/apache/pinot/common/utils/HashUtil.java | 11 +-
.../apache/pinot/common/utils/LLCSegmentName.java | 6 +-
.../pinot/common/utils/NamedThreadFactory.java | 6 +-
.../apache/pinot/common/utils/PinotAppConfigs.java | 50 +-
.../apache/pinot/common/utils/PinotDataType.java | 9 +-
.../pinot/common/utils/PrimitiveArrayUtils.java | 5 +-
.../org/apache/pinot/common/utils/SchemaUtils.java | 12 +-
.../apache/pinot/common/utils/SegmentUtils.java | 19 +-
.../apache/pinot/common/utils/ServiceStatus.java | 55 +-
.../pinot/common/utils/TarGzCompressionUtils.java | 22 +-
.../common/utils/config/TableConfigUtils.java | 28 +-
.../pinot/common/utils/config/TierConfigUtils.java | 13 +-
.../common/utils/fetcher/BaseSegmentFetcher.java | 5 +-
.../common/utils/fetcher/HttpSegmentFetcher.java | 14 +-
.../pinot/common/utils/grpc/GrpcQueryClient.java | 3 +-
.../common/utils/grpc/GrpcRequestBuilder.java | 7 +-
.../pinot/common/utils/helix/HelixHelper.java | 61 +-
.../common/utils/helix/LeadControllerUtils.java | 6 +-
.../PinotHelixPropertyStoreZnRecordProvider.java | 3 +-
.../pinot/common/utils/request/RequestUtils.java | 4 +-
.../common/utils/webhdfs/WebHdfsV1Client.java | 14 +-
.../utils/BrokerRequestComparisonUtils.java | 54 +-
.../parsers/PinotQuery2BrokerRequestConverter.java | 3 +-
.../pql/parsers/Pql2CompilationException.java | 3 +-
.../org/apache/pinot/pql/parsers/Pql2Compiler.java | 16 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 7 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 6 +-
.../pinot/pql/parsers/pql2/ast/FilterKind.java | 4 +-
.../pinot/pql/parsers/pql2/ast/OrderByAstNode.java | 12 +-
.../pql/parsers/pql2/ast/PredicateListAstNode.java | 3 +-
.../pinot/pql/parsers/pql2/ast/SelectAstNode.java | 4 +-
.../pql/parsers/pql2/ast/TableNameAstNode.java | 3 +-
.../pql2/ast/TextMatchPredicateAstNode.java | 3 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 64 +-
.../pinot/common/data/DateTimeFormatSpecTest.java | 173 +-
.../apache/pinot/common/data/FieldSpecTest.java | 65 +-
.../org/apache/pinot/common/data/SchemaTest.java | 48 +-
.../function/AggregationFunctionTypeTest.java | 36 +-
.../pinot/common/function/JsonFunctionsTest.java | 27 +-
.../common/function/TransformFunctionTypeTest.java | 3 +-
.../pinot/common/http/MultiGetRequestTest.java | 6 +-
.../pinot/common/lineage/SegmentLineageTest.java | 19 +-
.../metadata/MetadataEqualsHashCodeTest.java | 20 +-
.../RealtimeToOfflineSegmentsTaskMetadataTest.java | 3 +-
.../common/metadata/SegmentZKMetadataTest.java | 45 +-
.../segment/ColumnPartitionMetadataTest.java | 3 +-
.../pinot/common/metrics/PinotMetricUtilsTest.java | 15 +-
.../pinot/common/minion/MinionClientTest.java | 11 +-
.../protocols/SegmentCompletionProtocolTest.java | 54 +-
.../transform/TransformExpressionTreeTest.java | 4 +-
.../generation/SegmentGenerationUtilsTest.java | 34 +-
.../pinot/common/tier/TierConfigUtilsTest.java | 41 +-
.../pinot/common/tier/TierSegmentSelectorTest.java | 10 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 11 +-
.../common/utils/FileUploadDownloadClientTest.java | 12 +-
.../pinot/common/utils/PinotDataTypeTest.java | 63 +-
.../pinot/common/utils/SegmentNameBuilderTest.java | 15 +-
.../pinot/common/utils/ServiceStatusTest.java | 31 +-
.../apache/pinot/common/utils/URIUtilsTest.java | 16 +-
.../common/utils/config/InstanceUtilsTest.java | 3 +-
.../common/utils/config/TableConfigSerDeTest.java | 47 +-
.../common/utils/config/TableConfigUtilsTest.java | 25 +-
.../common/utils/config/TagNameUtilsTest.java | 7 +-
.../utils/fetcher/SegmentFetcherFactoryTest.java | 3 +-
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 120 +-
.../request/BrokerRequestSerializationTest.java | 6 +-
.../pinot/request/BrokerResponseNativeTest.java | 7 +-
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 846 +++--
.../test/java/org/apache/pinot/util/TestUtils.java | 3 +-
pinot-connectors/pinot-spark-connector/pom.xml | 494 +--
pinot-connectors/pom.xml | 38 +-
pinot-controller/pom.xml | 3 -
.../pinot/controller/BaseControllerStarter.java | 24 +-
.../apache/pinot/controller/ControllerConf.java | 24 +-
...ControllerUserDefinedMessageHandlerFactory.java | 134 +
.../pinot/controller/LeadControllerManager.java | 18 +-
.../pinot/controller/api/CallableHttpGet.java | 8 +-
.../api/ControllerAdminApiApplication.java | 3 +-
.../pinot/controller/api/access/AccessControl.java | 8 +-
.../api/access/BasicAuthAccessControlFactory.java | 5 +-
.../pinot/controller/api/debug/TableDebugInfo.java | 9 +-
.../api/events/DefaultMetadataEventNotifier.java | 1 +
.../api/events/MetadataEventNotifier.java | 1 +
.../pinot/controller/api/resources/Constants.java | 4 +
.../api/resources/ControllerFilePathProvider.java | 16 +-
.../api/resources/ControllerResponseFilter.java | 7 +-
.../resources/LLCSegmentCompletionHandlers.java | 69 +-
.../api/resources/PinotBrokerRestletResource.java | 35 +-
.../api/resources/PinotClusterConfigs.java | 36 +-
.../api/resources/PinotControllerAuthResource.java | 5 +-
.../api/resources/PinotControllerHealthCheck.java | 10 +-
...PinotControllerPeriodicTaskRestletResource.java | 123 +
.../resources/PinotIngestionRestletResource.java | 26 +-
.../PinotInstanceAssignmentRestletResource.java | 18 +-
.../resources/PinotInstanceRestletResource.java | 92 +-
.../PinotLeadControllerRestletResource.java | 6 +-
.../api/resources/PinotSchemaRestletResource.java | 71 +-
.../api/resources/PinotSegmentRestletResource.java | 94 +-
.../PinotSegmentUploadDownloadRestletResource.java | 62 +-
.../api/resources/PinotTableIndexingConfigs.java | 12 +-
.../api/resources/PinotTableInstances.java | 27 +-
.../api/resources/PinotTableMetadataConfigs.java | 12 +-
.../api/resources/PinotTableRestletResource.java | 50 +-
.../controller/api/resources/PinotTableSchema.java | 9 +-
.../api/resources/PinotTableSegmentConfigs.java | 18 +-
.../api/resources/PinotTableTenantConfigs.java | 11 +-
.../api/resources/PinotTaskRestletResource.java | 18 +-
.../api/resources/PinotTenantRestletResource.java | 166 +-
.../controller/api/resources/ResourceUtils.java | 2 +
.../api/resources/ServerTableSizeReader.java | 7 +-
.../api/resources/TableConfigsRestletResource.java | 31 +-
.../api/resources/TableDebugResource.java | 23 +-
.../pinot/controller/api/resources/TableSize.java | 17 +-
.../pinot/controller/api/resources/TableViews.java | 14 +-
.../resources/WebApplicationExceptionMapper.java | 10 +-
.../api/resources/ZookeeperResource.java | 111 +-
.../controller/api/upload/SegmentValidator.java | 7 +-
.../pinot/controller/api/upload/ZKOperator.java | 11 +-
.../controller/helix/SegmentStatusChecker.java | 13 +-
...erResourceOnlineOfflineStateModelGenerator.java | 2 +
.../helix/core/PinotHelixResourceManager.java | 56 +-
...lixSegmentOnlineOfflineStateModelGenerator.java | 2 +
.../helix/core/PinotTableIdealStateBuilder.java | 18 +-
.../helix/core/SegmentDeletionManager.java | 2 +-
.../InstanceReplicaGroupPartitionSelector.java | 4 +-
.../segment/OfflineDimTableSegmentAssignment.java | 3 +-
.../segment/OfflineSegmentAssignment.java | 6 +-
.../segment/RealtimeSegmentAssignment.java | 19 +-
.../core/assignment/segment/SegmentAssignment.java | 5 +-
.../segment/SegmentAssignmentFactory.java | 3 +-
.../assignment/segment/SegmentAssignmentUtils.java | 3 +-
.../helix/core/minion/ClusterInfoAccessor.java | 8 +-
.../core/minion/MinionInstancesCleanupTask.java | 13 +-
.../core/minion/PinotHelixTaskResourceManager.java | 4 +-
.../helix/core/minion/PinotTaskManager.java | 2 +-
.../helix/core/minion/TaskMetricsEmitter.java | 3 +-
.../core/minion/generator/TaskGeneratorUtils.java | 3 +
.../core/periodictask/ControllerPeriodicTask.java | 26 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 56 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 7 +-
.../core/realtime/SegmentCompletionManager.java | 207 +-
.../segment/CommittingSegmentDescriptor.java | 6 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 6 +-
.../core/rebalance/RebalanceConfigConstants.java | 2 +
.../helix/core/rebalance/TableRebalancer.java | 27 +-
.../helix/core/relocation/SegmentRelocator.java | 1 +
.../helix/core/retention/RetentionManager.java | 3 +-
.../helix/core/util/HelixSetupUtils.java | 18 +-
.../controller/helix/starter/HelixConfig.java | 3 +
.../controller/recommender/RecommenderDriver.java | 12 +-
.../recommender/data/generator/DataGenerator.java | 81 +-
.../data/generator/DataGeneratorSpec.java | 112 +-
.../data/generator/GeneratorFactory.java | 7 +-
.../data/generator/MultiValueGeneratorHelper.java | 2 +
.../data/generator/NumberGenerator.java | 42 +-
.../data/generator/PatternMixtureGenerator.java | 86 +-
.../data/generator/PatternSeasonalGenerator.java | 113 +-
.../data/generator/PatternSequenceGenerator.java | 47 +-
.../data/generator/PatternSpikeGenerator.java | 123 +-
.../data/generator/PatternStringGenerator.java | 39 +-
.../recommender/data/generator/PatternType.java | 6 +-
.../data/generator/SchemaAnnotation.java | 1 +
.../data/generator/StringGenerator.java | 42 +-
.../exceptions/InvalidInputException.java | 4 +-
.../controller/recommender/io/ConfigManager.java | 3 +-
.../controller/recommender/io/InputManager.java | 14 +-
.../recommender/io/metadata/FieldMetadata.java | 14 +-
.../io/metadata/SchemaWithMetaData.java | 10 +-
.../realtime/provisioning/MemoryEstimator.java | 46 +-
.../controller/recommender/rules/AbstractRule.java | 4 +-
.../recommender/rules/RulesToExecute.java | 13 +-
.../rules/impl/AggregateMetricsRule.java | 5 +-
.../recommender/rules/impl/BloomFilterRule.java | 15 +-
.../recommender/rules/impl/FlagQueryRule.java | 14 +-
.../rules/impl/InvertedSortedIndexJointRule.java | 19 +-
.../recommender/rules/impl/KafkaPartitionRule.java | 24 +-
.../NoDictionaryOnHeapDictionaryJointRule.java | 36 +-
.../rules/impl/PinotTablePartitionRule.java | 61 +-
.../recommender/rules/impl/RangeIndexRule.java | 15 +-
.../rules/impl/RealtimeProvisioningRule.java | 15 +-
.../recommender/rules/impl/SegmentSizeRule.java | 35 +-
.../rules/impl/VariedLengthDictionaryRule.java | 12 +-
.../recommender/rules/io/FlaggedQueries.java | 16 +-
.../recommender/rules/io/configs/IndexConfig.java | 196 +-
.../rules/io/configs/PartitionConfig.java | 18 +-
.../io/configs/SegmentSizeRecommendations.java | 32 +-
.../rules/io/params/BloomFilterRuleParams.java | 27 +-
.../rules/io/params/FlagQueryRuleParams.java | 11 +-
.../params/InvertedSortedIndexJointRuleParams.java | 111 +-
...oDictionaryOnHeapDictionaryJointRuleParams.java | 80 +-
.../rules/io/params/PartitionRuleParams.java | 56 +-
.../rules/io/params/RangeIndexRuleParams.java | 12 +-
.../io/params/RealtimeProvisioningRuleParams.java | 51 +-
.../rules/io/params/RecommenderConstants.java | 17 +-
.../rules/io/params/SegmentSizeRuleParams.java | 25 +-
.../recommender/rules/utils/FixedLenBitset.java | 69 +-
.../rules/utils/PredicateParseResult.java | 30 +-
.../utils/QueryInvertedSortedIndexRecommender.java | 157 +-
.../controller/tuner/TableConfigTunerRegistry.java | 10 +-
.../controller/tuner/TableConfigTunerUtils.java | 5 +-
.../controller/util/AutoAddInvertedIndex.java | 4 +-
.../util/ConsumingSegmentInfoReader.java | 20 +-
.../pinot/controller/util/FileIngestionHelper.java | 3 +-
.../controller/util/SegmentCompletionUtils.java | 5 +-
.../controller/util/SegmentIntervalUtils.java | 2 +
.../util/ServerSegmentMetadataReader.java | 3 +-
.../util/TableIngestionStatusHelper.java | 7 +-
.../pinot/controller/util/TableSizeReader.java | 99 +-
.../BrokerResourceValidationManager.java | 3 +-
.../controller/validation/StorageQuotaChecker.java | 72 +-
pinot-controller/src/main/resources/app/index.html | 2 +-
.../pinot/controller/ControllerTestSetup.java | 3 +-
.../pinot/controller/ControllerTestUtils.java | 70 +-
.../controller/LeadControllerManagerTest.java | 6 +-
.../pinot/controller/api/AccessControlTest.java | 3 +-
.../api/ConsumingSegmentInfoReaderTest.java | 75 +-
.../api/ControllerFilePathProviderTest.java | 6 +-
.../pinot/controller/api/PinotFileUploadTest.java | 6 +-
...PinotInstanceAssignmentRestletResourceTest.java | 25 +-
.../api/PinotSchemaRestletResourceTest.java | 33 +-
.../api/PinotSegmentRestletResourceTest.java | 12 +-
.../controller/api/PinotSegmentsMetadataTest.java | 120 +-
.../api/PinotTableRestletResourceTest.java | 3 +-
.../api/PinotTenantRestletResourceTest.java | 12 +-
.../api/SegmentCompletionProtocolDeserTest.java | 14 +-
.../controller/api/ServerTableSizeReaderTest.java | 95 +-
.../api/TableConfigsRestletResourceTest.java | 15 +-
.../pinot/controller/api/TableSizeReaderTest.java | 153 +-
.../pinot/controller/api/TableViewsTest.java | 85 +-
.../api/access/AuthenticationFilterTest.java | 16 +-
...otSegmentUploadDownloadRestletResourceTest.java | 8 +-
.../helix/ControllerInstanceToggleTest.java | 27 +-
.../controller/helix/ControllerSentinelTestV2.java | 36 +-
.../helix/ControllerTenantStatelessTest.java | 27 +-
.../pinot/controller/helix/ControllerTest.java | 92 +-
.../pinot/controller/helix/HelixHelperTest.java | 49 +-
.../helix/PinotControllerModeStatelessTest.java | 6 +-
.../controller/helix/SegmentStatusCheckerTest.java | 399 +--
.../pinot/controller/helix/TableCacheTest.java | 9 +-
.../PinotHelixResourceManagerStatelessTest.java | 1 +
.../helix/core/PinotHelixResourceManagerTest.java | 4 +-
.../OfflineDimTableSegmentAssignmentTest.java | 2 +-
...fflineNonReplicaGroupSegmentAssignmentTest.java | 3 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 3 +-
.../OfflineReplicaGroupSegmentAssignmentTest.java | 6 +-
...altimeNonReplicaGroupSegmentAssignmentTest.java | 6 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 9 +-
.../segment/SegmentAssignmentUtilsTest.java | 17 +-
.../minion/MinionInstancesCleanupTaskTest.java | 9 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 14 +-
.../helix/core/realtime/SegmentCompletionTest.java | 1218 ++++----
.../TableRebalancerClusterStatelessTest.java | 2 +-
.../helix/core/retention/RetentionManagerTest.java | 9 +-
.../core/retention/SegmentLineageCleanupTest.java | 34 +-
.../core/util/SegmentDeletionManagerTest.java | 44 +-
.../controller/recommender/TestConfigEngine.java | 89 +-
.../generator/MultiValueGeneratorHelperTest.java | 5 +-
.../data/generator/NumberGeneratorTest.java | 12 +-
.../data/generator/TimeGeneratorTest.java | 3 +-
.../realtime/provisioning/MemoryEstimatorTest.java | 12 +-
.../rules/impl/AggregateMetricsRuleTest.java | 26 +-
.../rules/impl/PinotTablePartitionRuleTest.java | 16 +-
.../rules/impl/SegmentSizeRuleTest.java | 63 +-
.../tuner/RealTimeAutoIndexTunerTest.java | 23 +-
.../controller/util/ListenerConfigUtilTest.java | 9 +-
.../validation/StorageQuotaCheckerTest.java | 8 +-
.../validation/ValidationManagerStatelessTest.java | 6 +-
pinot-core/pom.xml | 3 -
.../apache/pinot/core/auth/BasicAuthPrincipal.java | 8 +-
.../org/apache/pinot/core/auth/BasicAuthUtils.java | 3 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 152 +-
.../core/common/datatable/DataTableImplV3.java | 58 +-
.../core/data/manager/InstanceDataManager.java | 3 +-
.../manager/offline/DimensionTableDataManager.java | 8 +-
.../manager/realtime/DefaultSegmentCommitter.java | 3 +-
.../realtime/HLRealtimeSegmentDataManager.java | 11 +-
.../realtime/LLRealtimeSegmentDataManager.java | 194 +-
.../manager/realtime/PinotFSSegmentUploader.java | 6 +-
.../manager/realtime/RealtimeTableDataManager.java | 15 +-
.../realtime/SegmentBuildTimeLeaseExtender.java | 11 +-
.../manager/realtime/SegmentCommitterFactory.java | 12 +-
.../core/data/manager/realtime/TimerService.java | 5 +-
.../core/data/table/ConcurrentIndexedTable.java | 5 +-
.../pinot/core/data/table/SimpleIndexedTable.java | 3 +-
.../apache/pinot/core/data/table/TableResizer.java | 13 +-
.../table/UnboundedConcurrentIndexedTable.java | 4 +-
.../transform/function/ScalarFunctions.java | 2 +
.../transform/function/StAreaFunction.java | 10 +-
.../core/operator/InstanceResponseOperator.java | 6 +-
.../operator/blocks/PassThroughTransformBlock.java | 2 +-
.../core/operator/combine/BaseCombineOperator.java | 6 +-
.../combine/GroupByOrderByCombineOperator.java | 5 +-
...xValueBasedSelectionOrderByCombineOperator.java | 1 +
.../BaseRawValueBasedPredicateEvaluator.java | 1 -
.../FSTBasedRegexpPredicateEvaluatorFactory.java | 209 +-
.../operator/filter/predicate/PredicateUtils.java | 2 +
.../predicate/RangePredicateEvaluatorFactory.java | 3 +-
.../query/DictionaryBasedDistinctOperator.java | 7 +-
.../transform/PassThroughTransformOperator.java | 4 +-
.../function/AndOperatorTransformFunction.java | 4 +-
.../transform/function/BaseTransformFunction.java | 1 +
.../function/BinaryOperatorTransformFunction.java | 90 +-
.../transform/function/CaseTransformFunction.java | 121 +-
.../DateTimeConversionTransformFunction.java | 3 +-
.../function/DateTruncTransformFunction.java | 12 +-
.../function/GroovyTransformFunction.java | 12 +-
.../function/JsonExtractKeyTransformFunction.java | 11 +-
.../JsonExtractScalarTransformFunction.java | 19 +-
.../function/LiteralTransformFunction.java | 68 +-
.../function/LogicalOperatorTransformFunction.java | 2 +-
.../function/LookupTransformFunction.java | 3 +-
.../function/MapValueTransformFunction.java | 3 +-
.../function/OrOperatorTransformFunction.java | 4 +-
.../function/TimeConversionTransformFunction.java | 3 +-
.../function/TransformFunctionFactory.java | 36 +-
.../datetime/BaseDateTimeTransformer.java | 1 +
.../pinot/core/periodictask/BasePeriodicTask.java | 98 +-
.../pinot/core/periodictask/PeriodicTask.java | 11 +
.../core/periodictask/PeriodicTaskScheduler.java | 69 +-
.../plan/AggregationGroupByOrderByPlanNode.java | 3 +-
.../apache/pinot/core/plan/DocIdSetPlanNode.java | 2 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 6 +-
.../aggregation/function/AggregationFunction.java | 3 +-
.../function/AggregationFunctionFactory.java | 2 +
.../DistinctCountBitmapMVAggregationFunction.java | 1 +
.../DistinctCountMVAggregationFunction.java | 1 +
.../DistinctCountRawHLLAggregationFunction.java | 3 +-
...istinctCountThetaSketchAggregationFunction.java | 9 +-
.../function/ModeAggregationFunction.java | 691 ++++
.../groupby/DictionaryBasedGroupKeyGenerator.java | 6 +-
.../core/query/config/QueryExecutorConfig.java | 3 +-
.../core/query/config/QueryPlannerConfig.java | 2 +-
...ryBasedSingleColumnDistinctOrderByExecutor.java | 3 +-
.../pinot/core/query/optimizer/QueryOptimizer.java | 3 +-
.../optimizer/filter/NumericalFilterOptimizer.java | 37 +-
.../statement/JsonStatementOptimizer.java | 42 +-
.../query/pruner/ColumnValueSegmentPruner.java | 9 +-
.../core/query/reduce/BrokerReduceService.java | 7 +-
.../core/query/reduce/GroupByDataTableReducer.java | 10 +-
.../core/query/reduce/ResultReducerFactory.java | 2 +
.../query/reduce/SelectionDataTableReducer.java | 4 +-
.../core/query/request/context/QueryContext.java | 9 +-
.../core/query/request/context/ThreadTimer.java | 8 +-
.../query/scheduler/AbstractSchedulerGroup.java | 44 +-
.../query/scheduler/MultiLevelPriorityQueue.java | 84 +-
.../core/query/scheduler/PriorityScheduler.java | 66 +-
.../pinot/core/query/scheduler/QueryScheduler.java | 81 +-
.../query/scheduler/QuerySchedulerFactory.java | 5 +-
.../query/scheduler/SchedulerQueryContext.java | 23 +-
.../query/scheduler/fcfs/FCFSQueryScheduler.java | 6 +-
.../resources/BoundedAccountingExecutor.java | 50 +-
.../resources/PolicyBasedResourceManager.java | 14 +-
.../scheduler/resources/ResourceLimitPolicy.java | 26 +-
.../query/scheduler/resources/ResourceManager.java | 36 +-
.../resources/UnboundedResourceManager.java | 6 +-
.../scheduler/tokenbucket/TokenSchedulerGroup.java | 69 +-
.../query/selection/SelectionOperatorUtils.java | 3 +-
.../apache/pinot/core/query/utils/TrieNode.java | 6 +-
.../requesthandler/PinotQueryParserFactory.java | 3 +
.../processing/filter/RecordFilterFactory.java | 2 +
.../framework/SegmentProcessorFramework.java | 19 +-
.../processing/partitioner/PartitionerConfig.java | 3 +-
.../processing/partitioner/PartitionerFactory.java | 3 +-
.../TransformFunctionRecordTransformer.java | 3 +-
.../apache/pinot/core/startree/StarTreeUtils.java | 4 +
.../pinot/core/transport/AsyncQueryResponse.java | 4 +-
.../core/transport/InstanceRequestHandler.java | 24 +-
.../pinot/core/transport/ListenerConfig.java | 10 +-
.../apache/pinot/core/transport/QueryServer.java | 1 -
.../pinot/core/transport/ServerChannels.java | 6 +-
.../pinot/core/transport/ServerInstance.java | 2 +-
.../apache/pinot/core/util/ListenerConfigUtil.java | 53 +-
.../pinot/core/util/PeerServerSegmentFinder.java | 14 +-
.../core/util/SegmentCompletionProtocolUtils.java | 5 +
.../pinot/core/util/SegmentProcessorAvroUtils.java | 3 +-
.../java/org/apache/pinot/core/util/SizeUtil.java | 5 +-
.../pinot/core/util/SortedRangeIntersection.java | 19 +-
.../java/org/apache/pinot/core/util/TlsUtils.java | 16 +-
.../server/realtime/ControllerLeaderLocator.java | 24 +-
.../ServerSegmentCompletionProtocolHandler.java | 8 +-
.../pinot/core/common/ObjectSerDeUtilsTest.java | 68 +
.../core/common/datatable/DataTableSerDeTest.java | 4 +
.../core/common/datatable/DataTableUtilsTest.java | 5 +-
.../core/data/function/ArrayFunctionsTest.java | 159 +-
.../core/data/function/DateTimeFunctionsTest.java | 352 ++-
.../data/function/GroovyFunctionEvaluatorTest.java | 34 +-
.../core/data/function/JsonFunctionsTest.java | 41 +-
.../data/manager/BaseTableDataManagerTest.java | 30 +-
.../offline/DimensionTableDataManagerTest.java | 3 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 93 +-
.../realtime/PinotFSSegmentUploaderTest.java | 17 +-
.../pinot/core/data/table/IndexedTableTest.java | 26 +-
.../pinot/core/data/table/TableResizerTest.java | 49 +-
.../core/geospatial/serde/GeometrySerdeTest.java | 14 +-
.../core/geospatial/transform/GeoFunctionTest.java | 18 +-
.../geospatial/transform/StAreaFunctionTest.java | 6 +-
.../geospatial/transform/StPointFunctionTest.java | 10 +-
.../operator/ThreadCpuTimeMeasurementTest.java | 46 +-
.../dociditerators/AndDocIdIteratorTest.java | 7 +-
.../dociditerators/OrDocIdIteratorTest.java | 7 +-
.../predicate/PredicateEvaluatorTestUtils.java | 3 +
...ngeOfflineDictionaryPredicateEvaluatorTest.java | 23 +-
.../function/AdditionTransformFunctionTest.java | 7 +-
.../function/AndOperatorTransformFunctionTest.java | 4 +-
.../function/ArrayBaseTransformFunctionTest.java | 12 +-
.../BinaryOperatorTransformFunctionTest.java | 10 +-
.../function/CaseTransformFunctionTest.java | 68 +-
.../DateTimeConversionTransformFunctionTest.java | 18 +-
.../function/DivisionTransformFunctionTest.java | 10 +-
.../function/GroovyTransformFunctionTest.java | 49 +-
.../JsonExtractScalarTransformFunctionTest.java | 9 +-
.../function/LiteralTransformFunctionTest.java | 20 +-
.../LogicalOperatorTransformFunctionTest.java | 2 +-
.../function/LookupTransformFunctionTest.java | 80 +-
.../function/ModuloTransformFunctionTest.java | 10 +-
.../MultiplicationTransformFunctionTest.java | 7 +-
.../function/OrOperatorTransformFunctionTest.java | 4 +-
.../function/SubtractionTransformFunctionTest.java | 10 +-
.../TimeConversionTransformFunctionTest.java | 29 +-
.../function/ValueInTransformFunctionTest.java | 17 +-
.../datetime/DateTimeConverterTest.java | 257 +-
.../periodictask/PeriodicTaskSchedulerTest.java | 87 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 103 +-
.../DoubleAggregationResultHolderTest.java | 6 +-
.../function/AggregationFunctionFactoryTest.java | 7 +
.../DictionaryBasedGroupKeyGeneratorTest.java | 3 +-
.../query/aggregation/groupby/GroupByTrimTest.java | 9 +-
.../core/query/optimizer/QueryOptimizerTest.java | 26 +-
.../filter/NumericalFilterOptimizerTest.java | 106 +-
.../filter/TimePredicateFilterOptimizerTest.java | 34 +-
.../statement/JsonStatementOptimizerTest.java | 99 +-
.../pruner/SelectionQuerySegmentPrunerTest.java | 2 +-
.../core/query/reduce/HavingFilterHandlerTest.java | 13 +-
.../query/reduce/PostAggregationHandlerTest.java | 7 +-
.../BrokerRequestToQueryContextConverterTest.java | 21 +-
.../scheduler/MultiLevelPriorityQueueTest.java | 133 +-
.../query/scheduler/PrioritySchedulerTest.java | 90 +-
.../pinot/core/query/scheduler/TestHelper.java | 2 +
.../core/query/scheduler/TestSchedulerGroup.java | 6 +-
.../query/scheduler/TestSchedulerGroupFactory.java | 14 +-
.../scheduler/fcfs/FCFSSchedulerGroupTest.java | 6 +-
.../resources/BoundedAccountingExecutorTest.java | 36 +-
.../tokenbucket/TokenSchedulerGroupTest.java | 44 +-
.../selection/SelectionOperatorServiceTest.java | 120 +-
.../fakestream/FakePartitionLevelConsumer.java | 27 +-
.../impl/fakestream/FakeStreamConfigUtils.java | 3 +
.../impl/fakestream/FakeStreamConsumerFactory.java | 3 +-
.../impl/fakestream/FakeStreamLevelConsumer.java | 6 +-
.../fakestream/FakeStreamMetadataProvider.java | 9 +-
.../core/realtime/stream/StreamConfigTest.java | 8 +-
.../segment/processing/framework/ReducerTest.java | 24 +-
.../framework/SegmentProcessorFrameworkTest.java | 25 +
.../org/apache/pinot/core/util/CrcUtilsTest.java | 2 +-
.../pinot/core/util/DoubleComparisonUtil.java | 4 +-
.../core/util/PeerServerSegmentFinderTest.java | 12 +-
.../apache/pinot/core/util/SchemaUtilsTest.java | 53 +-
.../core/util/SortedRangeIntersectionTest.java | 2 +-
.../apache/pinot/queries/BooleanQueriesTest.java | 5 +-
.../queries/DistinctCountBitmapQueriesTest.java | 14 +-
.../pinot/queries/DistinctCountQueriesTest.java | 7 +-
.../DistinctCountThetaSketchQueriesTest.java | 6 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 86 +-
.../queries/FSTBasedRegexpLikeQueriesTest.java | 59 +-
.../apache/pinot/queries/H3IndexQueriesTest.java | 7 +-
.../org/apache/pinot/queries/IdSetQueriesTest.java | 6 +-
...erSegmentAggregationSingleValueQueriesTest.java | 3 +-
...terSegmentAggregationMultiValueQueriesTest.java | 15 +-
...erSegmentAggregationSingleValueQueriesTest.java | 2 +-
.../InterSegmentOrderByMultiValueQueriesTest.java | 11 +-
.../InterSegmentOrderBySingleValueQueriesTest.java | 284 +-
...terSegmentResultTableMultiValueQueriesTest.java | 386 ++-
...erSegmentResultTableSingleValueQueriesTest.java | 84 +-
.../pinot/queries/JsonDataTypeQueriesTest.java | 76 +-
.../apache/pinot/queries/JsonPathQueriesTest.java | 87 +-
.../org/apache/pinot/queries/ModeQueriesTest.java | 977 ++++++
.../NoDictionaryCompressionQueriesTest.java | 141 +-
.../queries/PercentileTDigestQueriesTest.java | 3 +-
.../org/apache/pinot/queries/QueriesTestUtils.java | 9 +-
.../RangePredicateWithSortedInvertedIndexTest.java | 4 +-
...SegmentPartitionedDistinctCountQueriesTest.java | 10 +-
.../queries/SegmentWithNullValueVectorTest.java | 14 +-
.../apache/pinot/queries/StUnionQueriesTest.java | 20 +-
.../pinot/queries/SumPrecisionQueriesTest.java | 41 +-
.../pinot/queries/TextSearchQueriesTest.java | 1196 ++++---
.../apache/pinot/queries/TimestampQueriesTest.java | 10 +-
.../apache/pinot/queries/TransformQueriesTest.java | 6 +-
.../realtime/ControllerLeaderLocatorTest.java | 32 +-
.../resources/data/dimBaseballTeams_schema.json | 4 +-
pinot-core/src/test/resources/log4j2.xml | 2 +-
pinot-distribution/pinot-assembly.xml | 128 +-
pinot-integration-tests/pom.xml | 75 +-
.../java/org/apache/pinot/compat/tests/BaseOp.java | 12 +-
.../pinot/compat/tests/ClusterDescriptor.java | 22 +-
.../pinot/compat/tests/CompatibilityOpsRunner.java | 6 +-
.../org/apache/pinot/compat/tests/QueryOp.java | 11 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 38 +-
.../pinot/compat/tests/SqlResultComparator.java | 5 +-
.../org/apache/pinot/compat/tests/StreamOp.java | 33 +-
.../org/apache/pinot/compat/tests/TableOp.java | 25 +-
.../java/org/apache/pinot/compat/tests/Utils.java | 2 +
.../tests/AdminConsoleIntegrationTest.java | 32 +-
.../tests/BaseClusterIntegrationTest.java | 4 +-
.../tests/BaseClusterIntegrationTestSet.java | 100 +-
.../tests/ChaosMonkeyIntegrationTest.java | 36 +-
.../tests/ClusterIntegrationTestUtils.java | 7 +-
.../pinot/integration/tests/ClusterTest.java | 7 +-
.../ControllerPeriodicTasksIntegrationTest.java | 26 +-
...eTimeFieldSpecHybridClusterIntegrationTest.java | 4 +-
...ridClusterIntegrationTestCommandLineRunner.java | 3 +-
.../IngestionConfigHybridIntegrationTest.java | 31 +-
.../tests/JsonPathClusterIntegrationTest.java | 48 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 3 +-
.../MergeRollupMinionClusterIntegrationTest.java | 27 +-
...onaryAggregationPlanClusterIntegrationTest.java | 6 +-
.../tests/NullHandlingIntegrationTest.java | 4 +-
.../tests/OfflineClusterIntegrationTest.java | 85 +-
...rDownloadLLCRealtimeClusterIntegrationTest.java | 11 +-
.../pinot/integration/tests/QueryGenerator.java | 22 +-
.../tests/RealtimeKinesisIntegrationTest.java | 56 +-
...fflineSegmentsMinionClusterIntegrationTest.java | 35 +-
.../tests/SegmentCompletionIntegrationTest.java | 5 +-
.../tests/ServerStarterIntegrationTest.java | 25 +-
.../tests/SimpleMinionClusterIntegrationTest.java | 2 +-
.../tests/StarTreeClusterIntegrationTest.java | 3 +-
.../tests/ThetaSketchIntegrationTest.java | 9 +-
.../UpsertTableSegmentUploadIntegrationTest.java | 4 +-
pinot-minion/pom.xml | 3 -
.../executor/MinionTaskZkMetadataManager.java | 8 +-
pinot-minion/src/test/resources/log4j2.xml | 2 +-
pinot-perf/pom.xml | 3 -
.../apache/pinot/perf/BenchmarkCombineGroupBy.java | 17 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 4 +-
.../pinot/perf/BenchmarkFixedBitIntReader.java | 5 +-
.../BenchmarkFixedBitSVForwardIndexReader.java | 5 +-
.../perf/BenchmarkFixedIntArrayOffHeapIdMap.java | 6 +-
.../pinot/perf/BenchmarkGeospatialSerde.java | 174 +-
.../apache/pinot/perf/BenchmarkIndexedTable.java | 13 +-
.../BenchmarkNoDictionaryIntegerCompression.java | 47 +-
.../perf/BenchmarkNoDictionaryLongCompression.java | 46 +-
.../BenchmarkNoDictionaryStringCompression.java | 46 +-
...BenchmarkOffheapBitmapInvertedIndexCreator.java | 50 +-
.../apache/pinot/perf/BenchmarkQueryEngine.java | 13 +-
.../org/apache/pinot/perf/DictionaryDumper.java | 2 +
.../pinot/perf/ForwardIndexWriterBenchmark.java | 5 +-
.../org/apache/pinot/perf/RawIndexBenchmark.java | 3 +-
.../pinot/perf/StringDictionaryPerfTest.java | 19 +-
.../batch/common/SegmentGenerationJobUtils.java | 1 +
.../batch/common/SegmentGenerationTaskRunner.java | 16 +-
.../src/test/resources/log4j2.xml | 2 +-
.../pinot-batch-ingestion-hadoop/pom.xml | 4 +-
.../src/test/resources/log4j2.xml | 2 +-
.../spark/SparkSegmentGenerationJobRunner.java | 3 +-
.../spark/SparkSegmentMetadataPushJobRunner.java | 3 +-
.../batch/spark/SparkSegmentTarPushJobRunner.java | 3 +-
.../batch/spark/SparkSegmentUriPushJobRunner.java | 3 +-
.../pinot-batch-ingestion-standalone/pom.xml | 4 +-
.../standalone/SegmentGenerationJobRunnerTest.java | 2 +-
.../src/test/resources/log4j2.xml | 2 +-
.../pinot/hadoop/PinotHadoopJobLauncher.java | 3 +-
.../pinot/hadoop/io/CombineAvroKeyInputFormat.java | 3 +-
.../apache/pinot/hadoop/io/PinotOutputFormat.java | 8 +-
.../apache/pinot/hadoop/io/PinotRecordWriter.java | 7 +-
.../pinot/hadoop/job/HadoopSegmentCreationJob.java | 7 +-
.../hadoop/job/HadoopSegmentPreprocessingJob.java | 3 +-
.../hadoop/job/mappers/SegmentCreationMapper.java | 6 +-
.../hadoop/data/IngestionSchemaValidatorTest.java | 59 +-
.../pinot/hadoop/io/PinotOutputFormatTest.java | 9 +-
.../ingestion/jobs/SegmentPreprocessingJob.java | 8 +-
.../preprocess/DataPreprocessingHelper.java | 6 +-
.../preprocess/DataPreprocessingHelperFactory.java | 6 +-
.../preprocess/OrcDataPreprocessingHelper.java | 5 +-
.../mappers/AvroDataPreprocessingMapper.java | 3 +-
.../mappers/SegmentPreprocessingMapper.java | 12 +-
.../AvroDataPreprocessingPartitioner.java | 9 +-
.../OrcDataPreprocessingPartitioner.java | 13 +-
.../reducers/AvroDataPreprocessingReducer.java | 3 +-
.../ingestion/utils/DataPreprocessingUtils.java | 3 +-
.../apache/pinot/spark/PinotSparkJobLauncher.java | 3 +-
.../spark/jobs/SparkSegmentCreationFunction.java | 51 +-
.../pinot/spark/jobs/SparkSegmentCreationJob.java | 16 +-
.../spark/jobs/SparkSegmentPreprocessingJob.java | 10 +-
.../pinot/spark/jobs/SparkSegmentTarPushJob.java | 13 +-
.../pinot/spark/jobs/SparkSegmentUriPushJob.java | 7 +-
.../preprocess/SparkDataPreprocessingHelper.java | 4 +-
.../utils/PinotSparkJobPreparationHelper.java | 3 +-
.../pinot/spark/SegmentCreationSparkTest.java | 6 +-
.../pinot-environment/pinot-azure/pom.xml | 56 +-
.../plugin/provider/AzureEnvironmentProvider.java | 33 +-
.../provider/AzureEnvironmentProviderTest.java | 27 +-
pinot-plugins/pinot-environment/pom.xml | 2 +-
.../pinot/plugin/filesystem/ADLSGen2PinotFS.java | 55 +-
.../pinot/plugin/filesystem/AzurePinotFS.java | 4 +-
.../filesystem/test/ADLSGen2PinotFSTest.java | 20 +-
.../plugin/filesystem/test/AzurePinotFSTest.java | 3 +-
.../filesystem/test/AzurePinotFSUtilTest.java | 3 +-
.../apache/pinot/plugin/filesystem/GcsPinotFS.java | 6 +-
.../org/apache/pinot/plugin/filesystem/GcsUri.java | 4 +-
.../pinot/plugin/filesystem/TestGcsPinotFS.java | 9 +-
.../pinot/plugin/filesystem/HadoopPinotFS.java | 12 +-
.../pinot-hdfs/src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 3 +-
.../apache/pinot/plugin/filesystem/S3PinotFS.java | 38 +-
.../pinot/plugin/filesystem/S3PinotFSTest.java | 52 +-
pinot-plugins/pinot-file-system/pom.xml | 2 +-
.../avro/AvroIngestionSchemaValidator.java | 36 +-
.../pinot/plugin/inputformat/avro/AvroUtils.java | 67 +-
.../avro/AvroRecordExtractorComplexTypesTest.java | 19 +-
.../inputformat/avro/AvroRecordExtractorTest.java | 20 +-
.../avro/AvroRecordToPinotRowGeneratorTest.java | 6 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 98 +-
.../pinot-avro-base/src/test/resources/log4j2.xml | 2 +-
.../inputformat/avro/KafkaAvroMessageDecoder.java | 19 +-
...aConfluentSchemaRegistryAvroMessageDecoder.java | 3 +-
.../plugin/inputformat/csv/CSVRecordExtractor.java | 6 +-
.../inputformat/csv/CSVRecordExtractorConfig.java | 4 +-
.../plugin/inputformat/csv/CSVRecordReader.java | 20 +
.../inputformat/csv/CSVRecordExtractorTest.java | 3 +-
.../inputformat/csv/CSVRecordReaderTest.java | 79 +-
.../pinot-csv/src/test/resources/log4j2.xml | 2 +-
.../inputformat/json/JSONMessageDecoder.java | 3 +-
.../inputformat/json/JSONRecordExtractorTest.java | 5 +-
.../inputformat/json/JSONRecordReaderTest.java | 6 +-
.../pinot-json/src/test/resources/log4j2.xml | 2 +-
.../plugin/inputformat/orc/ORCRecordReader.java | 21 +-
.../inputformat/orc/ORCRecordExtractorTest.java | 87 +-
.../inputformat/orc/ORCRecordReaderTest.java | 11 +-
.../pinot-orc/src/test/resources/log4j2.xml | 2 +-
.../parquet/ParquetNativeRecordExtractor.java | 13 +-
.../parquet/ParquetNativeRecordReader.java | 11 +-
.../plugin/inputformat/parquet/ParquetUtils.java | 10 +-
.../parquet/ParquetRecordReaderTest.java | 6 +-
.../pinot-parquet/src/test/resources/log4j2.xml | 2 +-
.../protobuf/ProtoBufRecordExtractor.java | 12 +-
.../inputformat/protobuf/ProtoBufRecordReader.java | 3 +-
.../protobuf/ProtoBufRecordExtractorTest.java | 30 +-
.../protobuf/ProtoBufRecordReaderTest.java | 10 +-
.../inputformat/thrift/ThriftRecordReader.java | 3 +-
.../thrift/ThriftRecordExtractorTest.java | 39 +-
.../inputformat/thrift/ThriftRecordReaderTest.java | 9 +-
pinot-plugins/pinot-input-format/pom.xml | 2 +-
.../metrics/yammer/YammerMetricsRegistry.java | 3 +-
.../BaseMultipleSegmentsConversionExecutor.java | 43 +-
.../tasks/BaseSingleSegmentConversionExecutor.java | 35 +-
.../plugin/minion/tasks/BaseTaskExecutor.java | 10 +-
.../pinot/plugin/minion/tasks/MergeTaskUtils.java | 21 +-
.../minion/tasks/SegmentConversionResult.java | 3 +-
.../minion/tasks/SegmentConversionUtils.java | 37 +-
.../ConvertToRawIndexTaskExecutor.java | 9 +-
.../ConvertToRawIndexTaskGenerator.java | 9 +-
.../tasks/mergerollup/MergeRollupTaskExecutor.java | 19 +-
.../mergerollup/MergeRollupTaskGenerator.java | 139 +-
.../minion/tasks/purge/PurgeTaskExecutor.java | 14 +-
.../RealtimeToOfflineSegmentsTaskExecutor.java | 37 +-
.../RealtimeToOfflineSegmentsTaskGenerator.java | 86 +-
.../SegmentGenerationAndPushResult.java | 3 +-
.../SegmentGenerationAndPushTaskExecutor.java | 48 +-
.../SegmentGenerationAndPushTaskGenerator.java | 55 +-
.../plugin/minion/tasks/MergeTaskUtilsTest.java | 25 +-
.../mergerollup/MergeRollupTaskExecutorTest.java | 6 +-
.../mergerollup/MergeRollupTaskGeneratorTest.java | 285 +-
.../minion/tasks/purge/PurgeTaskExecutorTest.java | 4 +-
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 125 +-
...RealtimeToOfflineSegmentsTaskGeneratorTest.java | 131 +-
.../SegmentGenerationAndPushTaskGeneratorTest.java | 9 +-
.../filebased/FileBasedSegmentWriter.java | 37 +-
.../filebased/FileBasedSegmentWriterTest.java | 90 +-
.../stream/kafka09/KafkaLowLevelStreamConfig.java | 4 +-
.../kafka09/KafkaPartitionLevelConsumer.java | 3 +-
.../stream/kafka09/KafkaStreamLevelConsumer.java | 3 +-
.../kafka09/KafkaStreamMetadataProvider.java | 3 +-
.../stream/kafka09/SimpleConsumerFactory.java | 3 +-
.../kafka09/server/KafkaDataServerStartable.java | 6 +-
.../pinot-kafka-0.9/src/test/resources/log4j2.xml | 2 +-
.../kafka20/KafkaPartitionLevelConsumer.java | 3 +-
.../stream/kafka20/KafkaStreamLevelConsumer.java | 3 +-
.../kafka20/KafkaStreamLevelConsumerManager.java | 6 +-
.../kafka20/KafkaStreamLevelStreamConfig.java | 21 +-
.../kafka20/KafkaStreamMetadataProvider.java | 7 +-
.../pinot-kafka-2.0/src/test/resources/log4j2.xml | 2 +-
.../stream/kafka/KafkaJSONMessageDecoder.java | 3 +-
.../pinot-kafka-base/src/test/resources/log4j2.xml | 2 +-
.../pinot-stream-ingestion/pinot-kinesis/pom.xml | 3 +-
.../kinesis/KinesisPartitionGroupOffset.java | 6 +-
.../kinesis/KinesisStreamMetadataProvider.java | 2 +-
.../kinesis/KinesisStreamMetadataProviderTest.java | 7 +-
.../plugin/stream/pulsar/PulsarMessageBatch.java | 6 +-
.../PulsarPartitionLevelConnectionHandler.java | 3 +-
.../pulsar/PulsarPartitionLevelConsumer.java | 6 +-
.../pulsar/PulsarStreamMetadataProvider.java | 3 +-
.../src/test/resources/standalone.properties | 28 +-
pinot-plugins/pom.xml | 5 +-
pinot-segment-local/pom.xml | 3 -
.../segment/local/customobject/QuantileDigest.java | 468 +--
.../local/function/FunctionEvaluatorFactory.java | 18 +-
.../immutable/ImmutableSegmentLoader.java | 36 +-
.../indexsegment/mutable/IntermediateSegment.java | 3 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 15 +-
.../local/io/compression/LZ4Compressor.java | 1 +
.../local/io/compression/LZ4Decompressor.java | 1 +
.../local/io/compression/ZstandardCompressor.java | 3 +-
.../io/compression/ZstandardDecompressor.java | 1 +
.../RealtimeIndexOffHeapMemoryManager.java | 3 +-
.../local/io/util/FixedBitIntReaderWriterV2.java | 5 +-
.../segment/local/io/util/PinotDataBitSetV2.java | 55 +-
.../writer/impl/BaseChunkSVForwardIndexWriter.java | 8 +-
.../writer/impl/FixedBitMVForwardIndexWriter.java | 117 +-
.../impl/FixedByteChunkSVForwardIndexWriter.java | 4 +-
.../writer/impl/MutableOffHeapByteArrayStore.java | 6 +-
.../impl/VarByteChunkSVForwardIndexWriter.java | 7 +-
.../local/loader/LocalSegmentDirectoryLoader.java | 3 +-
.../realtime/impl/RealtimeSegmentStatsHistory.java | 24 +-
.../dictionary/BaseOffHeapMutableDictionary.java | 7 +-
.../forward/FixedByteMVMutableForwardIndex.java | 3 +-
.../RealtimeLuceneDocIdCollector.java | 7 +-
.../RealtimeLuceneIndexReaderRefreshThread.java | 3 +-
.../RealtimeLuceneIndexRefreshState.java | 30 +-
.../local/realtime/impl/json/MutableJsonIndex.java | 3 +-
.../recordtransformer/ComplexTypeTransformer.java | 30 +-
.../recordtransformer/CompositeTransformer.java | 3 +-
.../recordtransformer/ExpressionTransformer.java | 3 +-
.../recordtransformer/NullValueTransformer.java | 3 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 107 +-
.../creator/impl/SegmentCreationDriverFactory.java | 7 +-
.../impl/SegmentIndexCreationDriverImpl.java | 152 +-
.../fwd/SingleValueFixedByteRawIndexCreator.java | 8 +-
.../fwd/SingleValueVarByteRawIndexCreator.java | 8 +-
.../creator/impl/inv/RangeIndexCreator.java | 13 +-
.../impl/nullvalue/NullValueVectorCreator.java | 1 +
.../stats/AbstractColumnStatisticsCollector.java | 68 +-
.../stats/BytesColumnPredIndexStatsCollector.java | 2 +-
.../stats/DoubleColumnPreIndexStatsCollector.java | 4 +-
.../stats/FloatColumnPreIndexStatsCollector.java | 4 +-
.../stats/IntColumnPreIndexStatsCollector.java | 4 +-
.../stats/LongColumnPreIndexStatsCollector.java | 4 +-
.../stats/SegmentPreIndexStatsCollectorImpl.java | 37 +-
.../stats/StringColumnPreIndexStatsCollector.java | 4 +-
.../segment/index/loader/IndexLoadingConfig.java | 3 +-
.../segment/index/loader/SegmentPreProcessor.java | 108 +-
.../ColumnMinMaxValueGenerator.java | 6 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 2 +
.../index/readers/BitmapInvertedIndexReader.java | 12 +-
.../index/readers/LuceneFSTIndexReader.java | 8 +-
.../forward/BaseChunkSVForwardIndexReader.java | 6 +-
.../readers/json/ImmutableJsonIndexReader.java | 4 +-
.../index/readers/text/LuceneDocIdCollector.java | 6 +-
.../local/segment/store/FilePerIndexDirectory.java | 14 +-
.../segment/local/segment/store/IndexEntry.java | 10 +-
.../segment/local/segment/store/IndexKey.java | 16 +-
.../segment/store/SegmentLocalFSDirectory.java | 41 +-
.../segment/store/SingleFileIndexDirectory.java | 149 +-
.../local/segment/store/TextIndexUtils.java | 22 +-
.../VirtualColumnProviderFactory.java | 3 +
.../startree/v2/builder/BaseSingleTreeBuilder.java | 4 +-
.../startree/v2/builder/StarTreeIndexCombiner.java | 6 +-
.../v2/builder/StarTreeV2BuilderConfig.java | 6 +-
.../startree/v2/store/StarTreeDataSource.java | 5 +-
.../startree/v2/store/StarTreeIndexMapUtils.java | 11 +-
.../upsert/PartitionUpsertMetadataManager.java | 165 +-
.../segment/local/upsert/merger/UnionMerger.java | 5 +-
.../apache/pinot/segment/local/utils/CrcUtils.java | 3 +-
.../segment/local/utils/GeometrySerializer.java | 8 +-
.../pinot/segment/local/utils/GeometryType.java | 8 +-
.../pinot/segment/local/utils/GeometryUtils.java | 12 +-
.../pinot/segment/local/utils/IngestionUtils.java | 18 +-
.../segment/local/utils/ReplicationUtils.java | 10 +-
.../pinot/segment/local/utils/SchemaUtils.java | 17 +-
.../segment/local/utils/SegmentPushUtils.java | 6 +-
.../segment/local/utils/TableConfigUtils.java | 19 +-
.../local/utils/fst/PinotBufferIndexInput.java | 94 +-
.../segment/local/utils/fst/RegexpMatcher.java | 37 +-
.../MutableSegmentImplUpsertComparisonColTest.java | 3 +-
.../mutable/MutableSegmentImplUpsertTest.java | 3 +-
.../segment/local/io/util/PinotDataBitSetTest.java | 5 +-
.../impl/RealtimeSegmentStatsHistoryTest.java | 3 +-
.../impl/dictionary/MutableDictionaryTest.java | 6 +-
.../recordtransformer/DataTypeTransformerTest.java | 12 +-
.../ExpressionTransformerTest.java | 39 +-
.../ExpressionTransformerTimeTest.java | 24 +-
.../recordtransformer/RecordTransformerTest.java | 52 +-
.../segment/local/segment/creator/BlocksTest.java | 4 +-
.../local/segment/creator/DictionariesTest.java | 24 +-
.../local/segment/creator/SegmentTestUtils.java | 3 +
.../local/segment/index/ColumnMetadataTest.java | 8 +-
.../segment/local/segment/index/JsonIndexTest.java | 27 +-
.../segment/index/SegmentMetadataImplTest.java | 8 +-
.../SegmentGenerationWithBytesTypeTest.java | 22 +-
.../SegmentGenerationWithFilterRecordsTest.java | 7 +-
.../SegmentGenerationWithNoRecordsTest.java | 21 +-
.../SegmentGenerationWithTimeColumnTest.java | 48 +-
.../index/creator/SegmentPartitionTest.java | 3 +-
.../forward/VarByteChunkSVForwardIndexTest.java | 10 +-
.../BaseDefaultColumnHandlerTest.java | 35 +-
...FixedByteWidthRowColForwardIndexWriterTest.java | 11 +-
.../MultiplePinotSegmentRecordReaderTest.java | 24 +-
.../readers/PinotSegmentRecordReaderTest.java | 22 +-
.../local/segment/readers/PinotSegmentUtil.java | 3 +-
.../readers/RecordReaderSampleDataTest.java | 27 +-
.../store/ColumnIndexDirectoryTestHelper.java | 13 +-
.../segment/store/FilePerIndexDirectoryTest.java | 92 +-
.../segment/local/segment/store/IndexKeyTest.java | 43 +
.../segment/store/SegmentLocalFSDirectoryTest.java | 38 +-
.../store/SingleFileIndexDirectoryTest.java | 212 +-
.../upsert/PartitionUpsertMetadataManagerTest.java | 24 +-
.../upsert/merger/PartialUpsertMergerTest.java | 8 +-
.../segment/local/utils/IngestionUtilsTest.java | 6 +-
.../segment/local/utils/TableConfigUtilsTest.java | 12 +-
.../segment/local/utils/fst/FSTBuilderTest.java | 98 +-
.../src/test/resources/data/log4j2.xml | 2 +-
pinot-segment-spi/pom.xml | 5 +-
.../pinot/segment/spi/AggregationFunctionType.java | 1 +
.../org/apache/pinot/segment/spi/Constants.java | 2 +
.../org/apache/pinot/segment/spi/V1Constants.java | 6 +-
.../segment/spi/compression/ChunkDecompressor.java | 3 +-
.../spi/creator/ColumnIndexCreationInfo.java | 52 +-
.../segment/spi/creator/ColumnStatistics.java | 3 +-
.../spi/creator/SegmentGeneratorConfig.java | 5 +-
.../segment/spi/creator/StatsCollectorConfig.java | 3 +-
.../creator/name/FixedSegmentNameGenerator.java | 2 +-
.../name/InputFileSegmentNameGenerator.java | 89 +
.../name/NormalizedDateSegmentNameGenerator.java | 5 +-
.../creator/name/SegmentNameGeneratorFactory.java | 72 +
.../creator/name/SimpleSegmentNameGenerator.java | 4 +-
.../spi/index/creator/GeoSpatialIndexCreator.java | 2 +-
.../creator/RawValueBasedInvertedIndexCreator.java | 2 -
.../index/creator/SegmentIndexCreationInfo.java | 6 +-
.../segment/spi/index/creator/TextIndexType.java | 4 +-
.../spi/index/reader/H3IndexResolution.java | 3 +-
.../spi/index/reader/MutableForwardIndex.java | 2 -
.../spi/index/reader/SortedIndexReader.java | 3 +-
.../spi/loader/SegmentDirectoryLoaderRegistry.java | 12 +-
.../pinot/segment/spi/memory/CleanerUtil.java | 85 +-
.../pinot/segment/spi/memory/PinotDataBuffer.java | 7 +-
.../spi/partition/MurmurPartitionFunction.java | 2 +
.../spi/partition/PartitionFunctionFactory.java | 3 +-
.../segment/spi/store/ColumnIndexDirectory.java | 6 -
.../pinot/segment/spi/store/ColumnIndexType.java | 6 +-
.../pinot/segment/spi/store/SegmentDirectory.java | 6 -
.../segment/spi/store/SegmentDirectoryPaths.java | 1 -
.../name/InputFileSegmentNameGeneratorTest.java | 64 +
.../NormalizedDateSegmentNameGeneratorTest.java | 18 +-
.../spi/partition/PartitionFunctionTest.java | 17 +-
pinot-server/pom.xml | 3 -
.../server/api/access/AccessControlFactory.java | 2 +-
.../pinot/server/api/resources/DebugResource.java | 3 +-
.../pinot/server/api/resources/ErrorInfo.java | 12 +-
.../pinot/server/api/resources/ErrorResponse.java | 6 +-
.../server/api/resources/HealthCheckResource.java | 5 +-
.../server/api/resources/MmapDebugResource.java | 12 +-
.../api/resources/SegmentMetadataFetcher.java | 3 +
.../server/api/resources/TableSizeResource.java | 15 +-
.../pinot/server/api/resources/TablesResource.java | 82 +-
.../org/apache/pinot/server/conf/ServerConf.java | 10 +-
.../pinot/server/starter/ServerInstance.java | 3 +-
.../starter/ServerQueriesDisabledTracker.java | 10 +-
.../server/starter/helix/AdminApiApplication.java | 40 +-
.../server/starter/helix/BaseServerStarter.java | 6 +-
.../helix/DefaultHelixStarterServerConfig.java | 3 +
.../helix/HelixInstanceDataManagerConfig.java | 3 +-
.../server/starter/helix/HelixServerStarter.java | 6 +-
.../pinot/server/starter/helix/SegmentLocks.java | 3 +
.../apache/pinot/server/api/AccessControlTest.java | 9 +-
.../apache/pinot/server/api/BaseResourceTest.java | 4 +-
pinot-server/src/test/resources/log4j2.xml | 2 +-
.../spi/annotations/metrics/MetricsFactory.java | 3 +-
.../annotations/minion/EventObserverFactory.java | 3 +-
.../annotations/minion/TaskExecutorFactory.java | 3 +-
.../spi/annotations/minion/TaskGenerator.java | 3 +-
.../org/apache/pinot/spi/config/ConfigUtils.java | 4 +-
.../org/apache/pinot/spi/config/TableConfigs.java | 3 +-
.../apache/pinot/spi/config/instance/Instance.java | 10 +-
.../pinot/spi/config/table/BloomFilterConfig.java | 3 +-
.../pinot/spi/config/table/IndexingConfig.java | 18 +-
.../spi/config/table/SegmentPartitionConfig.java | 4 +-
.../spi/config/table/StarTreeIndexConfig.java | 9 +-
.../apache/pinot/spi/config/table/TableConfig.java | 16 +-
.../apache/pinot/spi/config/table/TableStatus.java | 3 +-
.../pinot/spi/config/table/TableTaskConfig.java | 3 +-
.../apache/pinot/spi/config/table/TierConfig.java | 9 +-
.../table/assignment/InstanceAssignmentConfig.java | 12 +-
.../InstanceReplicaGroupPartitionConfig.java | 20 +-
.../table/assignment/InstanceTagPoolConfig.java | 15 +-
.../config/table/ingestion/IngestionConfig.java | 3 +-
.../config/table/ingestion/TransformConfig.java | 3 +-
.../org/apache/pinot/spi/config/tenant/Tenant.java | 5 +-
.../apache/pinot/spi/data/ComplexFieldSpec.java | 2 +-
.../apache/pinot/spi/data/DateTimeFieldSpec.java | 17 +-
.../pinot/spi/data/DateTimeFormatPatternSpec.java | 8 +-
.../apache/pinot/spi/data/DateTimeFormatSpec.java | 19 +-
.../pinot/spi/data/DateTimeFormatUnitSpec.java | 6 +-
.../pinot/spi/data/DateTimeGranularitySpec.java | 11 +-
.../apache/pinot/spi/data/DimensionFieldSpec.java | 8 +-
.../java/org/apache/pinot/spi/data/FieldSpec.java | 13 +-
.../org/apache/pinot/spi/data/MetricFieldSpec.java | 3 +-
.../java/org/apache/pinot/spi/data/Schema.java | 55 +-
.../pinot/spi/data/SchemaValidatorFactory.java | 12 +-
.../org/apache/pinot/spi/data/TimeFieldSpec.java | 8 +-
.../apache/pinot/spi/data/TimeGranularitySpec.java | 18 +-
.../apache/pinot/spi/data/readers/GenericRow.java | 6 +-
.../spi/data/readers/RecordReaderFactory.java | 31 +-
.../pinot/spi/env/CommonsConfigurationUtils.java | 3 +-
.../apache/pinot/spi/env/PinotConfiguration.java | 58 +-
.../PinotEnvironmentProviderFactory.java | 14 +-
.../pinot/spi/ingestion/batch/BatchConfig.java | 13 +-
.../spi/ingestion/batch/IngestionJobLauncher.java | 19 +-
.../batch/spec/SegmentGenerationTaskSpec.java | 7 +
.../apache/pinot/spi/metrics/PinotMetricName.java | 3 +-
.../pinot/spi/metrics/PinotMetricsRegistry.java | 3 +-
.../apache/pinot/spi/plugin/PluginClassLoader.java | 3 +-
.../org/apache/pinot/spi/plugin/PluginManager.java | 91 +-
.../apache/pinot/spi/stream/OffsetCriteria.java | 6 +-
.../stream/PartitionGroupConsumptionStatus.java | 3 +-
.../pinot/spi/stream/PartitionGroupMetadata.java | 3 +-
.../spi/stream/PartitionGroupMetadataFetcher.java | 10 +-
.../pinot/spi/stream/PartitionLevelConsumer.java | 3 +-
.../spi/stream/PartitionLevelStreamConfig.java | 12 +-
.../org/apache/pinot/spi/stream/StreamConfig.java | 82 +-
.../pinot/spi/stream/StreamConsumerFactory.java | 4 +-
.../spi/stream/StreamConsumerFactoryProvider.java | 3 +-
.../pinot/spi/stream/StreamDecoderProvider.java | 3 +-
.../pinot/spi/stream/StreamLevelConsumer.java | 3 +-
.../pinot/spi/stream/StreamMessageDecoder.java | 3 +-
.../pinot/spi/stream/StreamMetadataProvider.java | 14 +-
.../apache/pinot/spi/utils/CommonConstants.java | 74 +-
.../org/apache/pinot/spi/utils/DataSizeUtils.java | 3 +-
.../pinot/spi/utils/IngestionConfigUtils.java | 31 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 51 +-
.../pinot/spi/utils/PinotReflectionUtils.java | 8 +-
.../java/org/apache/pinot/spi/utils/ReadMode.java | 3 +-
.../org/apache/pinot/spi/utils/TimeConverter.java | 8 +-
.../java/org/apache/pinot/spi/utils/TimeUtils.java | 10 +-
.../spi/utils/builder/TableConfigBuilder.java | 9 +-
.../apache/pinot/spi/config/ConfigUtilsTest.java | 40 +-
.../pinot/spi/config/table/IndexingConfigTest.java | 5 +-
.../pinot/spi/data/DateTimeFieldSpecUtilsTest.java | 44 +-
.../spi/data/readers/AbstractRecordReaderTest.java | 18 +-
.../pinot/spi/env/PinotConfigurationTest.java | 3 +-
.../PinotEnvironmentProviderFactoryTest.java | 17 +-
.../pinot/spi/filesystem/LocalPinotFSTest.java | 13 +-
.../pinot/spi/ingestion/batch/BatchConfigTest.java | 9 +-
.../ingestion/batch/IngestionJobLauncherTest.java | 27 +-
.../apache/pinot/spi/plugin/PluginManagerTest.java | 50 +-
.../pinot/spi/utils/GroovyTemplateUtilsTest.java | 16 +-
.../pinot/spi/utils/IngestionConfigUtilsTest.java | 24 +-
.../org/apache/pinot/spi/utils/JsonUtilsTest.java | 94 +-
.../apache/pinot/spi/utils/TimeConverterTest.java | 15 +-
.../spi/utils/builder/TableNameBuilderTest.java | 3 +-
.../pinot/spi/utils/retry/RetryPolicyTest.java | 9 +-
pinot-spi/src/test/resources/TestRecordReader.java | 2 +-
pinot-tools/pom.xml | 3 -
.../org/apache/pinot/tools/AuthQuickstart.java | 3 +-
.../pinot/tools/AutoAddInvertedIndexTool.java | 30 +-
.../apache/pinot/tools/ClusterStateVerifier.java | 6 +-
.../apache/pinot/tools/GitHubEventsQuickstart.java | 3 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 14 +-
.../org/apache/pinot/tools/JoinQuickStart.java | 6 +-
.../apache/pinot/tools/JsonIndexQuickStart.java | 3 +-
.../OfflineComplexTypeHandlingQuickStart.java | 6 +-
.../pinot/tools/PartialUpsertQuickStart.java | 12 +-
.../apache/pinot/tools/PinotNumReplicaChanger.java | 12 +-
.../apache/pinot/tools/PinotTableRebalancer.java | 4 +-
.../org/apache/pinot/tools/PinotToolLauncher.java | 3 +-
.../org/apache/pinot/tools/PinotZKChanger.java | 26 +-
.../org/apache/pinot/tools/QuickStartBase.java | 2 +-
.../java/org/apache/pinot/tools/Quickstart.java | 7 +-
.../apache/pinot/tools/QuickstartTableRequest.java | 28 +-
.../RealtimeComplexTypeHandlingQuickStart.java | 6 +-
.../pinot/tools/RealtimeJsonIndexQuickStart.java | 4 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 6 +-
.../org/apache/pinot/tools/SegmentDumpTool.java | 4 +-
.../java/org/apache/pinot/tools/SpeedTest.java | 68 +-
.../apache/pinot/tools/StarTreeIndexViewer.java | 6 +-
.../org/apache/pinot/tools/UpdateSegmentState.java | 34 +-
.../apache/pinot/tools/ValidateTableRetention.java | 10 +-
.../pinot/tools/admin/PinotAdministrator.java | 9 +-
.../org/apache/pinot/tools/admin/PinotBroker.java | 3 +
.../apache/pinot/tools/admin/PinotController.java | 3 +
.../org/apache/pinot/tools/admin/PinotMinion.java | 3 +
.../org/apache/pinot/tools/admin/PinotServer.java | 3 +
.../tools/admin/PinotServiceManagerStarter.java | 4 +
.../admin/command/AbstractBaseAdminCommand.java | 11 +-
.../tools/admin/command/AddSchemaCommand.java | 3 +-
.../pinot/tools/admin/command/AddTableCommand.java | 12 +-
.../tools/admin/command/AddTenantCommand.java | 15 +-
.../tools/admin/command/AnonymizeDataCommand.java | 108 +-
.../admin/command/AvroSchemaToPinotSchema.java | 12 +-
.../tools/admin/command/BootstrapTableCommand.java | 9 +-
.../admin/command/ChangeNumReplicasCommand.java | 12 +-
.../tools/admin/command/ChangeTableState.java | 3 +-
.../tools/admin/command/CreateSegmentCommand.java | 10 +-
.../tools/admin/command/DeleteClusterCommand.java | 3 +-
.../tools/admin/command/GenerateDataCommand.java | 14 +-
.../command/GitHubEventsQuickStartCommand.java | 1 +
.../tools/admin/command/ImportDataCommand.java | 14 +-
.../tools/admin/command/JsonToPinotSchema.java | 6 +-
.../command/LaunchDataIngestionJobCommand.java | 12 +-
.../tools/admin/command/MoveReplicaGroup.java | 156 +-
.../OfflineSegmentIntervalCheckerCommand.java | 3 +-
.../admin/command/OperateClusterConfigCommand.java | 9 +-
.../tools/admin/command/PostQueryCommand.java | 3 +-
.../tools/admin/command/QuickStartCommand.java | 9 +-
.../command/RealtimeProvisioningHelperCommand.java | 76 +-
.../tools/admin/command/RebalanceTableCommand.java | 70 +-
.../command/SegmentProcessorFrameworkCommand.java | 10 +-
.../admin/command/ShowClusterInfoCommand.java | 95 +-
.../tools/admin/command/StartBrokerCommand.java | 10 +-
.../admin/command/StartControllerCommand.java | 14 +-
.../tools/admin/command/StartKafkaCommand.java | 3 +-
.../tools/admin/command/StartMinionCommand.java | 9 +-
.../tools/admin/command/StartServerCommand.java | 21 +-
.../admin/command/StartServiceManagerCommand.java | 55 +-
.../tools/admin/command/StartZookeeperCommand.java | 5 +-
.../tools/admin/command/StopProcessCommand.java | 3 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 10 +-
.../admin/command/StreamGitHubEventsCommand.java | 13 +-
.../tools/admin/command/UploadSegmentCommand.java | 16 +-
.../tools/admin/command/ValidateConfigCommand.java | 9 +-
.../admin/command/VerifyClusterStateCommand.java | 6 +-
.../tools/admin/command/VerifySegmentState.java | 26 +-
.../anonymizer/ArrayBasedGlobalDictionaries.java | 58 +-
.../pinot/tools/anonymizer/GlobalDictionaries.java | 3 +-
.../anonymizer/MapBasedGlobalDictionaries.java | 30 +-
.../anonymizer/PinotDataAndQueryAnonymizer.java | 190 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 2 +-
.../config/validator/TableConfigValidator.java | 1 +
.../tools/filesystem/PinotFSBenchmarkDriver.java | 35 +-
.../tools/filesystem/PinotFSBenchmarkRunner.java | 30 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 12 +-
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 157 +-
.../pinot/tools/perf/PerfBenchmarkRunner.java | 30 +-
.../org/apache/pinot/tools/perf/QueryRunner.java | 84 +-
.../tools/query/comparison/QueryComparison.java | 4 +-
.../query/comparison/SegmentInfoProvider.java | 3 +
.../tools/query/comparison/StatsGenerator.java | 3 +
.../apache/pinot/tools/scan/query/Aggregation.java | 2 +-
.../tools/scan/query/AggregationFuncFactory.java | 3 +
.../apache/pinot/tools/scan/query/AvgFunction.java | 6 +-
.../pinot/tools/scan/query/CountFunction.java | 6 +-
.../tools/scan/query/DistinctCountFunction.java | 6 +-
.../pinot/tools/scan/query/GroupByOperator.java | 2 +-
.../apache/pinot/tools/scan/query/MaxFunction.java | 4 +-
.../apache/pinot/tools/scan/query/MinFunction.java | 4 +-
.../tools/scan/query/MinMaxRangeFunction.java | 6 +-
.../tools/scan/query/SegmentQueryProcessor.java | 3 +-
.../apache/pinot/tools/scan/query/SumFunction.java | 4 +-
.../org/apache/pinot/tools/scan/query/Utils.java | 2 +
.../converter/PinotSegmentConvertCommand.java | 15 +-
.../segment/converter/SegmentMergeCommand.java | 21 +-
.../pinot/tools/service/PinotServiceManager.java | 6 +-
.../resources/PinotServiceManagerHealthCheck.java | 15 +-
.../PinotServiceManagerInstanceResource.java | 54 +-
.../pinot/tools/streams/AirlineDataStream.java | 79 +-
.../streams/githubevents/GitHubAPICaller.java | 26 +-
.../PullRequestMergedEventsStream.java | 29 +-
.../pinot/tools/utils/KafkaStarterUtils.java | 5 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 41 +-
pinot-tools/src/main/resources/conf/log4j2.xml | 4 +-
.../src/main/resources/conf/pinot-admin-log4j2.xml | 2 +-
.../src/main/resources/conf/pinot-tools-log4j2.xml | 2 +-
.../src/main/resources/conf/quickstart-log4j2.xml | 4 +-
...bEvents_offline_complexTypeHandling_schema.json | 81 +-
...s_offline_complexTypeHandling_table_config.json | 4 +-
.../starbucksStores_offline_table_config.json | 4 +-
.../starbucksStores/starbucksStores_schema.json | 1 -
...eHandling_meetupRsvp_realtime_table_config.json | 4 +-
.../complexTypeHandling_meetupRsvp_schema.json | 67 +-
.../upsert_meetupRsvp_realtime_table_config.json | 4 +-
.../meetupRsvp/upsert_meetupRsvp_schema.json | 4 +-
...t_partial_meetupRsvp_realtime_table_config.json | 6 +-
.../upsert_partial_meetupRsvp_schema.json | 4 +-
pinot-tools/src/main/resources/generator/README.md | 18 +-
.../resources/generator/complexWebsite_config.json | 22 +-
.../generator/complexWebsite_generator.json | 624 +++-
.../resources/generator/simpleWebsite_config.json | 22 +-
.../generator/simpleWebsite_generator.json | 22 +-
pinot-tools/src/main/resources/log4j2.xml | 57 +-
.../pinot/tools/TestCreateSegmentCommand.java | 21 +-
.../pinot/tools/TestDataAndQueryAnonymizer.java | 21 +-
pinot-tools/src/test/resources/log4j2.xml | 2 +-
.../resources/test_data/test_invalid_data.json | 11 +-
pom.xml | 23 +-
1198 files changed, 29313 insertions(+), 14022 deletions(-)
rename pinot-broker/src/test/java/org/apache/pinot/broker/routing/{IntervalST
=> intervalst}/IntervalTest.java (97%)
rename pinot-broker/src/test/java/org/apache/pinot/broker/routing/{IntervalST
=> intervalst}/IntervalTreeTest.java (98%)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/messages/RunPeriodicTaskMessage.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerUserDefinedMessageHandlerFactory.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerPeriodicTaskRestletResource.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/ModeQueriesTest.java
copy pinot-tools/src/main/java/org/apache/pinot/tools/QuickStartBase.java =>
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
(53%)
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/store/IndexKeyTest.java
create mode 100644
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/InputFileSegmentNameGenerator.java
create mode 100644
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGeneratorFactory.java
create mode 100644
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/creator/name/InputFileSegmentNameGeneratorTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
