This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch pinot-mock-data-deterministic-reloaded in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 5ccff42 Second attempt: We enhance the pinot-tools GenerateData command to produce pseudo-random (deterministic) time-series for templates and support user-specified table names in bin/generator.sh. add d54b04a Deep extraction in Avro and Json RecordExtractor (#5492) add 3443593 Enhance and simplify the filtering (#5444) add d72493e Update SegmentDumpTool to use PinotSegmentRecordReader. (#5505) add 04e12bd Config for raw index writer version (#5503) add bea55a0 [TE] Add data quality/sla rule to the detection template (#5515) add 2e834cf Support escaping single quote for SQL literal (#5501) add c139488 Support expression as the left-hand side for BETWEEN and IN clause (#5502) add 957b28d Moved StreamPartitionMsgOffset to be an interface (#5506) add 2b09676 One to many records from record reader/decoder (#5430) add 15fda7f Update Pinot quickstart to use sql mode (#5530) add 5b3837c Adding github actions for pinot tests (#5512) add a1a2434 Remove BlockValIterator and directly read values from BlockValSet with docId (#5510) add 5390698 Fix the failure caused by Reflections in FunctionRegistry (#5531) add 2d28c0f Enhance quickstart script for CI tests (#5508) add bbc3228 Update license and notice for 0.4.0 release (#5529) add b383373 Adding Support for SQL CASE Statement (#5461) add 4f4c587 Fixing grep thirdeye changes in github action scripts (#5532) add 66af506 Second attempt: We enhance the pinot-tools GenerateData command to produce pseudo-random (deterministic) time-series for templates and support user-specified table names in bin/generator.sh. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (5ccff42) \ N -- N -- N refs/heads/pinot-mock-data-deterministic-reloaded (66af506) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/pinot_tests.yml | 78 +++ .github/workflows/scripts/.pinot_quickstart.sh | 172 +++++++ .github/workflows/scripts/.pinot_test.sh | 51 ++ .travis.yml | 3 +- .travis/.travis_quickstart.sh | 84 ++-- LICENSE-binary | 155 ++++-- NOTICE-binary | 53 ++- licenses-binary/LICENSE-gpl-2.0.txt | 125 +++++ .../broker/broker/helix/HelixBrokerStarter.java | 8 +- .../pinot/common/function/FunctionRegistry.java | 94 ++-- .../pinot/common/function/JsonFunctions.java | 9 + .../common/function/TransformFunctionType.java | 8 + .../protocols/SegmentCompletionProtocol.java | 38 +- .../pinot/common/utils/request/RequestUtils.java | 7 +- .../parsers/pql2/ast/BetweenPredicateAstNode.java | 8 +- .../pql/parsers/pql2/ast/InPredicateAstNode.java | 2 + .../apache/pinot/sql/parsers/CalciteSqlParser.java | 66 ++- .../pinot/sql/parsers/CalciteSqlCompilerTest.java | 71 ++- .../apache/pinot/controller/ControllerStarter.java | 4 + .../resources/LLCSegmentCompletionHandlers.java | 15 +- .../realtime/PinotLLCRealtimeSegmentManager.java | 22 +- .../core/realtime/SegmentCompletionManager.java | 83 ++-- .../segment/CommittingSegmentDescriptor.java | 10 +- .../api/SegmentCompletionProtocolDeserTest.java | 21 +- .../PinotLLCRealtimeSegmentManagerTest.java | 20 +- .../helix/core/realtime/SegmentCompletionTest.java | 341 ++++++++------ .../segment/FlushThresholdUpdaterTest.java | 3 +- .../apache/pinot/core/common/BaseBlockValSet.java | 67 ++- .../pinot/core/common/BlockDocIdIterator.java | 22 +- .../apache/pinot/core/common/BlockDocIdSet.java | 10 +- .../org/apache/pinot/core/common/BlockValSet.java | 160 ++++--- .../org/apache/pinot/core/common/DataFetcher.java | 60 ++- .../data/function/InbuiltFunctionEvaluator.java | 8 +- .../realtime/HLRealtimeSegmentDataManager.java | 3 +- .../realtime/LLRealtimeSegmentDataManager.java | 96 ++-- .../realtime/SegmentBuildTimeLeaseExtender.java | 2 +- .../writer/impl/v1/BaseChunkSingleValueWriter.java | 8 +- .../impl/v1/FixedByteChunkSingleValueWriter.java | 10 +- .../impl/v1/VarByteChunkSingleValueWriter.java | 9 +- .../pinot/core/minion/RawIndexConverter.java | 14 +- .../core/operator/blocks/EmptyFilterBlock.java | 4 +- .../core/operator/blocks/MultiValueBlock.java | 5 +- .../core/operator/blocks/SingleValueBlock.java | 5 +- .../operator/dociditerators/AndDocIdIterator.java | 123 ++--- .../dociditerators/ArrayBasedDocIdIterator.java | 37 +- ...Iterator.java => BitmapBasedDocIdIterator.java} | 9 +- .../dociditerators/BitmapDocIdIterator.java | 79 +--- ...kDocIdIterator.java => EmptyDocIdIterator.java} | 16 +- .../ExpressionScanDocIdIterator.java | 72 +-- .../dociditerators/MVScanDocIdIterator.java | 142 ++---- ...cIdIterator.java => MatchAllDocIdIterator.java} | 33 +- .../operator/dociditerators/OrDocIdIterator.java | 102 ++-- .../RangelessBitmapDocIdIterator.java | 53 +-- .../dociditerators/SVScanDocIdIterator.java | 207 +++----- .../dociditerators/ScanBasedDocIdIterator.java | 22 +- .../dociditerators/SortedDocIdIterator.java | 99 ++-- .../core/operator/docidsets/AndBlockDocIdSet.java | 181 ------- .../pinot/core/operator/docidsets/AndDocIdSet.java | 156 ++++++ .../operator/docidsets/ArrayBasedDocIdSet.java | 21 +- .../core/operator/docidsets/BitmapDocIdSet.java | 70 +-- ...FilterBlockDocIdSet.java => EmptyDocIdSet.java} | 38 +- .../docidsets/ExpressionFilterDocIdSet.java | 51 +- .../operator/docidsets/FilterBlockDocIdSet.java | 49 +- ...{SizeBasedDocIdSet.java => MVScanDocIdSet.java} | 44 +- ...canBasedDocIdSet.java => MatchAllDocIdSet.java} | 18 +- .../core/operator/docidsets/OrBlockDocIdSet.java | 126 ----- .../pinot/core/operator/docidsets/OrDocIdSet.java | 120 +++++ ...ArrayBasedDocIdSet.java => SVScanDocIdSet.java} | 29 +- .../docidsets/ScanBasedMultiValueDocIdSet.java | 85 ---- .../docidsets/ScanBasedSingleValueDocIdSet.java | 88 ---- .../core/operator/docidsets/SortedDocIdSet.java | 74 +-- .../DictionaryBasedMultiValueIterator.java | 108 ----- .../DictionaryBasedSingleValueIterator.java | 88 ---- .../docvaliterators/MultiValueIterator.java | 84 ---- .../docvaliterators/SingleValueIterator.java | 84 ---- .../core/operator/docvalsets/MultiValueSet.java | 22 +- .../core/operator/docvalsets/SingleValueSet.java | 142 +++--- .../core/operator/filter/AndFilterOperator.java | 19 +- .../operator/filter/BitmapBasedFilterOperator.java | 79 ++-- .../core/operator/filter/FilterOperatorUtils.java | 45 +- .../pinot/core/operator/filter/IntRanges.java | 84 ---- .../operator/filter/MatchAllFilterOperator.java | 10 +- .../core/operator/filter/OrFilterOperator.java | 23 +- .../filter/RangeIndexBasedFilterOperator.java | 63 ++- .../operator/filter/ScanBasedFilterOperator.java | 53 +-- .../filter/SortedIndexBasedFilterOperator.java | 138 ++++++ .../SortedInvertedIndexBasedFilterOperator.java | 179 ------- .../operator/filter/TextMatchFilterOperator.java | 40 +- .../BaseDictionaryBasedPredicateEvaluator.java | 8 + .../predicate/EqualsPredicateEvaluatorFactory.java | 38 +- .../predicate/InPredicateEvaluatorFactory.java | 38 +- .../NotEqualsPredicateEvaluatorFactory.java | 38 +- .../predicate/NotInPredicateEvaluatorFactory.java | 38 +- .../filter/predicate/PredicateEvaluator.java | 8 +- .../predicate/PredicateEvaluatorProvider.java | 6 - .../predicate/RangePredicateEvaluatorFactory.java | 30 ++ .../RegexpLikePredicateEvaluatorFactory.java | 11 +- .../TextMatchPredicateEvaluatorFactory.java | 58 --- .../function/BinaryOperatorTransformFunction.java | 334 +++++++++++++ .../transform/function/CaseTransformFunction.java | 450 ++++++++++++++++++ .../function/EqualsTransformFunction.java | 53 +++ .../GreaterThanOrEqualTransformFunction.java | 61 +++ .../function/GreaterThanTransformFunction.java | 61 +++ .../function/LessThanOrEqualTransformFunction.java | 61 +++ .../function/LessThanTransformFunction.java | 61 +++ .../function/LiteralTransformFunction.java | 59 ++- .../function/NotEqualsTransformFunction.java | 53 +++ .../function/TransformFunctionFactory.java | 49 +- .../org/apache/pinot/core/plan/FilterPlanNode.java | 51 +- .../RecordReaderSegmentCreationDataSource.java | 22 +- .../creator/impl/SegmentColumnarIndexCreator.java | 43 +- .../impl/SegmentIndexCreationDriverImpl.java | 38 +- .../fwd/SingleValueFixedByteRawIndexCreator.java | 20 +- .../fwd/SingleValueVarByteRawIndexCreator.java | 29 +- .../defaultcolumn/BaseDefaultColumnHandler.java | 3 +- .../startree/operator/StarTreeFilterOperator.java | 9 +- .../org/apache/pinot/core/util/ArrayCopyUtils.java | 5 +- .../core/common/RealtimeNoDictionaryTest.java | 26 +- .../core/common/docidsets/BitmapDocIdSetTest.java | 78 --- .../core/common/docidsets/SortedDocIdSetTest.java | 136 ------ .../function/InbuiltFunctionEvaluatorTest.java | 42 +- ...valuatorTest.java => InbuiltFunctionsTest.java} | 56 ++- .../realtime/LLRealtimeSegmentDataManagerTest.java | 117 +++-- .../mutable/MutableSegmentImplTest.java | 76 +-- .../dociditerators/AndDocIdIteratorTest.java | 55 +++ .../dociditerators/BitmapDocIdIteratorTest.java | 61 +-- .../dociditerators/OrDocIdIteratorTest.java | 58 +++ .../dociditerators/SortedDocIdIteratorTest.java | 85 ++++ .../RealtimeSingleValueIteratorTest.java | 167 ------- .../operator/filter/AndFilterOperatorTest.java | 2 +- .../pinot/core/operator/filter/IntRangesTest.java | 103 ---- .../core/operator/filter/OrFilterOperatorTest.java | 8 +- .../core/operator/filter/TestFilterOperator.java | 57 +-- .../function/BaseTransformFunctionTest.java | 15 + .../BinaryOperatorTransformFunctionTest.java | 109 +++++ .../function/CaseTransformFunctionTest.java | 522 +++++++++++++++++++++ .../function/EqualsTransformFunctionTest.java} | 36 +- .../GreaterThanOrEqualTransformFunctionTest.java} | 36 +- .../GreaterThanTransformFunctionTest.java} | 36 +- .../LessThanOrEqualTransformFunctionTest.java} | 36 +- .../function/LessThanTransformFunctionTest.java} | 36 +- .../function/LiteralTransformFunctionTest.java | 41 ++ .../function/NotEqualsTransformFunctionTest.java} | 36 +- .../impl/fakestream/FakeStreamConsumerFactory.java | 2 +- .../core/realtime/stream/StreamConfigTest.java | 2 + .../SegmentGenerationWithMultipleRecordsKey.java | 111 +++++ .../pinot/core/startree/v2/BaseStarTreeV2Test.java | 100 ++-- .../FixedByteChunkSingleValueReaderWriteTest.java | 225 ++++++--- .../VarByteChunkSingleValueReaderWriteTest.java | 49 +- .../src/test/resources/data/test_data-mv.avro | Bin 9583323 -> 7700192 bytes .../tests/OfflineClusterIntegrationTest.java | 143 ++++-- .../tests/SegmentCompletionIntegrationTest.java | 3 +- .../pinot/perf/BenchmarkOrDocIdIterator.java | 44 +- .../inputformat/avro/AvroRecordExtractor.java | 3 +- .../pinot/plugin/inputformat/avro/AvroUtils.java | 67 ++- .../avro/AvroRecordExtractorComplexTypesTest.java | 209 +++++++++ .../avro/AvroRecordExtractorMapTypeTest.java | 115 ----- .../inputformat/json/JSONRecordExtractor.java | 54 +-- ...xtractor.java => JSONRecordExtractorUtils.java} | 90 ++-- .../json/JSONRecordExtractorUtilsTest.java | 120 +++++ .../kafka20/KafkaPartitionLevelConsumerTest.java | 2 +- .../pinot/server/starter/ServerInstance.java | 3 + .../src/test/resources/data/test_data-mv.avro | Bin 9583323 -> 7700192 bytes .../apache/pinot/spi/config/table/FieldConfig.java | 13 +- .../apache/pinot/spi/data/readers/GenericRow.java | 3 + .../pinot/spi/data/readers/RecordExtractor.java | 3 + .../org/apache/pinot/spi/stream/LongMsgOffset.java | 41 +- .../pinot/spi/stream/LongMsgOffsetFactory.java | 27 +- .../pinot/spi/stream/PartitionCountFetcher.java | 2 +- .../pinot/spi/stream/PartitionLevelConsumer.java | 6 + .../pinot/spi/stream/PartitionOffsetFetcher.java | 2 +- .../org/apache/pinot/spi/stream/StreamConfig.java | 12 + .../pinot/spi/stream/StreamConfigProperties.java | 1 + .../spi/stream/StreamConsumerFactoryProvider.java | 19 +- .../pinot/spi/stream/StreamPartitionMsgOffset.java | 43 +- .../stream/StreamPartitionMsgOffsetFactory.java | 47 +- .../data/readers/AbstractRecordExtractorTest.java | 47 +- .../org/apache/pinot/tools/HybridQuickstart.java | 65 ++- .../java/org/apache/pinot/tools/Quickstart.java | 51 +- .../org/apache/pinot/tools/RealtimeQuickStart.java | 43 +- .../org/apache/pinot/tools/SegmentDumpTool.java | 122 ++--- .../apache/pinot/tools/StarTreeIndexViewer.java | 9 - .../tools/admin/command/QuickstartRunner.java | 4 +- .../apache/pinot/tools/scan/query/Projection.java | 19 +- .../tools/scan/query/SegmentQueryProcessor.java | 57 ++- .../converter/DictionaryToRawIndexConverter.java | 48 +- thirdeye/thirdeye-frontend/app/utils/yaml-tools.js | 38 +- 187 files changed, 6555 insertions(+), 4713 deletions(-) create mode 100644 .github/workflows/pinot_tests.yml create mode 100755 .github/workflows/scripts/.pinot_quickstart.sh create mode 100755 .github/workflows/scripts/.pinot_test.sh create mode 100644 licenses-binary/LICENSE-gpl-2.0.txt rename pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/{IndexBasedDocIdIterator.java => BitmapBasedDocIdIterator.java} (73%) copy pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/{EmptyBlockDocIdIterator.java => EmptyDocIdIterator.java} (72%) rename pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/{SizeBasedDocIdIterator.java => MatchAllDocIdIterator.java} (69%) delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/AndBlockDocIdSet.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/AndDocIdSet.java rename pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/{EmptyFilterBlockDocIdSet.java => EmptyDocIdSet.java} (57%) rename pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/{SizeBasedDocIdSet.java => MVScanDocIdSet.java} (52%) rename pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/{ScanBasedDocIdSet.java => MatchAllDocIdSet.java} (67%) delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/OrBlockDocIdSet.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/OrDocIdSet.java copy pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/{ArrayBasedDocIdSet.java => SVScanDocIdSet.java} (56%) delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/ScanBasedMultiValueDocIdSet.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/ScanBasedSingleValueDocIdSet.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docvaliterators/DictionaryBasedMultiValueIterator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docvaliterators/DictionaryBasedSingleValueIterator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docvaliterators/MultiValueIterator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/docvaliterators/SingleValueIterator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/IntRanges.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedIndexBasedFilterOperator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedInvertedIndexBasedFilterOperator.java delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/TextMatchPredicateEvaluatorFactory.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CaseTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/EqualsTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GreaterThanOrEqualTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GreaterThanTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LessThanOrEqualTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LessThanTransformFunction.java create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/NotEqualsTransformFunction.java delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/common/docidsets/BitmapDocIdSetTest.java delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/common/docidsets/SortedDocIdSetTest.java rename pinot-core/src/test/java/org/apache/pinot/core/data/function/{DateTimeFunctionEvaluatorTest.java => InbuiltFunctionsTest.java} (77%) create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/dociditerators/AndDocIdIteratorTest.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/dociditerators/OrDocIdIteratorTest.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/dociditerators/SortedDocIdIteratorTest.java delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/docvaliterators/RealtimeSingleValueIteratorTest.java delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/filter/IntRangesTest.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunctionTest.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/CaseTransformFunctionTest.java copy pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/EqualsTransformFunctionTest.java} (53%) copy pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/GreaterThanOrEqualTransformFunctionTest.java} (52%) copy pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/GreaterThanTransformFunctionTest.java} (53%) copy pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/LessThanOrEqualTransformFunctionTest.java} (53%) copy pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/LessThanTransformFunctionTest.java} (53%) create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunctionTest.java rename pinot-core/src/{main/java/org/apache/pinot/core/operator/dociditerators/EmptyBlockDocIdIterator.java => test/java/org/apache/pinot/core/operator/transform/function/NotEqualsTransformFunctionTest.java} (53%) create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/segment/index/creator/SegmentGenerationWithMultipleRecordsKey.java create mode 100644 pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordExtractorComplexTypesTest.java delete mode 100644 pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordExtractorMapTypeTest.java copy pinot-plugins/pinot-input-format/pinot-json/src/main/java/org/apache/pinot/plugin/inputformat/json/{JSONRecordExtractor.java => JSONRecordExtractorUtils.java} (51%) create mode 100644 pinot-plugins/pinot-input-format/pinot-json/src/test/java/org/apache/pinot/plugin/inputformat/json/JSONRecordExtractorUtilsTest.java rename pinot-core/src/main/java/org/apache/pinot/core/common/BlockMultiValIterator.java => pinot-spi/src/main/java/org/apache/pinot/spi/stream/LongMsgOffset.java (54%) rename pinot-core/src/main/java/org/apache/pinot/core/common/BlockValIterator.java => pinot-spi/src/main/java/org/apache/pinot/spi/stream/LongMsgOffsetFactory.java (55%) rename pinot-core/src/main/java/org/apache/pinot/core/common/BlockSingleValIterator.java => pinot-spi/src/main/java/org/apache/pinot/spi/stream/StreamPartitionMsgOffsetFactory.java (50%) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org