[GitHub] [incubator-pinot] kishoreg commented on pull request #5670: Pinot S3Fs fix

kishoreg commented on pull request #5670: URL: https://github.com/apache/incubator-pinot/pull/5670#issuecomment-655937575 LGTM! @Jackie-Jiang @mayankshriv do you see any issue in upgrading the httpclient version? This is an

[incubator-pinot] branch rewrite-non-groupby-to-distinct updated (3a2e1fd -> 74f8567)

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch rewrite-non-groupby-to-distinct in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 3a2e1fd Rewrite non-aggregate group by query to distinct query add 74f8567

[incubator-pinot] branch rewrite-non-groupby-to-distinct updated (b1c65e7 -> 3a2e1fd)

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch rewrite-non-groupby-to-distinct in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard b1c65e7 Rewrite non-aggregate group by query to distinct query add 3a2e1fd

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

Jackie-Jiang commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655900425 @pradeepgv42 I added an optimization for the first case (no filter) in #5672 , can you please try it out and see how it works?

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5672: Add SelectionQuerySegmentPruner to prune segments for simple selection queries

Jackie-Jiang opened a new pull request #5672: URL: https://github.com/apache/incubator-pinot/pull/5672 - For selection query with LIMIT 0, keep 1 segment to create the data schema - For selection only query without filter, keep enough documents to fulfill the LIMIT requirement - For s

[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5671: Rewrite non-aggregate group by query to distinct query

fx19880617 opened a new pull request #5671: URL: https://github.com/apache/incubator-pinot/pull/5671 ## Description Per https://github.com/apache/incubator-pinot/issues/5663, this PR rewrite non-aggregation groupBy query to distinct query. e.g. `SELECT col1+col2*5 FROM foo GROUP B

[incubator-pinot] branch rewrite-non-groupby-to-distinct created (now b1c65e7)

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch rewrite-non-groupby-to-distinct in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at b1c65e7 Rewrite non-aggregate group by query to distinct query This branch inclu

[incubator-pinot] 01/01: Rewrite non-aggregate group by query to distinct query

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch rewrite-non-groupby-to-distinct in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit b1c65e7746af7e310bf733127be5e84db407fdcd Author: Xiang Fu AuthorDate: Wed Jul 8 21:29

[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5669: Fixing code to fetch the fsConfig from the right parent

kishoreg commented on a change in pull request #5669: URL: https://github.com/apache/incubator-pinot/pull/5669#discussion_r451954273 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFSFactory.java ## @@ -60,19 +60,19 @@ public static void register(Str

[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5669: Fixing code to fetch the fsConfig from the right parent

kishoreg commented on a change in pull request #5669: URL: https://github.com/apache/incubator-pinot/pull/5669#discussion_r451954329 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFSFactory.java ## @@ -60,19 +60,19 @@ public static void register(Str

[incubator-pinot] branch pinot-fs-bug-fix updated (d9745c9 -> c703924)

This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch pinot-fs-bug-fix in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d9745c9 Adding test case add c703924 addressing comments No new revisions were added by

[GitHub] [incubator-pinot] pradeepgv42 opened a new pull request #5670: Pinot S3Fs fix

pradeepgv42 opened a new pull request #5670: URL: https://github.com/apache/incubator-pinot/pull/5670 ## Description Fixing the S3 issue from https://github.com/apache/incubator-pinot/issues/5616 ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade or

[GitHub] [incubator-pinot] kishoreg closed issue #5245: Add pinot-s3 filesystem plugin

kishoreg closed issue #5245: URL: https://github.com/apache/incubator-pinot/issues/5245 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [incubator-pinot] kishoreg closed issue #81: Server directly pull data from hdfs/s3

kishoreg closed issue #81: URL: https://github.com/apache/incubator-pinot/issues/81 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [incubator-pinot] pradeepgv42 commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

pradeepgv42 commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655868047 Thanks Jackie, just to add more information. I am looking at mainly following use-cases: * Get the most recent elements without any filter (eg: for displaying la

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

Jackie-Jiang commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655845398 @pradeepgv42 Actually, after a second thought, I think for this query (order-by time & no filter), we can do some segment level optimization to further reduce the s

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

Jackie-Jiang commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655842051 @pradeepgv42 We need to scan 2 columns, one is the order-by column (unavoidable for ordering purpose), the other is the virtual docId column (no actual scan as ever

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5667: Support text index on dictionary encoded columns

Jackie-Jiang commented on a change in pull request #5667: URL: https://github.com/apache/incubator-pinot/pull/5667#discussion_r451904412 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java ## @@ -379,6 +370,13 @@ p

[GitHub] [incubator-pinot] pradeepgv42 edited a comment on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

pradeepgv42 edited a comment on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655838063 select * from tablename order by timestampCol desc limit 10 Much better than previously but still taking ~30sec. For example, this query is still sca

[GitHub] [incubator-pinot] pradeepgv42 commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

pradeepgv42 commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-655838063 select * from tablename order by timestamp desc limit 10 Much better than previously but still taking ~30sec "numServersQueried": 2, "numServe

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5666: TEXT_MATCH support for dictionary encoded columns

Jackie-Jiang commented on issue #5666: URL: https://github.com/apache/incubator-pinot/issues/5666#issuecomment-655832500 We should also consider supporting standalone text index (without forward index). This could potentially save a lot of spaces.

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5669: Fixing code to fetch the fsConfig from the right parent

Jackie-Jiang commented on a change in pull request #5669: URL: https://github.com/apache/incubator-pinot/pull/5669#discussion_r451902803 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFSFactory.java ## @@ -60,19 +60,19 @@ public static void register

[GitHub] [incubator-pinot] mayankshriv commented on a change in pull request #5669: Fixing code to fetch the fsConfig from the right parent

mayankshriv commented on a change in pull request #5669: URL: https://github.com/apache/incubator-pinot/pull/5669#discussion_r451902715 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFSFactory.java ## @@ -60,19 +60,19 @@ public static void register(

[GitHub] [incubator-pinot] kishoreg opened a new pull request #5669: Fixing code to fetch the fsConfig from the right parent

kishoreg opened a new pull request #5669: URL: https://github.com/apache/incubator-pinot/pull/5669 ## Description We were initializing the pinot fs class with incorrect configuration. Before: > Got scheme s3, initializing class org.apache.pinot.plugin.filesystem.LocalF

[incubator-pinot] branch pinot-fs-bug-fix updated: Adding test case

This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a commit to branch pinot-fs-bug-fix in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/pinot-fs-bug-fix by this push: new d9745c9 Adding t

[incubator-pinot] branch pinot-fs-bug-fix updated: Fixing code to fetch the fsConfig from the right parent

This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a commit to branch pinot-fs-bug-fix in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/pinot-fs-bug-fix by this push: new 4c3eed8 Fixing c

[incubator-pinot] branch pinot-fs-bug-fix created (now 2ec7dee)

This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch pinot-fs-bug-fix in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 2ec7dee Optimize selection order-by when not all selected expressions are ordered (#5661) No

[GitHub] [incubator-pinot] siddharthteotia opened a new pull request #5668: Type

siddharthteotia opened a new pull request #5668: URL: https://github.com/apache/incubator-pinot/pull/5668 Remove ForwardIndexType and InvertedIndexType. These enums are not used anywhere in the persistent structures and are simply used in ColumnMetadata only in the constructor and don't

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5667: Support text index on dictionary encoded columns

siddharthteotia commented on a change in pull request #5667: URL: https://github.com/apache/incubator-pinot/pull/5667#discussion_r451884394 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/text/LuceneTextIndexCreator.java ## @@ -115,14 +115

[GitHub] [incubator-pinot] Jackie-Jiang closed issue #5577: Deprecate BrokerRequest in query engine

Jackie-Jiang closed issue #5577: URL: https://github.com/apache/incubator-pinot/issues/5577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5599: Add Streaming Server Endpoint

Jackie-Jiang commented on a change in pull request #5599: URL: https://github.com/apache/incubator-pinot/pull/5599#discussion_r451746775 ## File path: pom.xml ## @@ -1140,10 +1160,38 @@ reflections ${reflections.version} + + +com.go

[GitHub] [incubator-pinot] kishoreg closed pull request #5632: Fix realtime segment download url

kishoreg closed pull request #5632: URL: https://github.com/apache/incubator-pinot/pull/5632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] kishoreg commented on pull request #5632: Fix realtime segment download url

kishoreg commented on pull request #5632: URL: https://github.com/apache/incubator-pinot/pull/5632#issuecomment-655676032 closing since this is already handled in another PR. This is an automated message from the Apache Git S

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5667: Support co-existence of text index with other indexes

Jackie-Jiang commented on a change in pull request #5667: URL: https://github.com/apache/incubator-pinot/pull/5667#discussion_r451711537 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/text/LuceneTextIndexCreator.java ## @@ -115,14 +115,31

[GitHub] [incubator-pinot] kishoreg commented on pull request #5661: Optimize selection order-by when not all selected expressions are ordered

kishoreg commented on pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661#issuecomment-66444 All the tests passed, only thing that failed was uploading coverage report. will merging it. T

[GitHub] [incubator-pinot] kishoreg merged pull request #5661: Optimize selection order-by when not all selected expressions are ordered

kishoreg merged pull request #5661: URL: https://github.com/apache/incubator-pinot/pull/5661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated: Optimize selection order-by when not all selected expressions are ordered (#5661)

This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 2ec7dee Optimize selection order-by

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5667: Support co-existence of text index with other indexes

siddharthteotia commented on a change in pull request #5667: URL: https://github.com/apache/incubator-pinot/pull/5667#discussion_r451354267 ## File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataSource.java ## @@ -60,6 +60,13 @@ @Nullable InvertedIndexRea

[GitHub] [incubator-pinot] siddharthteotia opened a new pull request #5667: Support co-existence of text index with other indexes

siddharthteotia opened a new pull request #5667: URL: https://github.com/apache/incubator-pinot/pull/5667 Support co-existence of text index with other indexes and structures - Remove the constraint that text index should be on raw columns - Remove the constraint that text index sh