[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845771043 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/access/BasicAuthAccessControlFactory.java: ## @@ -50,10 +51,16 @@ private AccessControl _access

[GitHub] [pinot] kmozaid commented on pull request #8255: Added multi column partitioning for offline table

2022-04-07 Thread GitBox
kmozaid commented on PR #8255: URL: https://github.com/apache/pinot/pull/8255#issuecomment-1092476697 Hi @Jackie-Jiang , Could you please provide review feedback? please also trigger the workflow again. -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845764286 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotUserRestletResource.java: ## @@ -0,0 +1,244 @@ +/** + * Licensed to the Apache Softwa

[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845763208 ## pinot-core/src/main/java/org/apache/pinot/server/access/ZkBasicAuthAccessFactory.java: ## @@ -0,0 +1,130 @@ +/** + * Licensed to the Apache Software Foundation (AS

[GitHub] [pinot] siddharthteotia commented on issue #8493: Add functions for statistical analysis in SQL

2022-04-07 Thread GitBox
siddharthteotia commented on issue #8493: URL: https://github.com/apache/pinot/issues/8493#issuecomment-1092451710 Things like histogram, entropy could also be potentially useful -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [pinot] siddharthteotia commented on issue #8493: Add functions for statistical analysis in SQL

2022-04-07 Thread GitBox
siddharthteotia commented on issue #8493: URL: https://github.com/apache/pinot/issues/8493#issuecomment-1092450806 @jasperjiaguo , I am unable to assign this to you. So self-assigning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [pinot] jasperjiaguo commented on issue #8493: Add functions for statistical analysis in SQL

2022-04-07 Thread GitBox
jasperjiaguo commented on issue #8493: URL: https://github.com/apache/pinot/issues/8493#issuecomment-1092450051 Designing the one request - multiple (sequential) queries model for statistical functions. Planning to use mini-batch stochastic gradient descent for regression algorithms. --

[GitHub] [pinot] jasperjiaguo opened a new issue, #8493: Add functions for statistical analysis in SQL

2022-04-07 Thread GitBox
jasperjiaguo opened a new issue, #8493: URL: https://github.com/apache/pinot/issues/8493 As discussed with @siddharthteotia, consider adding some common statistical analysis methods SQL language. Few examples: 1. Pearson's coefficient 2. Linear regression 3. Logistic regress

[GitHub] [pinot] saurabhd336 commented on pull request #8467: pinot-java-client controller based broker selector

2022-04-07 Thread GitBox
saurabhd336 commented on PR #8467: URL: https://github.com/apache/pinot/pull/8467#issuecomment-1092444164 > There is a controller rest API to directly return the live brokers for a table: `GET /tables/{tableName}/livebrokers` @Jackie-Jiang But we need all the brokers for all tables wh

[pinot] branch helix-1.0 updated (257f406dd3 -> 4aa4dd9d31)

2022-04-07 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch helix-1.0 in repository https://gitbox.apache.org/repos/asf/pinot.git omit 257f406dd3 Upgrade Helix to 1.0.2 add 4aa4dd9d31 Upgrade Helix to 1.0.2 This update added new revisions after un

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8467: pinot-java-client controller based broker selector

2022-04-07 Thread GitBox
saurabhd336 commented on code in PR #8467: URL: https://github.com/apache/pinot/pull/8467#discussion_r845736352 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ControllerURLUtils.java: ## @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundati

[GitHub] [pinot] jackjlli opened a new issue, #8492: Need ways to check whether segment reload/refresh is done on server side

2022-04-07 Thread GitBox
jackjlli opened a new issue, #8492: URL: https://github.com/apache/pinot/issues/8492 Currently segment reload/refresh is done asynchronously. There is no way for Pinot admin to know whether the segment reload/refresh request is fully done on server side. It could be possible that multiple r

[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845705567 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/access/BasicAuthAccessControlFactory.java: ## @@ -50,10 +51,16 @@ private AccessControl _access

[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845705232 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlUtils.java: ## @@ -93,4 +93,17 @@ public void validatePermission(Optional table

[GitHub] [pinot] INNOCENT-BOY commented on a diff in pull request #8314: Access Control management Development

2022-04-07 Thread GitBox
INNOCENT-BOY commented on code in PR #8314: URL: https://github.com/apache/pinot/pull/8314#discussion_r845702999 ## pinot-common/src/main/java/org/apache/pinot/common/utils/config/UserConfigUtils.java: ## @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [pinot] xiangfu0 commented on pull request #8343: Timestamp type index

2022-04-07 Thread GitBox
xiangfu0 commented on PR #8343: URL: https://github.com/apache/pinot/pull/8343#issuecomment-1092347020 > Can we add some validation to the `TableConfigUtils.validate()`? The timestamp index can only be applied to columns with millis granularity (`TIMESTAMP` data type, or `DATE_TIME` field w

[GitHub] [pinot] mayankshriv commented on pull request #8118: Infer SDF for Timestamp in dimension field spec

2022-04-07 Thread GitBox
mayankshriv commented on PR #8118: URL: https://github.com/apache/pinot/pull/8118#issuecomment-1092344581 Thanks @gwenshap for flagging this. Looking at the conversation, it seems that there were comments from another reviewer that still need to be addressed before this can be merged. cc: @

[GitHub] [pinot] npawar commented on a diff in pull request #8455: Make the PR template simpler

2022-04-07 Thread GitBox
npawar commented on code in PR #8455: URL: https://github.com/apache/pinot/pull/8455#discussion_r845664825 ## .github/PULL_REQUEST_TEMPLATE.md: ## @@ -1,30 +1,15 @@ -## Description - -## Upgrade Notes -Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Contro

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8343: Timestamp type index

2022-04-07 Thread GitBox
Jackie-Jiang commented on code in PR #8343: URL: https://github.com/apache/pinot/pull/8343#discussion_r845659983 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java: ## @@ -202,6 +207,33 @@ public SegmentGeneratorConfig(TableConfig

[GitHub] [pinot] Jackie-Jiang commented on issue #8489: Add startOffset and endOffset into segment metadata.properties file

2022-04-07 Thread GitBox
Jackie-Jiang commented on issue #8489: URL: https://github.com/apache/pinot/issues/8489#issuecomment-1092326184 Certain fields should be in ZK metadata only though. Checked the ZK metadata, and adding start/end offset for debugging purpose should be good enough. The missing fields are `segm

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8468: Support Single-valued BigDecimal columns across different SQL statements, and Pinot transforms and operators

2022-04-07 Thread GitBox
Jackie-Jiang commented on code in PR #8468: URL: https://github.com/apache/pinot/pull/8468#discussion_r845649173 ## pinot-core/src/main/java/org/apache/pinot/core/common/BlockValSet.java: ## @@ -84,6 +85,13 @@ */ double[] getDoubleValuesSV(); + /** + * Returns the Bi

[GitHub] [pinot] Jackie-Jiang commented on issue #8477: Airline dataset from the talk about the Star-Tree Index

2022-04-07 Thread GitBox
Jackie-Jiang commented on issue #8477: URL: https://github.com/apache/pinot/issues/8477#issuecomment-1092320445 The data is available here: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HG7NV7 -- This is an automated message from the Apache Git Service. To respo

[GitHub] [pinot] codecov-commenter commented on pull request #8491: Add connection based FailureDetector

2022-04-07 Thread GitBox
codecov-commenter commented on PR #8491: URL: https://github.com/apache/pinot/pull/8491#issuecomment-1092304874 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8491?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on issue #8474: [Feature] allow PinotFSFactory register same scheme with different configurations

2022-04-07 Thread GitBox
Jackie-Jiang commented on issue #8474: URL: https://github.com/apache/pinot/issues/8474#issuecomment-1092291349 We can potentially add an optional id under each scheme to differentiate credentials. When retrieving the `PinotFS`, user can pass the optional id. If no id is provided, we keep t

[GitHub] [pinot] Jackie-Jiang commented on pull request #8466: Implement CONTAINS Operator

2022-04-07 Thread GitBox
Jackie-Jiang commented on PR #8466: URL: https://github.com/apache/pinot/pull/8466#issuecomment-1092287657 @atris The problem of allowing `CONTAINS` to use lucene text index is that we have to use the lucene query syntax for `CONTAINS` which might not be desired. Also it doesn't add much va

[GitHub] [pinot] siddharthteotia commented on issue #8164: Fault domain (Azure) awareness

2022-04-07 Thread GitBox
siddharthteotia commented on issue #8164: URL: https://github.com/apache/pinot/issues/8164#issuecomment-1092283651 The implementation is in progress and is not ready for review https://github.com/apache/pinot/pull/8434 -- This is an automated message from the Apache Git Service. To

[GitHub] [pinot] xiangfu0 commented on pull request #8455: Make the PR template simpler

2022-04-07 Thread GitBox
xiangfu0 commented on PR #8455: URL: https://github.com/apache/pinot/pull/8455#issuecomment-1092282670 As part of the flow, I feel we can also introduce the ci check on the PR title/description and fail automatically if any required field is missing. Example on superset side: https://githu

[GitHub] [pinot] xiangfu0 commented on pull request #8484: adding custome parser

2022-04-07 Thread GitBox
xiangfu0 commented on PR #8484: URL: https://github.com/apache/pinot/pull/8484#issuecomment-1092279112 > Don't we need to write new handlers (java code) for the new grammar ? Yes, I'm on the java part. We plan to add the query syntax to insert raw data files into pinot using t

[GitHub] [pinot] codecov-commenter commented on pull request #8486: Add endpoints for some finer control on minion tasks

2022-04-07 Thread GitBox
codecov-commenter commented on PR #8486: URL: https://github.com/apache/pinot/pull/8486#issuecomment-1092276640 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8486?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #8491: Add connection based FailureDetector

2022-04-07 Thread GitBox
Jackie-Jiang opened a new pull request, #8491: URL: https://github.com/apache/pinot/pull/8491 For issue #8490 ## Description - Add FailureDetector module to the broker - Add QueryResponse interface - Add ConnectionFailureDetector to detect server connection failures ##

[GitHub] [pinot] Jackie-Jiang opened a new issue, #8490: Server Failure Detector

2022-04-07 Thread GitBox
Jackie-Jiang opened a new issue, #8490: URL: https://github.com/apache/pinot/issues/8490 We want to add a `FailureDetector` module on the broker to detect the server failures, and take failed servers out of the routing Design doc: https://docs.google.com/document/d/1X32OMT6lC4pCveQVz

[GitHub] [pinot] klsince commented on a diff in pull request #8486: Add endpoints for some finer control on minion tasks

2022-04-07 Thread GitBox
klsince commented on code in PR #8486: URL: https://github.com/apache/pinot/pull/8486#discussion_r845584439 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java: ## @@ -419,12 +440,17 @@ public SuccessResponse toggleTaskQueueS

[GitHub] [pinot] npawar commented on issue #8489: Add startOffset and endOffset into segment metadata.properties file

2022-04-07 Thread GitBox
npawar commented on issue #8489: URL: https://github.com/apache/pinot/issues/8489#issuecomment-1092212963 cc: @mayankshriv -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [pinot] npawar opened a new issue, #8489: Add startOffset and endOffset into segment metadata.properties file

2022-04-07 Thread GitBox
npawar opened a new issue, #8489: URL: https://github.com/apache/pinot/issues/8489 And any other props that are missing. Goal: every property in segment zk metadata should be in the metadata file on the segment. -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [pinot] npawar opened a new issue, #8488: Add common comparison scalar functions for filterConfig during ingestion

2022-04-07 Thread GitBox
npawar opened a new issue, #8488: URL: https://github.com/apache/pinot/issues/8488 The filter config needs a function that evaluates to boolean https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#filtering So far, we were able to use Groovy for expressing suc

[GitHub] [pinot] xiangfu0 commented on pull request #8465: Add adhoc minion task creation endpoint

2022-04-07 Thread GitBox
xiangfu0 commented on PR #8465: URL: https://github.com/apache/pinot/pull/8465#issuecomment-1092180679 @mcvsubbu can you take a look again ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8486: Add endpoints for some finer control on minion tasks

2022-04-07 Thread GitBox
Jackie-Jiang commented on code in PR #8486: URL: https://github.com/apache/pinot/pull/8486#discussion_r845531899 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java: ## @@ -213,6 +224,16 @@ public StringResultResponse getTask

[GitHub] [pinot] walterddr opened a new issue, #8487: boolean function not recognized in filter clause

2022-04-07 Thread GitBox
walterddr opened a new issue, #8487: URL: https://github.com/apache/pinot/issues/8487 - `SELECT * FROM t WHERE startsWith(stringCol, 'myStr') = true` this query works fine - `SELECT * FROM t WHERE startsWith(stringCol, 'myStr')` throws exception ``` ProcessingException

[GitHub] [pinot] klsince opened a new pull request, #8486: Add endpoints for some finer control on minion tasks

2022-04-07 Thread GitBox
klsince opened a new pull request, #8486: URL: https://github.com/apache/pinot/pull/8486 add endpoints for some finer control on minion tasks: - inspect the states or configs of the subtasks of a task - allow to delete individual tasks w/o deleting the task queue ## Description

[GitHub] [pinot] siddharthteotia commented on pull request #8484: adding custome parser

2022-04-07 Thread GitBox
siddharthteotia commented on PR #8484: URL: https://github.com/apache/pinot/pull/8484#issuecomment-1092158583 Don't we need to write new handlers (java code) for the new grammar ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [pinot] codecov-commenter commented on pull request #8485: Manual tracing

2022-04-07 Thread GitBox
codecov-commenter commented on PR #8485: URL: https://github.com/apache/pinot/pull/8485#issuecomment-1092138816 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8485?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8467: pinot-java-client controller based broker selector

2022-04-07 Thread GitBox
Jackie-Jiang commented on code in PR #8467: URL: https://github.com/apache/pinot/pull/8467#discussion_r845489285 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ControllerURLUtils.java: ## @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundat

[GitHub] [pinot] richardstartin closed pull request #8473: Add fine-grained tracing hooks

2022-04-07 Thread GitBox
richardstartin closed pull request #8473: Add fine-grained tracing hooks URL: https://github.com/apache/pinot/pull/8473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[pinot] branch master updated (52e5a2c3a6 -> 6849441b5b)

2022-04-07 Thread richardstartin
This is an automated email from the ASF dual-hosted git repository. richardstartin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 52e5a2c3a6 make tableNameMap always cache list of tables (#8475) add 6849441b5b add tracing SPI (#8457) N

[GitHub] [pinot] richardstartin merged pull request #8457: add tracing SPI

2022-04-07 Thread GitBox
richardstartin merged PR #8457: URL: https://github.com/apache/pinot/pull/8457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] walterddr commented on a diff in pull request #8484: adding custome parser

2022-04-07 Thread GitBox
walterddr commented on code in PR #8484: URL: https://github.com/apache/pinot/pull/8484#discussion_r845399574 ## pinot-common/src/main/codegen/includes/parserImpls.ftl: ## @@ -0,0 +1,73 @@ +<#-- +// Licensed to the Apache Software Foundation (ASF) under one or more +// contribut

[GitHub] [pinot] npawar closed issue #7966: Disable query-level Groovy function by default

2022-04-07 Thread GitBox
npawar closed issue #7966: Disable query-level Groovy function by default URL: https://github.com/apache/pinot/issues/7966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [pinot] walterddr commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-07 Thread GitBox
walterddr commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r845247951 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,72 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

[GitHub] [pinot] MrNeocore commented on pull request #6719: Add TIMESTAMP and BOOLEAN data type support

2022-04-07 Thread GitBox
MrNeocore commented on PR #6719: URL: https://github.com/apache/pinot/pull/6719#issuecomment-1091804029 What are the upsides to using TIMESTAMP datatype compared to STRING ? (say with format `SIMPLE_DATE_FORMAT:-MM-dd`) -- This is an automated message from the Apache Git Service. To

[GitHub] [pinot] MrNeocore commented on pull request #7916: FILTER Clauses for Aggregates

2022-04-07 Thread GitBox
MrNeocore commented on PR #7916: URL: https://github.com/apache/pinot/pull/7916#issuecomment-1091793936 Is `IN_SUBQUERY` inside FILTER supported ? For example: `SELECT SUM(value) FILTER(WHERE IN_SUBQUERY(entityId, 'SELECT ID_SET(entityId) FROM other_table WHERE cond =

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8467: pinot-java-client controller based broker selector

2022-04-07 Thread GitBox
saurabhd336 commented on code in PR #8467: URL: https://github.com/apache/pinot/pull/8467#discussion_r845180855 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ControllerBasedBrokerSelector.java: ## @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Softwa

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8467: pinot-java-client controller based broker selector

2022-04-07 Thread GitBox
saurabhd336 commented on code in PR #8467: URL: https://github.com/apache/pinot/pull/8467#discussion_r845178107 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/PollingBasedBrokerCache.java: ## @@ -0,0 +1,104 @@ +/** + * Licensed to the Apache Software Fo

[GitHub] [pinot] richardstartin commented on a diff in pull request #8457: add tracing SPI

2022-04-07 Thread GitBox
richardstartin commented on code in PR #8457: URL: https://github.com/apache/pinot/pull/8457#discussion_r844950545 ## pinot-spi/src/main/java/org/apache/pinot/spi/trace/Tracer.java: ## @@ -0,0 +1,35 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [pinot] richardstartin commented on a diff in pull request #8457: add tracing SPI

2022-04-07 Thread GitBox
richardstartin commented on code in PR #8457: URL: https://github.com/apache/pinot/pull/8457#discussion_r844941314 ## pinot-spi/src/main/java/org/apache/pinot/spi/trace/InvocationRecording.java: ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [pinot] richardstartin commented on a diff in pull request #8457: add tracing SPI

2022-04-07 Thread GitBox
richardstartin commented on code in PR #8457: URL: https://github.com/apache/pinot/pull/8457#discussion_r844920295 ## pinot-spi/src/main/java/org/apache/pinot/spi/trace/InvocationRecording.java: ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [pinot] richardstartin commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-07 Thread GitBox
richardstartin commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r844833699 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,72 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag