[GitHub] [incubator-pinot] fx19880617 closed issue #3998: Upgrade to use Kafka release 2.1.1

2020-05-15 Thread GitBox
fx19880617 closed issue #3998: URL: https://github.com/apache/incubator-pinot/issues/3998 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-pinot] fx19880617 commented on issue #582: Aggregation Functions on Multi-value Columns

2020-05-15 Thread GitBox
fx19880617 commented on issue #582: URL: https://github.com/apache/incubator-pinot/issues/582#issuecomment-629597140 Close as this is already supported This is an automated message from the Apache Git Service. To respond to t

[GitHub] [incubator-pinot] fx19880617 closed issue #582: Aggregation Functions on Multi-value Columns

2020-05-15 Thread GitBox
fx19880617 closed issue #582: URL: https://github.com/apache/incubator-pinot/issues/582 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [incubator-pinot] buchireddy opened a new issue #5402: Rebalancing the segments didn't work when there is only one kafka partition

2020-05-15 Thread GitBox
buchireddy opened a new issue #5402: URL: https://github.com/apache/incubator-pinot/issues/5402 **Scenario** Have a Pinot clustered setup with 4 Pinot servers. The Kafka topic from which Pinot is consuming has only one partition, because of which naturally only one server will be consum

[GitHub] [incubator-pinot] rasin215 opened a new issue #5401: Query Supporting datetime column comparing against currenttime

2020-05-15 Thread GitBox
rasin215 opened a new issue #5401: URL: https://github.com/apache/incubator-pinot/issues/5401 Feature Request. Most of the time you need a report where we need to compare data against current date/time. Currently there is no function which is supporting to compare data against current d

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
haibow commented on a change in pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114511 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java ## @@ -34,6 +34,8 @@ import org.ap

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
haibow commented on a change in pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114511 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java ## @@ -34,6 +34,8 @@ import org.ap

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
haibow commented on a change in pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114553 ## File path: pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluatorFactory.java ## @@ -59,7 +59,8 @@ public FunctionEvalu

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
haibow commented on a change in pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114279 ## File path: pinot-broker/src/test/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManagerTest.java ## @@ -75,85 +75,112 @@ public

[incubator-pinot] branch master updated: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner (#5397)

2020-05-15 Thread haibow
This is an automated email from the ASF dual-hosted git repository. haibow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 5dfefe7 Remove timeFieldSpec getter fr

[GitHub] [incubator-pinot] haibow merged pull request #5397: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner

2020-05-15 Thread GitBox
haibow merged pull request #5397: URL: https://github.com/apache/incubator-pinot/pull/5397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-pinot] haibow commented on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
haibow commented on pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629580638 This diff looks like a milestone for TimeFieldSpec deprecation. @npawar shall we include this in 0.4.0 release? We could probably hold off on the release a bit. cc @kisho

[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
codecov-io edited a comment on pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629578859 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr&el=h1) Report > Merging [#5399](https://codecov.io/gh/apache/incubator-pinot/pull

[GitHub] [incubator-pinot] codecov-io commented on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
codecov-io commented on pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629578859 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr&el=h1) Report > Merging [#5399](https://codecov.io/gh/apache/incubator-pinot/pull/5399?s

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5400: Do not release the PinotDataBuffer when closing the index

2020-05-15 Thread GitBox
Jackie-Jiang opened a new pull request #5400: URL: https://github.com/apache/incubator-pinot/pull/5400 The PinotDataBuffers are tracked and maintained inside SegmentDirectory for ImmutableSegment and PinotDataBufferMemoryManager for MutableSegment. They are created when loading the ind

[GitHub] [incubator-pinot] npawar opened a new pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox
npawar opened a new pull request #5399: URL: https://github.com/apache/incubator-pinot/pull/5399 https://github.com/apache/incubator-pinot/issues/2756 This PR ensures that Pinot can use a DateTimeFieldSpec as a primary time column for the table. After this change, we no longer

[GitHub] [incubator-pinot] akshayrai opened a new pull request #5398: [TE] Added generic support for configuring Data Quality rules like SLA from Yaml

2020-05-15 Thread GitBox
akshayrai opened a new pull request #5398: URL: https://github.com/apache/incubator-pinot/pull/5398 **Overall Updates:** * Added ability to configure Data Quality (DQ) alerts from Yaml. See a sample below. * DQ pipeline leverages & follows the existing detection merging logic pattern

[GitHub] [incubator-pinot] npawar opened a new pull request #5397: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner

2020-05-15 Thread GitBox
npawar opened a new pull request #5397: URL: https://github.com/apache/incubator-pinot/pull/5397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [incubator-pinot] kishoreg commented on pull request #4680: Ensure segment converter not fail for schemas without time fieldSpec

2020-05-15 Thread GitBox
kishoreg commented on pull request #4680: URL: https://github.com/apache/incubator-pinot/pull/4680#issuecomment-629540276 I think it was to make the behavior similar to batch mode. One has a schema that works on offline table and then moves it to real-time and it suddenly doesn't work.

[incubator-pinot] branch master updated: The last of the getTimeFieldSpec calls (#5378)

2020-05-15 Thread nehapawar
This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 430d48e The last of the getTimeFiel

[GitHub] [incubator-pinot] npawar merged pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
npawar merged pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-pinot] npawar commented on pull request #4680: Ensure segment converter not fail for schemas without time fieldSpec

2020-05-15 Thread GitBox
npawar commented on pull request #4680: URL: https://github.com/apache/incubator-pinot/pull/4680#issuecomment-629538360 @fx19880617 i know this was a while ago, but do you remember the context of why this was done? I thought we don't want to support a realtime table without a time column.

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox
mcvsubbu commented on issue #5391: URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629520902 > IMO, servers should not change the segment ZK metadata. It should only be managed by the controller. We have multiple replicas of the same segment on different servers, a

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox
Jackie-Jiang commented on issue #5391: URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629517968 IMO, servers should not change the segment ZK metadata. It should only be managed by the controller. We have multiple replicas of the same segment on different servers,

[GitHub] [incubator-pinot] npawar commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox
npawar commented on issue #5391: URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629515069 Some context - when users add inverted indexes and reload, we provide no indication whether reload was successful or not. This is one of the things that can help. Other optio

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
Jackie-Jiang commented on a change in pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426064442 ## File path: pinot-core/src/test/java/org/apache/pinot/realtime/converter/RealtimeSegmentConverterTest.java ## @@ -46,28 +46,11 @@ public v

[incubator-pinot] branch master updated: Reduce number of servers in ControllerPeriodicTasksIntegrationTest (#5396)

2020-05-15 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new d7808ac Reduce number of servers in C

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox
Jackie-Jiang merged pull request #5396: URL: https://github.com/apache/incubator-pinot/pull/5396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [incubator-pinot] npawar commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
npawar commented on a change in pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426058077 ## File path: pinot-core/src/test/java/org/apache/pinot/realtime/converter/RealtimeSegmentConverterTest.java ## @@ -46,28 +46,11 @@ public void te

[GitHub] [incubator-pinot] npawar commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
npawar commented on a change in pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426057601 ## File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/RealtimeSegmentConverter.java ## @@ -156,20 +156,8 @@ public void

[GitHub] [incubator-pinot] npawar commented on pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
npawar commented on pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378#issuecomment-629497281 > Not related to this PR, but inside the Schema can we do the TimeFieldSpec to DateTimeFieldSpec conversion on the setter side instead of the getter side? > The problem

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox
Jackie-Jiang commented on pull request #5396: URL: https://github.com/apache/incubator-pinot/pull/5396#issuecomment-629473738 > Is this the cause of the failure? No, the LinkedIn internal test failure is caused by port already in use. But this should be able to fix the issue. -

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox
Jackie-Jiang commented on a change in pull request #5378: URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426030144 ## File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/RealtimeSegmentConverter.java ## @@ -156,20 +156,8 @@ public

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox
Jackie-Jiang opened a new pull request #5396: URL: https://github.com/apache/incubator-pinot/pull/5396 Reduce the number of servers started in ControllerPeriodicTasksIntegrationTest to reduce the cost of the test This is an

[GitHub] [incubator-pinot] npawar opened a new pull request #5395: Clean slate on every quickstart run

2020-05-15 Thread GitBox
npawar opened a new pull request #5395: URL: https://github.com/apache/incubator-pinot/pull/5395 Using tmp dir generated within quickstart (which has a timestamp) as the directory for the zk, controller and server, instead of the default. This will ensure we have a clean slate for every qu

[GitHub] [incubator-pinot] reallocf commented on issue #5313: Add date time transform functions

2020-05-15 Thread GitBox
reallocf commented on issue #5313: URL: https://github.com/apache/incubator-pinot/issues/5313#issuecomment-629431050 With https://github.com/apache/incubator-pinot/pull/5326 merging, can we close this? Or should we leave it open for some follow-up work? --

[incubator-pinot] branch master updated: Add toDateTime DateTimeFunction (#5326)

2020-05-15 Thread nehapawar
This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new e682d49 Add toDateTime DateTimeFunc

[GitHub] [incubator-pinot] npawar merged pull request #5326: Add toDateTime DateTimeFunction (#5313)

2020-05-15 Thread GitBox
npawar merged pull request #5326: URL: https://github.com/apache/incubator-pinot/pull/5326 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[incubator-pinot] branch master updated: Include dateTimeFieldSpec in schema columns of Pinot Query Console (#5392)

2020-05-15 Thread nehapawar
This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 4552477 Include dateTimeFieldSpec i

[GitHub] [incubator-pinot] npawar merged pull request #5392: Include dateTimeFieldSpec in schema columns of Pinot Query Console

2020-05-15 Thread GitBox
npawar merged pull request #5392: URL: https://github.com/apache/incubator-pinot/pull/5392 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-pinot] npawar closed pull request #5369: TimeColumnName and schema to all Integration test tables

2020-05-15 Thread GitBox
npawar closed pull request #5369: URL: https://github.com/apache/incubator-pinot/pull/5369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-pinot] akshayrai merged pull request #5386: [TE] Add support for configuring additional custom fields in Jira Alert

2020-05-15 Thread GitBox
akshayrai merged pull request #5386: URL: https://github.com/apache/incubator-pinot/pull/5386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated: [TE] Add support for configuring additional custom fields in Jira Alert (#5386)

2020-05-15 Thread akshayrai09
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 516a168 [TE] Add support for conf

[GitHub] [incubator-pinot] jamesyfshao opened a new pull request #5394: add callback interface for upsert component

2020-05-15 Thread GitBox
jamesyfshao opened a new pull request #5394: URL: https://github.com/apache/incubator-pinot/pull/5394 added list of critical interface that we need to inject to pinot core/server to ensure we can add proper logic for upsert-enabled pinotserver and tables and making sure existing pinot tabl

[GitHub] [incubator-pinot] xiaohui-sun opened a new issue #5393: [TE] Let's put this name in configuration

2020-05-15 Thread GitBox
xiaohui-sun opened a new issue #5393: URL: https://github.com/apache/incubator-pinot/issues/5393 https://github.com/apache/incubator-pinot/blob/a1401de18c10fc07b557a6e5ce0efacd8d486869/thirdeye/thirdeye-frontend/app/pods/home/index/route.js#L81 @harleyjj --