[GitHub] [pinot] npawar commented on issue #7262: All table columns have default values

[GitBox]

npawar commented on issue #7262:
URL: https://github.com/apache/pinot/issues/7262#issuecomment-897903962


   @HJK181 , i was able to get this working with your avro schema, avro data, 
and Pinot schema. This does not use the complex config, but instead just 
transform configs for all fields, as @Jackie-Jiang  and I were suggesting.
   
   Here's my schema:
   ```
   {
     "schemaName": "kpis",
     "dimensionFieldSpecs": [
       {
         "name": "userQuery",
         "dataType": "STRING"
       },
       {
         "name": "masterQuery",
         "dataType": "STRING"
       }
     ],
     "metricFieldSpecs": [
       {
         "name": "clicks",
         "dataType": "INT"
       },
       {
         "name": "checkout",
         "dataType": "INT"
       },
       {
         "name": "cart",
         "dataType": "INT"
       }
     ],
     "dateTimeFieldSpecs": [
       {
         "name": "timestamp",
         "dataType": "LONG",
         "format": "1:SECONDS:EPOCH",
         "granularity": "15:MINUTES"
       },
       {
         "name": "mappingTime",
         "dataType": "LONG",
         "format": "1:SECONDS:EPOCH",
         "granularity": "15:MINUTES"
       }
     ]
   }
   ```
   
   Here's my table:
   ```
   {
       "tableName": "kpis_OFFLINE",
       "tableType": "OFFLINE",
       "segmentsConfig": {
         "timeColumnName": "timestamp",
         "replication": "1"
       },
       "tenants": {
         "broker": "DefaultTenant",
         "server": "DefaultTenant"
       },
       "tableIndexConfig": {
         "loadMode": "MMAP"
       },
       "metadata": {},
       "ingestionConfig": {
         "transformConfigs": [
           {
             "columnName": "userQuery",
             "transformFunction": "jsonPathString(query, '$.userQuery')"
           },
           {
             "columnName": "masterQuery",
             "transformFunction": "jsonPathString(query, '$.masterQuery')"
           },
           {
             "columnName": "mappingTime",
             "transformFunction": "jsonPathLong(query, '$.mappingTime')"
           },
           {
             "columnName": "timestamp",
             "transformFunction": "Groovy({mappingTime == -1 ? 31536000 : 
mappingTime}, mappingTime)"
           },
           {
             "columnName": "clicks",
             "transformFunction": "jsonPathDouble(kpis, '$.clicks')"
           },
           {
             "columnName": "checkout",
             "transformFunction": "jsonPathDouble(kpis, '$.checkout')"
           },
           {
             "columnName": "cart",
             "transformFunction": "jsonPathDouble(kpis, '$.cart')"
           }
         ]
       },
       "isDimTable": false
     }
   
   ```
   
   And here's the query result:
   ![Screen Shot 2021-08-12 at 12 17 08 
PM](https://user-images.githubusercontent.com/19693933/129255703-2c6d1a3b-ee42-45ff-bf1a-0aa63387321e.png)
   
   
   I suggest you use transform configs to unblock yourself, while we improve 
the documentation for the complex config. @yupeng9 would it be possible to 
figure out what is the source of confusion/issue with the instructions for 
using the complex config and fix up the documentation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org