[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6930: add command to infer pinot schema from json data

[GitBox]

yupeng9 opened a new pull request #6930:
URL: https://github.com/apache/incubator-pinot/pull/6930


   ## Description
   Part of https://github.com/apache/incubator-pinot/issues/6904
   
   Add a command to infer the Pinot schema from the JSON data, with the complex 
type handling.
   
   For example, a run of 
   ```
   JsonToPinotSchema -timeColumnName hoursSinceEpoch -jsonFile  
//tmp/test/test.json -pinotSchemaName json-schema -outputDir /tmp/test 
-unnestFields=payload.commits 
   ```
   
   On the Github event data
   ```
   {
        "id": "7044874109",
        "type": "PushEvent",
        "actor": {
                "id": 18542751,
                "login": "LimeVista",
                "display_login": "LimeVista",
                "gravatar_id": "",
                "url": "https://api.github.com/users/LimeVista";,
                "avatar_url": 
"https://avatars.githubusercontent.com/u/18542751?";
        },
        "repo": {
                "id": 115911530,
                "name": "LimeVista/Tapes",
                "url": "https://api.github.com/repos/LimeVista/Tapes";
        },
        "payload": {
                "push_id": 2226018068,
                "size": 1,
                "distinct_size": 1,
                "ref": "refs/heads/master",
                "head": "c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774",
                "before": "892d872c5d3f24cc6837900c9f4618dc2fe92930",
                "commits": [{
                        "sha": "c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774",
                        "author": {
                                "name": "Lime",
                                "email": 
"4cc153d999e24274955157fc813e6f92f8215...@outlook.com"
                        },
                        "message": "Merge branch 'master' of 
https://github.com/LimeVista/Tapes\\n\\n# Conflicts:\\n#\\t.gitignore",
                        "distinct": true,
                        "url": 
"https://api.github.com/repos/LimeVista/Tapes/commits/c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774";
                }]
        },
        "public": true,
        "created_at": "2018-01-01T11:00:00Z"
   }
   ```
   
   Will output
   ```
   {
     "schemaName" : "json-schema",
     "dimensionFieldSpecs" : [ {
       "name" : "id",
       "dataType" : "STRING"
     }, {
       "name" : "type",
       "dataType" : "STRING"
     }, {
       "name" : "actor.id",
       "dataType" : "INT"
     }, {
       "name" : "actor.login",
       "dataType" : "STRING"
     }, {
       "name" : "actor.display_login",
       "dataType" : "STRING"
     }, {
       "name" : "actor.gravatar_id",
       "dataType" : "STRING"
     }, {
       "name" : "actor.url",
       "dataType" : "STRING"
     }, {
       "name" : "actor.avatar_url",
       "dataType" : "STRING"
     }, {
       "name" : "repo.id",
       "dataType" : "INT"
     }, {
       "name" : "repo.name",
       "dataType" : "STRING"
     }, {
       "name" : "repo.url",
       "dataType" : "STRING"
     }, {
       "name" : "payload.push_id",
       "dataType" : "LONG"
     }, {
       "name" : "payload.size",
       "dataType" : "INT"
     }, {
       "name" : "payload.distinct_size",
       "dataType" : "INT"
     }, {
       "name" : "payload.ref",
       "dataType" : "STRING"
     }, {
       "name" : "payload.head",
       "dataType" : "STRING"
     }, {
       "name" : "payload.before",
       "dataType" : "STRING"
     }, {
       "name" : "payload.commits.sha",
       "dataType" : "STRING"
     }, {
       "name" : "payload.commits.author.name",
       "dataType" : "STRING"
     }, {
       "name" : "payload.commits.author.email",
       "dataType" : "STRING"
     }, {
       "name" : "payload.commits.message",
       "dataType" : "STRING"
     }, {
       "name" : "payload.commits.distinct",
       "dataType" : "BOOLEAN"
     }, {
       "name" : "payload.commits.url",
       "dataType" : "STRING"
     }, {
       "name" : "public",
       "dataType" : "BOOLEAN"
     }, {
       "name" : "created_at",
       "dataType" : "STRING"
     } ]
   }%
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org