npawar commented on pull request #6354: URL: https://github.com/apache/incubator-pinot/pull/6354#issuecomment-749177669
> some thoughts re API design. I understand that this one is mainly for testing purposes so it probably doesn't matter too much. > > If this is to ever become a production feature for "quickly uploading some data" it may be worthwhile to avoid the "json blob" as a URL fragment in the upload methods. > > This could be achieved by either (a) using a JSON payload with the batch configs, as well as a base64-encoded representation of the data, or (b) split out the multipart-upload and launching the ingestion job with "batch configs" as two separate invocations. > > Both have merit: > (a) the JSON one is straight forward and makes this a clean API at the cost of expanding the size of the uploaded data via encoding. If the data is tiny anyways, this would be my suggested approach. > (b) if we want to operate the controller like a small temp file server, then might as well have the ability to upload a multipart file. Then, separately, call an API to invoke the ingestion "locally" with a JSON payload that has the configs and a pointer to the uploaded file Thanks for the thoughts Alex. Yes the json blob as a query param is certainly ugly. But this is strictly going to be used for testing purposes, and unllikely that it be operated in a productionized setup. For production, we will always expect users to use the deep store for dropping files and a minion task picking them up periodically/on trigger. Added a comment on the methods and in the swagger APIs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
