xiangfu0 opened a new issue, #8718: URL: https://github.com/apache/pinot/issues/8718
The current pinot docker image size is pretty large, 1GB after 0.10.0 release. This is majorly due to the shaded plugins. ``` 144K plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone 144K plugins/pinot-batch-ingestion 87M plugins/pinot-environment/pinot-azure 87M plugins/pinot-environment 17M plugins/pinot-file-system/pinot-adls 14M plugins/pinot-file-system/pinot-gcs 12K plugins/pinot-file-system/pinot-hdfs 29M plugins/pinot-file-system/pinot-s3 60M plugins/pinot-file-system 3.4M plugins/pinot-input-format/pinot-avro 17M plugins/pinot-input-format/pinot-confluent-avro 360K plugins/pinot-input-format/pinot-csv 328K plugins/pinot-input-format/pinot-json 46M plugins/pinot-input-format/pinot-orc 49M plugins/pinot-input-format/pinot-parquet 2.0M plugins/pinot-input-format/pinot-protobuf 2.1M plugins/pinot-input-format/pinot-thrift 119M plugins/pinot-input-format 20M plugins/pinot-metrics/pinot-dropwizard 20M plugins/pinot-metrics/pinot-yammer 40M plugins/pinot-metrics 92K plugins/pinot-minion-tasks/pinot-minion-builtin-tasks 92K plugins/pinot-minion-tasks 16K plugins/pinot-segment-uploader/pinot-segment-uploader-default 16K plugins/pinot-segment-uploader 20K plugins/pinot-segment-writer/pinot-segment-writer-file-based 20K plugins/pinot-segment-writer 19M plugins/pinot-stream-ingestion/pinot-kafka-0.9 32M plugins/pinot-stream-ingestion/pinot-kafka-2.0 14M plugins/pinot-stream-ingestion/pinot-kinesis 60M plugins/pinot-stream-ingestion/pinot-pulsar 124M plugins/pinot-stream-ingestion 428M plugins ``` In order to solve this, there are a few ways: 1. Remove unnecessary shaded jars, e.g. Hadoop ingestion jars. 2. Have the build docker script taking parameters for which plugins to be included. And we can just package the most popular shaded jars. 3. Have a script associated `pinot-admin.sh` to take the env variable of loading plugins then download released shaded jars from the release maven Artifactory before startup. For users: 1. For using jars not packaged, users can load jars when starting pinot. It requires extra env variable to specify the plugins. For plugin developers: 1. If the plugin is not packaged, users can build their customized pinot image by using the existing pinot image as the base image, build and copy the jar to the plugin directory then publish their own jars. cc: @kishoreg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
