xy720 opened a new pull request #4163:
URL: https://github.com/apache/incubator-doris/pull/4163
## Proposed changes
Please see the main description in issue #4101
### Resume
When users use spark load, they have to upload the dependent jars to hdfs
every time.
This cl will add a self-generated repository under working_dir folder in
hdfs for saving dependecies of spark dpp programe and spark platform.
Note that, the dependcies we upload to repository include:
1、`spark-dpp.jar`
2、`spark2x.zip`
1 is the dpp library which built with spark-dpp submodule. See details about
spark-dpp submodule in pr #4146 .
2 is the spark2.x.x platform library which contains all jars in
$SPARK_HOME/jars
**The repository structure** will be like this:
```
__spark_repository__/
|-__archive_1_0_0/
| |-__lib_990325d2c0d1d5e45bf675e54e44fb16_spark-dpp.jar
| |-__lib_7670c29daf535efe3c9b923f778f61fc_spark-2x.zip
|-__archive_2_2_0/
| |-__lib_64d5696f99c379af2bee28c1c84271d5_spark-dpp.jar
| |-__lib_1bbb74bb6b264a270bc7fca3e964160f_spark-2x.zip
|-__archive_3_2_0/
| |-...
```
The followinng conditions will force fe to upload dependencies:
1、When fe find its dppVersion is absent in repository.
2、The MD5 value of remote file does not match the local file.
Before Fe uploads the dependencies, it will create an archive directory with
name `__archive_{dppVersion}` under the repository.
## Types of changes
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
- [x] I have create an issue on [Doris's
issues](https://github.com/apache/incubator-doris/issues), and have described
the bug/feature there in detail
- [x] Commit messages in my PR start with the related issues ID, like "#4071
Add pull request template to doris project"
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [ ] If this change need a document change, I have updated the document
- [ ] Any dependent changes have been merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
