[
https://issues.apache.org/jira/browse/HADOOP-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098262#comment-18098262
]
ASF GitHub Bot commented on HADOOP-19569:
-----------------------------------------
steveloughran opened a new pull request, #8623:
URL: https://github.com/apache/hadoop/pull/8623
All upload operations are in MultipartIO service, which has been renamed and
move to package org.apache.hadoop.fs.s3a.impl.write to match.
For completeness deletion should also go into this class or an adjacent one
on deletion.
Pulled out multipart IO such that
there are no back references from it to
S3AStore -the final change is to define a store
statistics class which it and other things can
use to update stats.
Executors in hadoop-common to
- pick up shutdown of inner executor and shut themselves down.
- semaphore executor to decrement counters in this process so that queue
state is updated
- semaphored delegating executor unit test in common This stops callers
being able to submit work when the inner executor has shut down.
WriteOperationHelper
* make all calls through its callback interface, rather than given a ref to
S3AFS.
* Move WriteOperationHelper callbacks to S3Store layer,
Multipart IO operations
* move nearly all Multipart IO operationss out of s3afs and into a new
mulitpart service interface and impl
* Multipart service retrieved and invoked as appropriate
* StoreImpl stores a map of ServiceName -> service. with a lookupService()
method in S3AStore interface, it's possible to retrieve services through the
API just by knowing their name and type
* registering all current services this way
StoreImpl to IllegalStateException on method invocation whene the service
isn't running.
Some methods are kept open as they do seem needed.
### How was this patch tested?
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> S3A: stream write/close fails badly once FS is closed
> -----------------------------------------------------
>
> Key: HADOOP-19569
> URL: https://issues.apache.org/jira/browse/HADOOP-19569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> when closing a process during a large upload, and NPE is triggered in the
> abort call. This is because the S3 client has already been released.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$abortMultipartUpload$41(S3AFileSystem.java:5337)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.abortMultipartUpload(S3AFileSystem.java:5336)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$abortMultipartUpload$4(WriteOperationHelper.java:392)
> {code}
> * close() in small writes also fails, just with a different exception
> * and on some large writes, the output stream hangs as it awaits the end of
> the queued writes. This is a problem inside the semaphore executor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
