[
https://issues.apache.org/jira/browse/HADOOP-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331292#comment-17331292
]
Steve Loughran commented on HADOOP-17112:
-----------------------------------------
I can replicate this :)
{code}
[ERROR]
testBulkCommitFiles(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)
Time elapsed: 2.85 s <<< ERROR!
org.apache.hadoop.fs.s3a.commit.PathCommitException: `null': Failed to commit
upload against unknown destination, described in null:
java.lang.IllegalStateException: Cannot parse URI
s3a://stevel-london/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir/file3
with space: Cannot parse URI
s3a://stevel-london/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir/file3
with space
at
org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:208)
at
org.apache.hadoop.fs.s3a.commit.CommitOperations.commitOrFail(CommitOperations.java:168)
at
org.apache.hadoop.fs.s3a.commit.CommitOperations.access$100(CommitOperations.java:84)
at
org.apache.hadoop.fs.s3a.commit.CommitOperations$CommitContext.commitOrFail(CommitOperations.java:670)
at
org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testBulkCommitFiles(ITestCommitOperations.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot parse URI
s3a://stevel-london/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir/file3
with space
at
org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.destinationPath(SinglePendingCommit.java:270)
at
org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.validate(SinglePendingCommit.java:210)
at
org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:191)
... 19 more
{code}
> whitespace not allowed in paths when saving files to s3a via committer
> ----------------------------------------------------------------------
>
> Key: HADOOP-17112
> URL: https://issues.apache.org/jira/browse/HADOOP-17112
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Krzysztof Adamski
> Priority: Blocker
> Attachments: image-2020-07-03-16-08-52-340.png
>
>
> When saving results through spark dataframe on latest 3.0.1-snapshot compiled
> against hadoop-3.2 with the following specs
> --conf
> spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> --conf
> spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
>
> --conf
> spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
>
> --conf spark.hadoop.fs.s3a.committer.name=partitioned
> --conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
> we are unable to save the file with whitespace character in the path. It
> works fine without.
> I was looking into the recent commits with regards to qualifying the path,
> but couldn't find anything obvious. Is this a known bug?
> When saving results through spark dataframe on latest 3.0.1-snapshot compiled
> against hadoop-3.2 with the following specs
> --conf
> spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> --conf
> spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
>
> --conf
> spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
>
> --conf spark.hadoop.fs.s3a.committer.name=partitioned
> --conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
> we are unable to save the file with whitespace character in the path. It
> works fine without.
> I was looking into the recent commits with regards to qualifying the path,
> but couldn't find anything obvious. Is this a known bug?
> !image-2020-07-03-16-08-52-340.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
