[
https://issues.apache.org/jira/browse/HADOOP-15712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602212#comment-16602212
]
Steve Loughran commented on HADOOP-15712:
-----------------------------------------
Related: ABFS: failing output stream close() in can confuse try-with-resources.
{code}
[ERROR]
testSequentialReadAfterReverseSeekPerformance(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead)
Time elapsed: 472.61 s <<< ERROR!
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.close(NativeAzureFileSystem.java:1046)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead.createTestFile(ITestAzureBlobFileSystemRandomRead.java:557)
at
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead.assumeHugeFileExists(ITestAzureBlobFileSystemRandomRead.java:566)
at
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead.testSequentialReadAfterReverseSeekPerformance(ITestAzureBlobFileSystemRandomRead.java:387)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Operation could not be completed within the
specified time. Please see the cause for further information.
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:462)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:47)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:406)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:403)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: Operation could not be
completed within the specified time.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:315)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:185)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1097)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1069)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:456)
... 9 more
{code}
What I think is happening is that in a try-with-resources clause, close() is
called explicitly, which fails. And when the implicit clause also causes
close(), *the same exception is raised*.
{code}
try (FSDataOutputStream outputStream = fs.create(TEST_FILE_PATH)) {
int bytesWritten = 0;
while (bytesWritten < TEST_FILE_SIZE) {
outputStream.write(buffer);
bytesWritten += buffer.length;
}
LOG.info("Closing stream {}", outputStream);
ContractTestUtils.NanoTimer closeTimer
= new ContractTestUtils.NanoTimer();
outputStream.close(); /* HERE */
closeTimer.end("time to close() output stream");
}
{code}
try-with-resources catches the second exception and then adds it to the first
exception as a suppressed exception, which it can't as it is the same
exception. So in the attempt to not hide the first exception with the second
exception, t-w-r is doing exactly that . oops
> ABFS to increase output stream close more robustly
> --------------------------------------------------
>
> Key: HADOOP-15712
> URL: https://issues.apache.org/jira/browse/HADOOP-15712
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Reporter: Steve Loughran
> Priority: Minor
>
> if {{BlobOutputStream.close()}} raises an exception, then
> {{NativeAzureFsOutputStream.close()}} doesn't set it's {{this.out}} field to
> null, so close() can still be re-invoked, this time with an error.
> the {{out.close()}} needs to be moved into the try/finally clause
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
