[
https://issues.apache.org/jira/browse/HADOOP-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13459:
-----------------------------------
Attachment: HADOOP-13459.001.patch
This patch refactors the tests to introduce a new abstract class:
{{AbstractWasbTestBase}}. The vast majority of test suites can subclass this
to get just the setup and teardown logic without accidentally re-running test
cases.
Here is some information on running time for the impacted test suites before
and after the patch.
*Before:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 325.178 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 316.563 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 392.583 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.985 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 300.001 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.564 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.994 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
*After:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.088 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.198 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.855 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.6 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 307.217 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.915 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
> hadoop-aws runs several test cases repeatedly, causing unnecessarily long
> running time.
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-13459
> URL: https://issues.apache.org/jira/browse/HADOOP-13459
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13459.001.patch
>
>
> Within hadoop-azure, we have the {{NativeAzureFileSystemBaseTest}} abstract
> class, which defines setup and teardown to handle the Azure storage account
> and also defines multiple test cases. This class originally was contributed
> to provide a layer of indirection for running the same test cases in live
> mode or mock mode: {{TestNativeAzureFileSystemLive}} and
> {{TestNativeAzureFileSystemMocked}}. It appears that since then, we created
> multiple new test suites that subclassed {{NativeAzureFileSystemBaseTest}}
> for the benefit of getting the common setup and teardown code, but also with
> the side effect of running the inherited test cases repeatedly. This is a
> significant factor in the overall execution time of the hadoop-azure tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
