[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573672#comment-16573672
]
Thomas Marquardt edited comment on HADOOP-15546 at 8/8/18 6:30 PM:
-------------------------------------------------------------------
I will commit this shortly.
+1 from me. All tests pass with fs.azure.scale.test.enabled set to true for
both WASB and ABFS.
*Tests run: 257, Failures: 0, Errors: 0, Skipped: 11*
*Tests run: 1, Failures: 0, Errors: 0, Skipped: 0*
*Tests run: 769, Failures: 0, Errors: 0, Skipped: 183*
*Tests run: 186, Failures: 0, Errors: 0, Skipped: 10*
We will address the Yetus issue later.
We will address the issues around config in a separate JIRA. For now, the
recommendation is to use separate files (azure-auth-keys.xml and
azure-bfs-auth-keys.xml) for WASB and ABFS. The contents are as shown in the
example below:
*azure-auth-keys.xml:*
<property>
<name>fs.contract.test.fs.wasb</name>
<value>wasb://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value>
<description>The name of the azure file system for testing.</description>
</property>
<property>
<name>fs.azure.test.account.name</name>
<value>*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.scale.test.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.azure.secure.mode</name>
<value>false</value>
</property>
*azure-bfs-auth-keys.xml*:
<property>
<name>fs.contract.test.fs.abfs</name>
<value>abfs://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value>
<description>The name of the azure file system for testing.</description>
</property>
<property>
<name>fs.azure.test.account.name</name>
<value>*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.scale.test.enabled</name>
<value>true</value>
</property>
was (Author: tmarquardt):
I will commit this shortly.
+1 from me. All tests pass with fs.azure.scale.test.enabled set to true for
both WASB and ABFS.
*Tests run: 257, Failures: 0, Errors: 0, Skipped: 11*
*Tests run: 1, Failures: 0, Errors: 0, Skipped: 0*
*Tests run: 769, Failures: 0, Errors: 0, Skipped: 183*
*Tests run: 186, Failures: 0, Errors: 0, Skipped: 10*
We will address the Yetus issue later.
We will address the issues around config in a separate JIRA. For now, the
recommendation is to use separate files (azure-auth-keys.xml and
azure-bfs-auth-keys.xml) for WASB and ABFS. The contents are as shown in the
example below:
*azure-auth-keys.xml:*
<property>
<name>fs.contract.test.fs.wasb</name>
<value>wasb://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value>
<description>The name of the azure file system for testing.</description>
</property>
<property>
<name>fs.azure.test.account.name</name>
<value>*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.scale.test.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.azure.secure.mode</name>
<value>false</value>
</property>
*azure-bfs-auth-keys.xml*:
<property>
<name>fs.contract.test.fs.abfs</name>
<value>abfs://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value>
<description>The name of the azure file system for testing.</description>
</property>
<property>
<name>fs.azure.test.account.name</name>
<value>*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name>
<value>*{YOUR_ACCOUNT_KEY}*</value>
</property>
<property>
<name>fs.azure.scale.test.enabled</name>
<value>true</value>
</property>
> ABFS: tune imports & javadocs; stabilise tests
> ----------------------------------------------
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: HADOOP-15407
> Reporter: Steve Loughran
> Assignee: Thomas Marquardt
> Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch,
> HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
