[
https://issues.apache.org/jira/browse/HDFS-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068570#comment-18068570
]
Pavan Pothnis commented on HDFS-17898:
--------------------------------------
Thanks for the details, I did try with *.blob. endpoint in place of .dfs.
endpoint.*
*Configuration settings*
*configMap.put("fs.defaultFS", "abfss://" +
connection.getContainerClient().getBlobContainerName() + "@" +
connection.getAccountName() + ".blob."+endpointSuffix+"/");*
**
if(authenticationType.equals(AzureStorageAuthorization.SHARED_KEY_AUTHORIZATION.getValue()))
{
String accountKey = "fs.azure.account.key." + connection.getAccountName() +
".blob." + endpointSuffix;
configMap.put(accountKey, connection.getAccountKey());
}
if(authenticationType.equals(AzureStorageAuthorization.SHARED_ACCESS_SIGNATURES.getValue()))
{
// fs.azure.sas.fixed.token is read directly by the ABFS driver when
auth.type=SAS
configMap.put("fs.azure.account.auth.type." + connection.getAccountName() +
".blob." + endpointSuffix, "SAS");
configMap.put("fs.azure.sas.fixed.token." + connection.getAccountName() +
".blob." + endpointSuffix, connection.getSharedAccessSignature());
}
And I get the following exception
2026-03-26 02:18:21 - INFO :getMetadataContext without Hadoop dist (new
version) for file
abfss://[email protected]/alldatatype.avro
Failed to initialize filesystem
abfss://[email protected]/alldatatype.avro:
Invalid *configuration value detected for "fs.defaultFS". Blob Endpoint Url
Cannot be used to initialize filesystem for HNS Account*
*java.lang.RuntimeException: Invalid configuration value detected for
"fs.defaultFS". Blob Endpoint Url Cannot be used to initialize filesystem for
HNS Account 2026-03-26 02:18:21* - ERROR:java.lang.RuntimeException: Invalid
configuration value detected for "fs.defaultFS". Blob Endpoint Url Cannot be
used to initialize filesystem for HNS Account
at
com.informatica.adapter.parser.hadoopImpl.fs.FilePathImpl.getFileSystem(FilePathImpl.java:63)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.parser.hadoopImpl.fs.FilePathImpl.getFileSystem(FilePathImpl.java:63)
at
com.informatica.adapter.parser.hadoopApi.HadoopFactory.registerHadoopFileSystem(HadoopFactory.java:128)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.parser.hadoopApi.HadoopFactory.registerHadoopFileSystem(HadoopFactory.java:128)
at
com.informatica.adapter.parser.filesystem.metadataimpl.MetadataContextLoaderImpl.loadContext(MetadataContextLoaderImpl.java:49)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.parser.filesystem.metadataimpl.MetadataContextLoaderImpl.loadContext(MetadataContextLoaderImpl.java:49)
at
com.informatica.adapter.utils.metadata.MetadataUtil.getMetadataContext(MetadataUtil.java:270)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.utils.metadata.MetadataUtil.getMetadataContext(MetadataUtil.java:270)
at
com.informatica.adapter.utils.metadata.MetadataUtil.getSchema(MetadataUtil.java:282)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.utils.metadata.MetadataUtil.getSchema(MetadataUtil.java:282)
at
com.informatica.adapter.azureblob.metadata.adapter.AzureBlobMetadataAdapter.populateObjectDetails(AzureBlobMetadataAdapter.java:423)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.azureblob.metadata.adapter.AzureBlobMetadataAdapter.populateObjectDetails(AzureBlobMetadataAdapter.java:423)
at
com.informatica.adapter.sdkadapter.metadata.semantic.consumer.SemanticMetadataAdapter.getObjectDetails(SemanticMetadataAdapter.java:318)2026-03-26
02:18:21 - ERROR: at
com.informatica.adapter.sdkadapter.metadata.semantic.consumer.SemanticMetadataAdapter.getObjectDetails(SemanticMetadataAdapter.java:318)
at
com.informatica.ctf.product.cctf.helper.MetadataHelper.yetAnotherCatalogBuilder(MetadataHelper.java:333)2026-03-26
02:18:21 - ERROR: at
com.informatica.ctf.product.cctf.helper.MetadataHelper.yetAnotherCatalogBuilder(MetadataHelper.java:333)
at
com.informatica.ctf.product.cctf.helper.MetadataHelper.yetAnotherCatalogBuilder(MetadataHelper.java:436)2026-03-26
02:18:21 - ERROR: at
com.informatica.ctf.product.cctf.helper.MetadataHelper.yetAnotherCatalogBuilder(MetadataHelper.java:436)
at
com.informatica.ctf.product.cctf.helper.MetadataHelper.getMetadataForAll(MetadataHelper.java:576)2026-03-26
02:18:21 - ERROR: at
com.informatica.ctf.product.cctf.helper.MetadataHelper.getMetadataForAll(MetadataHelper.java:576)
at
com.informatica.ctf.product.cctf.impl.CCTFCTFMetadataBrowse.getCatalogForObjects(CCTFCTFMetadataBrowse.java:443)2026-03-26
02:18:21 - ERROR: at
com.informatica.ctf.product.cctf.impl.CCTFCTFMetadataBrowse.getCatalogForObjects(CCTFCTFMetadataBrowse.java:443)
I have verified from Azure Portal that Hierarchical Name Space is disabled
,Here is the screenshot from the Portal for the account being used.
!image-2026-03-26-14-53-53-821.png!
I was able to fix by adding the following configuration
_*fs.azure.account.hns.enabled = false*_
_*Could you please confirm if this is by design and clients applications are
expected to set this explicitly, even if the account is FNS ?*_
> ABFS migration fails on FNS blob storage with Soft Delete enabled
> ------------------------------------------------------------------
>
> Key: HDFS-17898
> URL: https://issues.apache.org/jira/browse/HDFS-17898
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.4.2, 3.4.3
> Environment: *Environment*
> Hadoop version: _*3.4.2 and 3.4.3*_
> Storage SDK - Tried with both *6.x* and *12.X*
> Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
> Storage account features enabled: FNS and Soft Delete
> Migration path: *WASB → ABFS (using abfss:// scheme)*
> Reporter: Pavan Pothnis
> Priority: Blocker
> Attachments: image-2026-03-26-14-53-53-821.png
>
>
> When migrating from WASB to ABFS on a FlatNameSpace (FNS) Azure Blob Storage
> account that has both FNS and Soft Delete features enabled, the migration
> fails with an incompatibility error.
> *Environment*
> Hadoop version: _*3.4.2 and 3.4.3*_
> Storage SDK - Tried with both *6.x* and *12.X*
> Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
> Storage account features enabled: FNS and Soft Delete
> Migration path: *WASB → ABFS (using abfss:// scheme)*
> *Issue Description*
> Following the WASB to ABFS migration guide (as documented in wasb.md), users
> are instructed to simply change URIs from WASB scheme to ABFS scheme.
> However, when the underlying Azure Blob Storage account has Soft Delete
> enabled, the ABFS driver encounters a critical error that prevents data
> access.
> *Steps to Reproduce*
> Create an Azure Blob Storage account (GPv2) with FlatNameSpace enabled
> Enable Soft Delete feature on the blob storage account
> Upgrade Hadoop from WASB to ABFS (3.4.2 )
> Change URIs from wasb:// to abfss://
> Attempt to read/parse Avro files or any data from the storage
> *Error Message*
> java.lang.RuntimeException: java.lang.RuntimeException: The file or partition
> directory
> [{*}abfss{*}://{*}azureblobv3-cctf{*}@{*}adapterqablob{*}.{*}dfs{*}.core.windows.net/alldatatype.avro]
> is not valid. The parser encountered the following error while parsing the
> content: [org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
> "This endpoint does not support BlobStorageEvents or SoftDelete. Please
> disable these account features if you would like to use this endpoint.", 409,
> HEAD,
> https://adapterqablob.{*}dfs{*}.core.windows.net/azureblobv3-cctf/alldatatype.avro?upn=false&action=getStatus&timeout=90s,
> rId: 0b404e0b-301f-00a7-4c18-bc227f000000]. Select a valid [Avro] file or
> partition directory.
> Changes done as part of WABS to ABFS Migration
> *From -*
>
> String fileName = "wasbs://" +
> connection.getContainerClient().getBlobContainerName() + "@" +
> connection.getAccountName() + ".blob." +connection.getEndpointSuffix()+ "/";
>
> *To -*
>
> String fileName = "abfss://" +
> connection.getContainerClient().getBlobContainerName() + "@" +
> connection.getAccountName() + ".dfs." +connection.getEndpointSuffix()+ "/";
> *From -*
>
> String wasbFileSystemUrl = "wasbs://" + containerName + "@" +
> connection.getAccountName() +".blob."+connection.getEndpointSuffix()+"/";
>
> *To -*
>
> String wasbFileSystemUrl = "abfss://" + containerName + "@" +
> connection.getAccountName() +".dfs."+connection.getEndpointSuffix()+"/";
>
> *Configuration - Shared Key Authentication*
>
> if(authenticationType.equals(AzureStorageAuthorization.SHARED_KEY_AUTHORIZATION.getValue()))
> {
> *From -*
> String accountKey = "fs.azure.account.key." + connection.getAccountName() +
> ".blob." + endpointSuffix;
> *To -*
> **
> String accountKey = "fs.azure.account.key." + connection.getAccountName() +
> ".dfs." + endpointSuffix;
>
> configMap.put(accountKey, connection.getAccountKey());
>
> }
> *Configuration - Shared Access Signature*
>
> if(authenticationType.equals(AzureStorageAuthorization.SHARED_ACCESS_SIGNATURES.getValue()))
> {
> *From -*
> String containerName = StringUtils.isNotBlank(containerNameOverride) ?
> containerNameOverride : connection.getContainer().getName();
> String sharedaccesssignature = "fs.azure.sas." + containerName +"." +
> connection.getAccountName() + ".blob."+ endpointSuffix;
> configMap.put(sharedaccesssignature, connection.getSharedAccessSignature());
> *To -*
> configMap.put("fs.azure.account.auth.type." + connection.getAccountName() +
> ".dfs." + endpointSuffix, "SAS");
> configMap.put("fs.azure.sas.fixed.token." + connection.getAccountName() +
> ".dfs." + endpointSuffix, connection.getSharedAccessSignature());
>
> }
> All the changes made in To section works fine as long as the Blob account is
> not configured with Soft Delete / Event and fails when enabled, causing out
> of box failure for all account with those settings and these are extensively
> used settings.
>
> *Current Workaround*
> The issue only resolves after manually disabling Soft Delete on the Azure
> Blob Storage account. This workaround defeats the purpose of having this
> protective feature enabled and is not viable for production environments.
> *Expected Behavior*
> ABFS driver should support Soft Delete feature on FNS blob storage accounts,
> or the migration documentation should clearly document this incompatibility
> as a prerequisite check before migration.
> *Impact*
> Out-of-box failure for users migrating from WASB to ABFS on FNS storage with
> Soft Delete enabled
> Requires manual intervention and disabling of security features
> Migration path documented in wasb.md does not account for this limitation
> *Potential Solutions*
> Add support for Soft Delete in ABFS driver when using FNS blob storage
> Update WASB to ABFS migration guide to document this incompatibility and
> require Soft Delete to be disabled before migration
> Provide configuration option in ABFS to work with Soft Delete enabled
> accounts.
> Mention minimum required Hadoop Version for FNS Blob Storage with Soft delete
> migration from WASBS to ABFSS.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
