[
https://issues.apache.org/jira/browse/HDFS-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068910#comment-18068910
]
Anuj Modi commented on HDFS-17898:
----------------------------------
Glad to know that you were able to fix the issue.
Coming to your query regarding the config.
No, client applications are not expected to set this compulsarily. ABFS driver
rely on this config to understand the account type, but if account type is not
set via this config driver itself tries to determine the account type and
proceed accordingly. But if this config is set, there is no cross check on the
value set by user and driver believes it to be accurate. If someone sets this
config as true for FNS accounts, driver will consider the account to be HNS and
act accordingly.
In your case it is clearly verified from portal that account is indeed FNS. If
still driver is considering account to be HNS then it might be the case that
config was earlier present with a value true. Can you cross check that?
Completely removing this config from everywhere will also work as driver will
itself determine account type.
> ABFS migration fails on FNS blob storage with Soft Delete enabled
> ------------------------------------------------------------------
>
> Key: HDFS-17898
> URL: https://issues.apache.org/jira/browse/HDFS-17898
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.4.2, 3.4.3
> Environment: *Environment*
> Hadoop version: _*3.4.2 and 3.4.3*_
> Storage SDK - Tried with both *6.x* and *12.X*
> Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
> Storage account features enabled: FNS and Soft Delete
> Migration path: *WASB → ABFS (using abfss:// scheme)*
> Reporter: Pavan Pothnis
> Priority: Blocker
> Attachments: image-2026-03-26-14-53-53-821.png
>
>
> When migrating from WASB to ABFS on a FlatNameSpace (FNS) Azure Blob Storage
> account that has both FNS and Soft Delete features enabled, the migration
> fails with an incompatibility error.
> *Environment*
> Hadoop version: _*3.4.2 and 3.4.3*_
> Storage SDK - Tried with both *6.x* and *12.X*
> Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
> Storage account features enabled: FNS and Soft Delete
> Migration path: *WASB → ABFS (using abfss:// scheme)*
> *Issue Description*
> Following the WASB to ABFS migration guide (as documented in wasb.md), users
> are instructed to simply change URIs from WASB scheme to ABFS scheme.
> However, when the underlying Azure Blob Storage account has Soft Delete
> enabled, the ABFS driver encounters a critical error that prevents data
> access.
> *Steps to Reproduce*
> Create an Azure Blob Storage account (GPv2) with FlatNameSpace enabled
> Enable Soft Delete feature on the blob storage account
> Upgrade Hadoop from WASB to ABFS (3.4.2 )
> Change URIs from wasb:// to abfss://
> Attempt to read/parse Avro files or any data from the storage
> *Error Message*
> java.lang.RuntimeException: java.lang.RuntimeException: The file or partition
> directory
> [{*}abfss{*}://{*}azureblobv3-cctf{*}@{*}adapterqablob{*}.{*}dfs{*}.core.windows.net/alldatatype.avro]
> is not valid. The parser encountered the following error while parsing the
> content: [org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
> "This endpoint does not support BlobStorageEvents or SoftDelete. Please
> disable these account features if you would like to use this endpoint.", 409,
> HEAD,
> https://adapterqablob.{*}dfs{*}.core.windows.net/azureblobv3-cctf/alldatatype.avro?upn=false&action=getStatus&timeout=90s,
> rId: 0b404e0b-301f-00a7-4c18-bc227f000000]. Select a valid [Avro] file or
> partition directory.
> Changes done as part of WABS to ABFS Migration
> *From -*
>
> String fileName = "wasbs://" +
> connection.getContainerClient().getBlobContainerName() + "@" +
> connection.getAccountName() + ".blob." +connection.getEndpointSuffix()+ "/";
>
> *To -*
>
> String fileName = "abfss://" +
> connection.getContainerClient().getBlobContainerName() + "@" +
> connection.getAccountName() + ".dfs." +connection.getEndpointSuffix()+ "/";
> *From -*
>
> String wasbFileSystemUrl = "wasbs://" + containerName + "@" +
> connection.getAccountName() +".blob."+connection.getEndpointSuffix()+"/";
>
> *To -*
>
> String wasbFileSystemUrl = "abfss://" + containerName + "@" +
> connection.getAccountName() +".dfs."+connection.getEndpointSuffix()+"/";
>
> *Configuration - Shared Key Authentication*
>
> if(authenticationType.equals(AzureStorageAuthorization.SHARED_KEY_AUTHORIZATION.getValue()))
> {
> *From -*
> String accountKey = "fs.azure.account.key." + connection.getAccountName() +
> ".blob." + endpointSuffix;
> *To -*
> **
> String accountKey = "fs.azure.account.key." + connection.getAccountName() +
> ".dfs." + endpointSuffix;
>
> configMap.put(accountKey, connection.getAccountKey());
>
> }
> *Configuration - Shared Access Signature*
>
> if(authenticationType.equals(AzureStorageAuthorization.SHARED_ACCESS_SIGNATURES.getValue()))
> {
> *From -*
> String containerName = StringUtils.isNotBlank(containerNameOverride) ?
> containerNameOverride : connection.getContainer().getName();
> String sharedaccesssignature = "fs.azure.sas." + containerName +"." +
> connection.getAccountName() + ".blob."+ endpointSuffix;
> configMap.put(sharedaccesssignature, connection.getSharedAccessSignature());
> *To -*
> configMap.put("fs.azure.account.auth.type." + connection.getAccountName() +
> ".dfs." + endpointSuffix, "SAS");
> configMap.put("fs.azure.sas.fixed.token." + connection.getAccountName() +
> ".dfs." + endpointSuffix, connection.getSharedAccessSignature());
>
> }
> All the changes made in To section works fine as long as the Blob account is
> not configured with Soft Delete / Event and fails when enabled, causing out
> of box failure for all account with those settings and these are extensively
> used settings.
>
> *Current Workaround*
> The issue only resolves after manually disabling Soft Delete on the Azure
> Blob Storage account. This workaround defeats the purpose of having this
> protective feature enabled and is not viable for production environments.
> *Expected Behavior*
> ABFS driver should support Soft Delete feature on FNS blob storage accounts,
> or the migration documentation should clearly document this incompatibility
> as a prerequisite check before migration.
> *Impact*
> Out-of-box failure for users migrating from WASB to ABFS on FNS storage with
> Soft Delete enabled
> Requires manual intervention and disabling of security features
> Migration path documented in wasb.md does not account for this limitation
> *Potential Solutions*
> Add support for Soft Delete in ABFS driver when using FNS blob storage
> Update WASB to ABFS migration guide to document this incompatibility and
> require Soft Delete to be disabled before migration
> Provide configuration option in ABFS to work with Soft Delete enabled
> accounts.
> Mention minimum required Hadoop Version for FNS Blob Storage with Soft delete
> migration from WASBS to ABFSS.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
