[
https://issues.apache.org/jira/browse/HADOOP-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099699#comment-18099699
]
ASF GitHub Bot commented on HADOOP-19941:
-----------------------------------------
anmolanmol1234 commented on code in PR #8611:
URL: https://github.com/apache/hadoop/pull/8611#discussion_r3664939759
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1673,30 +1750,101 @@ public Hashtable<String, String>
getXMSProperties(AbfsHttpOperation result)
@Override
public ListResponseData parseListPathResults(AbfsHttpOperation result, URI
uri)
throws AzureBlobFileSystemException {
+ final ListBlobResponseParser parser = ResponseParserFactory.getParser(
+ result.getResponseHeaderIgnoreCase(CONTENT_TYPE),
+ getBaseUrl().toString(),
+ () -> saxParserThreadLocal.get(),
+ getAbfsConfiguration().getPhotonArrowMemoryLimit());
try (InputStream stream = result.getListResultStream()) {
try {
- BlobListResultSchema listResultSchema;
- final SAXParser saxParser = saxParserThreadLocal.get();
- saxParser.reset();
- listResultSchema = new BlobListResultSchema();
- saxParser.parse(stream,
- new BlobListXmlParser(listResultSchema, getBaseUrl().toString()));
+ BlobListResultSchema listResultSchema = parser.parse(stream);
Review Comment:
parser.parse() on the Arrow path can throw unchecked exceptions
(org.apache.arrow.memory.OutOfMemoryException,
IllegalArgumentException/IllegalStateException on schema or vector-type
mismatch). These skip the inner catch (IOException) and land in the outer catch
(Exception), which logs"Unable to get stream for list results". The resulting
exception and metric are correct, but the log message is misleading — it points
at response-stream retrieval when the body was read successfully and it was
parsing that failed. On a Photon rollout this is the line on-call will grep.
Should catch RuntimeException alongside IOException in the inner block so parse
failures are wrapped and logged as parse failures:
> ABFS: Support Photon (Apache Arrow) based ListBlobs on Blob endpoint with XML
> fallback
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-19941
> URL: https://issues.apache.org/jira/browse/HADOOP-19941
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
> Reporter: Manish Bhatt
> Assignee: Manish Bhatt
> Priority: Major
> Labels: pull-request-available
>
> Add config-gated support (fs.azure.photon.enabled, default off) for consuming
> ListBlobs responses in the Apache Arrow (Photon) format on the ABFS Blob
> endpoint. When enabled, ABFS advertises Arrow via an Accept header; the
> response Content-Type selects an Arrow or the existing XML parser, both
> producing identical FileStatus results, so downstream behaviour is unchanged.
> Includes automatic, transparent fallback to XML, full parsing parity
> (metadata, directory markers, implicit directories, copy properties, native
> timestamp/length vectors), interrupt-safe Arrow parsing, and Photon telemetry
> (request, response, fallback, parse-failure counts and listing latency). No
> public API changes. Covered by unit and integration tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
