0AyanamiRei opened a new pull request, #67412:
URL: https://github.com/apache/doris/pull/67412
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Schema inference skips empty compressed CSV and JSON files
using format-specific minimum sizes, but the check used only the explicitly
configured compression type. With the default `UNKNOWN` type, an empty gzip
file selected by a glob was treated as non-empty and prevented a later valid
file from supplying the schema. Reuse the existing path-based compression
inference before applying the empty-content threshold.
### Release note
File table-valued functions now skip empty gzip CSV and JSON files during
schema inference when compression is inferred from the path.
### Check List (For Author)
- Test: Regression test and manual test
- Regression test: `test_empty_first_compressed_schema` for CSV and JSON
gzip globs
- Manual test: local CSV and JSON `DESC FUNCTION` and queries returned
the later valid file rows
- Behavior changed: Yes. Empty compressed files no longer block schema
inference for a matching glob.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
