hacklu-tu opened a new pull request, #68131:
URL: https://github.com/apache/doris/pull/68131
### What problem does this PR solve?
Issue Number: #48203
Related PR: None
Problem Summary:
Doris does not provide Hive-compatible `encode(string, charset)` and
`decode(binary, charset)` scalar functions, which makes Hive workload migration
harder.
This PR:
- adds Nereids signatures and constant folding for `encode` and `decode`;
- adds vectorized BE implementations backed by ICU with strict malformed and
unmappable input handling;
- supports the six character sets documented by Hive: US-ASCII, ISO-8859-1,
UTF-8, UTF-16BE, UTF-16LE, and UTF-16, matched case-insensitively;
- follows Java/Hive UTF-16 BOM behavior and propagates nulls;
- adds FE/BE unit tests and SQL regression coverage for supported
conversions and error cases.
### Release note
Add Hive-compatible `encode` and `decode` scalar functions for US-ASCII,
ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE, and UTF-16.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_encode_decode` passed in generated-output and comparison
modes on a native Linux Doris cluster.
- [x] Unit Test
- BE `function_character_encoding_test.*`: 3 tests passed.
- FE `StringArithmeticTest`: 10 tests passed; Maven reactor
succeeded.
- Native Linux FE/BE build succeeded.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. Adds Hive-compatible `encode` and `decode` functions,
including UTF-16 BOM semantics, null propagation, and strict conversion errors.
- Does this need documentation?
- [ ] No.
- [x] Yes. A follow-up doris-website PR is required.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
