morningman opened a new pull request, #69084:
URL: https://github.com/apache/doris/pull/69084
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** `information_schema.columns` reported a VARBINARY column as
`unknown`, with no length, while `DESC` shows it as `varbinary(16)`. FE did not
send the length of a VARBINARY column to BE, and BE's columns scanner had no
VARBINARY case. This PR reports `varbinary` / `varbinary(n)` and `n` as the
maximum length in characters and in bytes, as MySQL does for a VARBINARY(n)
column.
**Background.** Doris tables cannot hold VARBINARY, but views and external
tables (JDBC, Hive, Iceberg, Paimon binary columns) can. BE fills
`information_schema.columns` from FE's `describeTables`, which sends each
column's type and, from `Type.getColumnSize()`, its length; BE's
`SchemaColumnsScanner` turns them into DATA_TYPE, COLUMN_TYPE and the length
columns.
**The problem, and what it cost.** For `CREATE VIEW v AS SELECT cast('abc'
AS varbinary(16)) AS b`:
| | `DESC v` | `information_schema.columns`, master | with this PR |
|---|---|---|---|
| type | `varbinary(16)` | DATA_TYPE `unknown`, COLUMN_TYPE `unknown` |
`varbinary`, `varbinary(16)` |
| length | | CHARACTER_MAXIMUM_LENGTH, CHARACTER_OCTET_LENGTH and
COLUMN_SIZE NULL | 16, 16, 16 |
Tools that read column metadata from `information_schema.columns` saw a
column of unknown type and length.
**How this PR fixes it.**
- FE: `Type.getColumnSize()` returns the length of a VARBINARY type, as its
comment already says for binary data, so `describeTables` sends it.
- BE: `SchemaColumnsScanner` maps VARBINARY to DATA_TYPE `varbinary` and
COLUMN_TYPE `varbinary(n)`, and reports `n` as CHARACTER_MAXIMUM_LENGTH and
CHARACTER_OCTET_LENGTH, since the length of a binary column is in bytes already
(a string column's octet length is four times its length). From an FE without
this change, which sends no length, COLUMN_TYPE is `varbinary` and the lengths
are NULL.
**Results.**
- **FE UT.** `TypeTest.testVarbinaryColumnSizeIsItsLengthInBytes`: the
column size of `varbinary(16)` is 16, and of an unbounded VARBINARY 2147483647.
All 17 tests of `TypeTest` pass.
- **BE UT.** `SchemaColumnsScannerTest.varbinary_metadata_in_block` feeds
the scanner a VARBINARY column with length 16 and one without a length, as an
older FE sends it, and checks DATA_TYPE, COLUMN_TYPE and both length columns.
The local BE UT build is on an older base, so this test was only compiled
syntax-only with its flags here; CI runs it.
- **End to end.** On a local cluster with this change in FE and BE, the view
above reports `varbinary`, `varbinary(16)` and 16 in the three length columns.
A column cast to VARBINARY without a length, or returned by `to_binary`,
reports `varbinary(2147483647)` and 2147483647, matching `DESC`. VARCHAR
columns are unchanged (`varchar(10)`, 10, 40, 10).
- **Regression.** The new `test_varbinary_information_schema` (datatype_p0)
checks `DESC` and `information_schema.columns` of a view with a
`varbinary(16)`, an unbounded VARBINARY and a `varchar(10)` column. Its `.out`
was generated on that cluster; before this change the same query returned
`unknown` and NULL for both VARBINARY columns.
### Release note
`information_schema.columns` reports a VARBINARY column as `varbinary(n)`,
with `n` as its maximum length in characters and in bytes, instead of `unknown`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_varbinary_information_schema` (datatype_p0): `DESC` and
`information_schema.columns` of a view with a `varbinary(16)`, an unbounded
`varbinary` and a `varchar(10)` column.
- [x] Unit Test
- `SchemaColumnsScannerTest.varbinary_metadata_in_block` and
`TypeTest.testVarbinaryColumnSizeIsItsLengthInBytes`.
- [x] Manual test (add detailed scripts or steps below)
- The view above on a local cluster, before and after this change.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- `information_schema.columns` rows of VARBINARY columns carry their
type and length instead of `unknown` and NULL.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
