morningman commented on code in PR #70118:
URL: https://github.com/apache/doris/pull/70118#discussion_r4464958120
##########
fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java:
##########
@@ -1856,13 +1858,37 @@ public String getShowCreateInfo() {
if (isPartialUpdate) {
appendProperties(sb, CreateRoutineLoadInfo.PARTIAL_COLUMNS,
isPartialUpdate, false);
}
+ // UPDATE_FLEXIBLE_COLUMNS leaves isPartialUpdate false, so without
the mode the job would be recreated
+ // as an UPSERT one. CREATE only accepts the new key policy together
with a partial update mode.
+ if (uniqueKeyUpdateMode != TUniqueKeyUpdateMode.UPSERT) {
+ appendProperties(sb, CreateRoutineLoadInfo.UNIQUE_KEY_UPDATE_MODE,
uniqueKeyUpdateMode.name(), false);
+ if (partialUpdateNewKeyPolicy == TPartialUpdateNewRowPolicy.ERROR)
{
+ appendProperties(sb,
CreateRoutineLoadInfo.PARTIAL_UPDATE_NEW_KEY_POLICY, "ERROR", false);
+ }
+ }
appendProperties(sb, JsonFileFormatProperties.PROP_JSON_PATHS,
getJsonPaths(), false);
appendProperties(sb, JsonFileFormatProperties.PROP_STRIP_OUTER_ARRAY,
isStripOuterArray(), false);
appendProperties(sb, JsonFileFormatProperties.PROP_NUM_AS_STRING,
isNumAsString(), false);
appendProperties(sb, JsonFileFormatProperties.PROP_FUZZY_PARSE,
isFuzzyParse(), false);
appendProperties(sb, JsonFileFormatProperties.PROP_JSON_ROOT,
getJsonRoot(), false);
+ if (getEnclose() != 0) {
+ appendProperties(sb, CsvFileFormatProperties.PROP_ENCLOSE, new
String(new byte[]{getEnclose()}), false);
Review Comment:
Not changed here. The byte this line prints is not the byte the job's tasks
use either: when a task is planned, `NereidsDataDescription` (lines 422-423,
since #50882) turns the job's enclose and escape back into text with the same
`new String(new byte[]{...})`, and `CsvFileFormatProperties` parses that text
again. Every byte from 0x80 up becomes U+FFFD there: a throwaway unit test got
0xFD as enclose and 0xEF as escape for every such byte through
`NereidsDataDescription`, and for 0xE9 / 0xC3 through the planner a routine
load task uses. So on master a job created with `"enclose" = "é"` (0xE9 on the
job) and `"escape" = "é"` (0xC3) already runs with 0xFD and 0xEF, and the job
recreated from the printed U+FFFD (0xFD and 0xEF on the job) runs with the same
two bytes. SHOW ROUTINE LOAD shows U+FFFD for both, since CREATE records the
byte with the same conversion. The two jobs differ in a field, not in how they
parse CSV.
What is wrong is that no byte from 0x80 up means anything as enclose or
escape here. Routine load data is validated as UTF-8 by default, and there such
a byte is either absent or part of a character: 0xC3 as escape would cut the
first byte out of every `é`. So #70121, which restores these values on replay
and applies them on ALTER, now makes CREATE and ALTER ROUTINE LOAD take only an
ASCII character for the two (`3f9e13b4f74`), and ignores the non-ASCII values
older versions recorded. Every byte a job can then hold prints and reads back
unchanged, whichever PR merges first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
