morningman commented on code in PR #70121:
URL: https://github.com/apache/doris/pull/70121#discussion_r4461647065
##########
fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java:
##########
@@ -2047,8 +2054,24 @@ public void gsonPostProcess() throws IOException {
} else {
partialUpdateNewKeyPolicy =
TPartialUpdateNewRowPolicy.APPEND;
}
+ } else if
(k.equals(CreateRoutineLoadInfo.MAX_FILTER_RATIO_PROPERTY)) {
+ maxFilterRatio = Double.parseDouble(v);
+ } else if (k.equals(CreateRoutineLoadInfo.SEND_BATCH_PARALLELISM))
{
+ sendBatchParallelism = Integer.parseInt(v);
+ } else if (k.equals(CreateRoutineLoadInfo.LOAD_TO_SINGLE_TABLET)) {
+ loadToSingleTablet = Boolean.parseBoolean(v);
+ } else if (k.equals(CsvFileFormatProperties.PROP_ENCLOSE)) {
+ enclose = csvControlByte(k, v, enclose);
Review Comment:
Fixed in 198ae6c5fc7.
Confirmed. The FE runs with `-Dfile.encoding=UTF-8`, and in that charset
`new String(new byte[]{b})` turns every byte from 0x80 up into U+FFFD. The
recording itself predates this PR; what this PR changed is that the replay now
reads the recorded value back. With the first commit, a job created with
`'enclose' = 'é'` (0xE9) therefore came back with enclose 0xFD, and its escape
(0xC3, since `parseEscape` takes the first UTF-8 byte) came back as 0xEF. The
CSV reader applies the escape to every string cell, so an escape of 0xEF would
also drop the first byte of full-width characters such as `，`.
CREATE now records the value the statement gave, the same thing ALTER
records, and the replay parses it with the same `parseEnclose` / `parseEscape`
that CREATE uses, so the job gets the same byte back. When the statement gives
no value, CREATE still records the zero byte, so SHOW ROUTINE LOAD prints
exactly what it did before for jobs without enclose or escape and for ASCII
values.
A job that an older version created with such a byte already has U+FFFD in
its metadata, and the original byte cannot be recovered from that. The replay
ignores U+FFFD with a warning, and the job keeps no enclose and no escape,
which is what it has run with after every restart until now. ALTER ROUTINE LOAD
can set the value again.
`RoutineLoadJobReplayTest` gets two cases. In the first, `é` round-trips to
the same bytes in the replayed tasks. In the second, metadata holding U+FFFD
replays to no enclose and no escape, and the job stays NEED_SCHEDULE. Each case
fails when its part of the change is reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
