matanper opened a new issue, #69062: URL: https://github.com/apache/doris/issues/69062
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 with the CLucene check from apache/doris-thirdparty#368. master is the same: `be/src/storage/index/inverted/inverted_index_writer.cpp:387-393` (`add_values`) and `:428-440` (`add_array_values`) pass untokenized string values to CLucene with only the `ignore_above` / empty checks. ### Description One string value with an embedded NUL byte fails the whole load when the column has an untokenized (parser none) inverted index. apache/doris-thirdparty#368 made CLucene reject a term that holds a NUL byte followed by another byte (`SDocumentsWriter::FieldData::addPosition`), to avoid a stack overflow. `InvertedIndexColumnWriter::add_document` turns the CLucene exception into an error, the memtable flush fails, and the Stream Load fails: ``` [E-6002] CLuceneError add_document: Inverted index does not support terms with null byte (\0) in the middle. Found null at position 6, but non-null character at position 7. Term prefix: ''. ``` This applies to a plain STRING column, an `ARRAY<STRING>` element, and a VARIANT sub-column indexed through `field_pattern` (the same writer). Tokenized indexes are not affected: their analyzers split on NUL. Log data often carries NUL bytes in otherwise ordinary fields. A sender that retries a failed batch (for example a Kafka consumer that does not commit the failed offsets) then fails on the same row forever, so one value stops ingestion for the whole table. ### Solution Treat such a value like one above `ignore_above`, for untokenized indexes only: - scalar column: add a null document for that row; - array column: skip that element. The stored value is unchanged; only its index term is left out, so the row is still found by a scan, and other rows keep their row ids and terms. A trailing NUL, which CLucene accepts, can stay indexed. A counter and a rate-limited warning make the skips visible. We have a patch for `InvertedIndexColumnWriter::add_values` / `add_array_values` with BE unit tests (`InvertedIndexWriterTest`) and can open a PR. An alternative is to escape NUL in index terms on both the write and the query side, so such values stay searchable; that needs a query-side change too. ### How to Reproduce? ```sql CREATE TABLE t (k int, s string, INDEX idx_s (s) USING INVERTED) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ('replication_num' = '1'); ``` Stream Load (JSON) two rows: `{"k": 1, "s": "abc\u0000def"}` and `{"k": 2, "s": "ok"}`. The load fails with the error above and neither row is stored. The same happens with `ARRAY<STRING>` and an element `"a\u0000b"`, and with a VARIANT column whose `field_pattern` index matches a path holding `"a\u0000b"`. ### Anything Else? Expected: the load succeeds, both rows are stored with the NUL intact, and `WHERE s = 'ok'` still uses the index. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
