airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4056032256
##########
be/src/storage/index/inverted/analyzer/custom_analyzer.cpp:
##########
@@ -70,12 +70,17 @@ ReaderPtr CustomAnalyzer::init_reader(ReaderPtr reader) {
}
TokenStreamComponentsPtr CustomAnalyzer::create_components() {
- auto tk = _tokenizer->create();
- TokenStreamPtr ts = tk;
- for (const auto& filter : _token_filters) {
- ts = filter->create(ts);
+ try {
+ auto tk = _tokenizer->create();
+ TokenStreamPtr ts = tk;
+ for (const auto& filter : _token_filters) {
+ ts = filter->create(ts);
+ }
+ return std::make_shared<TokenStreamComponents>(tk, ts);
+ } catch (const CLuceneError& e) {
Review Comment:
Fixed in `51d2e076add`.
The classic writer boundaries now catch `doris::Exception`, close the writer
on `add_document()` failures, and return `INVERTED_INDEX_ANALYZER_ERROR`.
`FullTextIndexReader::query()` applies the same conversion instead of allowing
a lazy analyzer exception to escape its `Status` API.
Before the production fix, the new writer and classic-reader failure-path
tests terminated on the uncaught exception. After the fix, the official ASAN BE
runner passed all 8 focused tests: the writer failure test, the reader
analysis-purpose suite (including
`ClassicReaderConvertsAnalyzerFailureToStatus`), and the tokenizer
dictionary-failure test. A full ASAN BE build also passed after this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
