sunchao commented on code in PR #5042:
URL: https://github.com/apache/datafusion-comet/pull/5042#discussion_r3876128752
##########
native/spark-expr/src/string_funcs/levenshtein.rs:
##########
@@ -26,67 +26,188 @@ use datafusion::common::{cast::as_generic_string_array,
DataFusionError, Result}
use datafusion::physical_plan::ColumnarValue;
use std::sync::Arc;
+// Thread-local scratch buffers to avoid heap allocations in the row
processing loop
+thread_local! {
+ static LEVENSHTEIN_SCRATCH: std::cell::RefCell<(Vec<i32>, Vec<i32>)> =
+ std::cell::RefCell::new((Vec::with_capacity(64),
Vec::with_capacity(64)));
Review Comment:
I rechecked `8456bc5`; this is only partly addressed. `prepare_scratch`
shrinks buffers only on a later small, nonempty ASCII calculation. Both Unicode
paths still use `resize`, and empty inputs return before cleanup. Processing
equal 16,000,000-character ASCII strings with threshold `0`, dropping the
inputs, then evaluating a one-character Unicode pair and an empty ASCII pair
still leaves 128,000,008 scratch bytes allocated. An oversized final row also
remains attached to the thread after evaluation. Could we use temporary buffers
above the cap or release oversized buffers after evaluation on both paths? This
was verified with Rust kernel/capacity probes, not an end-to-end Spark run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
