kaivalnp commented on code in PR #16723:
URL: https://github.com/apache/lucene/pull/16723#discussion_r4159248302
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissNativeWrapper.java:
##########
@@ -37,8 +38,44 @@
*/
@SuppressWarnings("restricted") // uses unsafe calls
final class FaissNativeWrapper {
- static {
+
+ static final String LOADED_LIBRARY = loadLibrary();
+
+ private static String loadLibrary() {
+ // Manual override via system property
+ String customLib = System.getProperty("lucene.faiss.libname");
Review Comment:
Nice! I think the library should also be present on `LD_LIBRARY_PATH`?
Should we document the new property somewhere?
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissNativeWrapper.java:
##########
@@ -37,8 +38,44 @@
*/
@SuppressWarnings("restricted") // uses unsafe calls
final class FaissNativeWrapper {
- static {
+
+ static final String LOADED_LIBRARY = loadLibrary();
+
+ private static String loadLibrary() {
+ // Manual override via system property
+ String customLib = System.getProperty("lucene.faiss.libname");
+ if (customLib != null && customLib.isBlank() == false) {
+ System.loadLibrary(customLib);
+ return customLib;
+ }
+ // Try SIMD variants matching the CPU architecture
+ switch (Constants.OS_ARCH) {
+ case "amd64", "x86_64" -> {
+ if (Constants.HAS_AVX512 && tryLoad("faiss_c_avx512")) {
+ return "faiss_c_avx512";
+ }
+ if (Constants.HAS_AVX2 && tryLoad("faiss_c_avx2")) {
+ return "faiss_c_avx2";
+ }
+ }
+ case "aarch64" -> {
+ if (Constants.HAS_SVE && tryLoad("faiss_c_sve")) {
+ return "faiss_c_sve";
+ }
+ }
+ }
+ // Fallback to generic baseline
System.loadLibrary(FaissLibrary.NAME);
+ return FaissLibrary.NAME;
Review Comment:
Let's have `FaissLibrary.NAME` point to the actual `LOADED_LIBRARY`?
##########
.github/workflows/run-special-checks-sandbox.yml:
##########
@@ -66,6 +66,12 @@ jobs:
- name: Prepare Lucene workspace
uses: ./.github/actions/prepare-for-build
+ - name: Prepare AVX2 Faiss C library
+ run: |
+ cp "$CONDA_PREFIX/lib/libfaiss_c.so"
"$CONDA_PREFIX/lib/libfaiss_c_avx2.so"
+ patchelf --replace-needed libfaiss.so libfaiss_avx2.so
"$CONDA_PREFIX/lib/libfaiss_c_avx2.so"
Review Comment:
Nice, I didn't know `libfaiss_avx2.so` was shipped with the Conda artifacts.
##########
.github/workflows/run-special-checks-sandbox.yml:
##########
@@ -66,6 +66,12 @@ jobs:
- name: Prepare Lucene workspace
uses: ./.github/actions/prepare-for-build
+ - name: Prepare AVX2 Faiss C library
+ run: |
+ cp "$CONDA_PREFIX/lib/libfaiss_c.so"
"$CONDA_PREFIX/lib/libfaiss_c_avx2.so"
Review Comment:
For the GH action: should we add the new file in a temp folder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
