[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-05 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1622126199 > However, I am not sure why many of the tests are failing, even though the tests for Pforutil and forutil are passing. I will take a closer look at the specific reasons when I h

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-05 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1622121123 > I agree. There are more complications: DataInput does not have a read method for int[], only one for float[] and long[]. So changing this is a bigger task. I just notic

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-04 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1620744388 I agree. There are more complications: DataInput does not have a read method for int[], only one for float[] and long[]. So changing this is a bigger task. I tend to think that w

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1618683855 Here is the example what I did with your code: #12412 It should demonstrate how I inlcuded it. Most interesting is the cleanup inside the lucene90 codec. I also defined the

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1618548308 Hi, the bug in the generator is fixed. I was able to include the ForUtil generator into a local branch and made everything working. What I am wondering about: The genera

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1618261123 Here is the bugfix for the generator. This bug caused a lot trouble here because I did not understand why the script wasn't executed at all. -- This is an automated message fro

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1618230924 Hi, I just noticed that the forUtil.gradle had a bug in Lucene 9. It does not regenerate anything because the actual python script is never executed. I will povide a separa

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1617817314 One small suggestion: Please stay with classical switch statement, as backporting the python code and java code to Java 11 is getting harder (please remember: also the vectorized

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1617809930 I am also not sure if we really need to unroll all that loops and have different impls for each size. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1617805484 > What about changing the format now, as part of this change? This would mean that users who do not enable the Parama vector API would get a bit worse performance compared to tod

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1617754157 The framework to plug in in custom formats into `org.apache.lucene.internal.vectorization` was merged, so stage is open to implement something. Hints: - Both the scalar

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-03 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1617678206 P.S.: I checked: ForUtil in 9.x is different than in 8.x because the byte order and some other slight changes. So I would only vectorize the core one (9.x) and leave out the back

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-02 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1616745892 Here is my PR about the general setup to easily plug in different interfaces to support vectorization at other places (not only VectorUtil). This uses a non-exported package and

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-02 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1616432561 Great! I am in weekend mode bit I have a rewrite of the provider lookup almost ready. I will check how to include it. -- This is an automated message from the Apache Git Ser

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-01 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1615899286 I had some ideas in mind that the general lookup part just makes sure theres is a paname impl available and then delegates to the panama provider to decide which implementation t

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-01 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1615890843 I like the idea to put everything under internal, we have that apckage already. But I would keep the implementations all under one package and let codes use them (specific

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-01 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1615852602 One we have some code here ready, I will submit a PR with the refactoring as preparation. -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-07-01 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1615851386 Hey, please for not let me do the integration. I have a plan already because with the current version it wont work. We have no secrets at moment, so the implementation can

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-06-28 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1611856694 The above would be a separate PR to cleanup the adhoc internal implementation of the Panama Integration a bit. The implementation devloped here could then be added to the new Luc

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-06-28 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1611841983 You can take the current python script as "basis" and work from there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-06-28 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1611811043 Hi, if you look at the first line of `ForUtil.java`, you will see the following comment: ```java // This file has been automatically generated, DO NOT EDIT ```

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-06-26 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1606834703 In my opinion, the best would be to proceed here without integrating in Lucene and I can have a look at the code. The provider interfaces in Lucene are still package private, so

[GitHub] [lucene] uschindler commented on issue #12396: Make ForUtil Vectorized

2023-06-25 Thread via GitHub
uschindler commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1606799756 Hi this is a good idea, ForUtil is next to PackedInts a good option for SIMD. If you have something implemented, open a PR, we will think of the best way ho to include it w