Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-26 Thread J. D. Jordan
If we look to postgresql it allows defining arrays using FLOAT[N] or FLOAT ARRAY[N]. So that is an extra point for me to just using FLOAT[N]. From my quick search neither oracle* nor MySQL directly support arrays in columns. * oracle supports declaring a custom type using VARRAY and then using

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-26 Thread David Capwell
> DENSE seems to just be an array? So very similar to a frozen list, but with a > fixed size? How I read the doc, DENSE = ARRAY, but knew that couldn’t be the case, so when I read the code its fixed size array…. So the real syntax was “DENSE FLOAT32[42]” Not a fan of the type naming, and feel

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-26 Thread Patrick McFadin
I guess this is an excellent example to explore the minima of what constitutes a CEP. So far, CEPs have been some large changes, so where does something like this fit? (Wait. Did I beat Benedict to a Bike Shed? I think I did.) This is a list of everything needed for a CEP: Status Scope Goals Appr

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-26 Thread Benedict
We probably at least need to bike shed naming as we already have FLOAT, DOUBLE, and LIST - which are similar/overlapping types, and we shoo on should be consistent.If we introduce FLOAT32 we probably need that to be an alias of FLOAT and introduce FLOAT64 to alias DOUBLE for consistency.DENSE seem

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-25 Thread Mick Semb Wever
I was soo happy when I saw this, I know many users are going to be thrilled about it. On Wed, 26 Apr 2023 at 05:15, Patrick McFadin wrote: > Not sure if this is what you are saying, Josh, but I believe this needs to > be its own CEP. It's a change in CQL syntax and changes how clusters > op

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-25 Thread Patrick McFadin
Not sure if this is what you are saying, Josh, but I believe this needs to be its own CEP. It's a change in CQL syntax and changes how clusters operate. The change needs to be documented and voted on. Jonathan, you know how to find me if you want me to help write it. :) As a side comment to all of

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-25 Thread Josh McKenzie
To be fair Dinesh kind of primed that: > Do you intend to make this part of CEP-7 or as an incremental update to SAI > once it is committed? ;) I think this body of work more than stands on its own. Great work Jonathan, Mike, and Zhao; having native support for more ML-oriented workloads in C*

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-25 Thread Henrik Ingo
Jonathan what a great proposal/code. An enjoyable read. And at least for me educational! (Which is notable, as you're on my turf, I'm a Data Science major.) Sorry for splitting hairs but CEP-7 (as a spec, and wiki page) is approved and voted on and I assume there's no proposal to change that. That

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-25 Thread Rahul Xavier Singh
Very exciting. Love it. "Retrieval augmented" or "Data augmented" LLMs are the easiest way to "fine-tune" the output of LLM without actually fine-tuning. Currently Pinecone/Weaviate/Milvus are eating up the scene, with new players like Chroma coming out soon. We've been working with Langchain / LL

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-24 Thread David Capwell
This work sounds interesting, I would recommend decoupling the types from the ANN support as the types require client changes and can go in now (would give a lot of breathing room to get this ready for 5.0), where as ANN depends on SAI which is still being worked on. > On Apr 22, 2023, at 1:02

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-22 Thread Jonathan Ellis
My guess is that I will be able to get this ready to upstream before the rest of CEP-7 goes in, so it would make sense to me to roll it into that. On Fri, Apr 21, 2023 at 5:34 PM Dinesh Joshi wrote: > Interesting proposal Jonathan. Will grok it over the weekend and play > around with the branch.

Re: Adding vector search to SAI with heirarchical navigable small world graph index

2023-04-21 Thread Dinesh Joshi
Interesting proposal Jonathan. Will grok it over the weekend and play around with the branch. Do you intend to make this part of CEP-7 or as an incremental update to SAI once it is committed? > On Apr 21, 2023, at 2:19 PM, Jonathan Ellis wrote: > > Happy Friday, everyone! > > Rich text forma