Re: Analysis of Lance storage format support

[ForwardXu]

Hi all,
I think it makes perfect sense to create a dedicated roadmap for Lance support. 
This will help us clarify our priorities and ensure we can deliver more 
comprehensive support, including advanced features like complex data types and 
blob types, among others.
Looking forward to discussing this further on Slack.


Best,
Forwardxu


         原始邮件
         
       
发件人：Lorenzo Affetti via dev <dev@fluss.apache.org&gt;
发件时间：2026年3月2日 18:48
收件人：dev <dev@fluss.apache.org&gt;
抄送：forwardxu <forwar...@apache.org&gt;, Lorenzo Affetti 
<lorenzo.affe...@ververica.com&gt;
主题：Re: Analysis of Lance storage format support



       Hello!&nbsp;Thanks&nbsp;for&nbsp;wrapping&nbsp;this&nbsp;up!

I&nbsp;do&nbsp;understand&nbsp;both&nbsp;Cheng&nbsp;and&nbsp;Keith.
For&nbsp;sure&nbsp;Lance&nbsp;support&nbsp;should&nbsp;be&nbsp;on&nbsp;par&nbsp;with&nbsp;other&nbsp;lake&nbsp;formats.&nbsp;If
something&nbsp;is&nbsp;not&nbsp;supported,&nbsp;there&nbsp;should&nbsp;be&nbsp;a&nbsp;concrete&nbsp;reason&nbsp;why&nbsp;(apart
from&nbsp;a&nbsp;lack&nbsp;of&nbsp;resources&nbsp;:)&nbsp;).

Still,&nbsp;input&nbsp;from&nbsp;the&nbsp;Lance&nbsp;community&nbsp;would&nbsp;be&nbsp;essential&nbsp;for
understanding&nbsp;evolution&nbsp;areas&nbsp;of&nbsp;the&nbsp;support&nbsp;itself.

For&nbsp;this&nbsp;item,&nbsp;I&nbsp;would&nbsp;take&nbsp;an&nbsp;approach&nbsp;similar&nbsp;to&nbsp;what&nbsp;Mehul&nbsp;did&nbsp;for
Iceberg&nbsp;support.
I&nbsp;think&nbsp;there&nbsp;is&nbsp;a&nbsp;lack&nbsp;of&nbsp;a&nbsp;roadmap&nbsp;for&nbsp;Lance&nbsp;support&nbsp;in&nbsp;2026.

Having&nbsp;a&nbsp;roadmap&nbsp;doesn't&nbsp;actually&nbsp;mean&nbsp;we&nbsp;will&nbsp;accomplish&nbsp;everything,&nbsp;but,
it&nbsp;signals&nbsp;that&nbsp;we&nbsp;understand&nbsp;the&nbsp;problem&nbsp;space&nbsp;and&nbsp;have&nbsp;an&nbsp;idea&nbsp;of&nbsp;the
sequence&nbsp;of&nbsp;actions&nbsp;to&nbsp;take.

@cheng,&nbsp;I&nbsp;think&nbsp;you&nbsp;are&nbsp;the&nbsp;de-facto&nbsp;owner&nbsp;of&nbsp;the&nbsp;Lance&nbsp;module.
Would&nbsp;it&nbsp;make&nbsp;sense&nbsp;to&nbsp;dedicate&nbsp;some&nbsp;of&nbsp;our&nbsp;resources&nbsp;to&nbsp;discuss&nbsp;this&nbsp;via
Slack&nbsp;and&nbsp;start&nbsp;drafting&nbsp;a&nbsp;roadmap?

On&nbsp;Sun,&nbsp;Mar&nbsp;1,&nbsp;2026&nbsp;at&nbsp;2:11 
PM&nbsp;Keith&nbsp;Lee&nbsp;<leekeiabstract...@gmail.com&gt;
wrote:

&gt;&nbsp;Hello&nbsp;Cheng,
&gt;
&gt;&nbsp;Good&nbsp;call.&nbsp;I&nbsp;agree&nbsp;that&nbsp;gathering&nbsp;input&nbsp;from&nbsp;Lance&nbsp;community&nbsp;will&nbsp;be
&gt;&nbsp;beneficial&nbsp;to&nbsp;inform&nbsp;integration&nbsp;of&nbsp;features&nbsp;such&nbsp;as&nbsp;vector&nbsp;search,&nbsp;vector
&gt;&nbsp;indexing&nbsp;and&nbsp;hybrid&nbsp;search.
&gt;
&gt;&nbsp;However,&nbsp;the&nbsp;issues&nbsp;I’ve&nbsp;outlined&nbsp;only&nbsp;meant&nbsp;to&nbsp;cover&nbsp;the&nbsp;scope&nbsp;of&nbsp;bringing
&gt;&nbsp;current&nbsp;fluss&nbsp;lance&nbsp;integration&nbsp;up&nbsp;to&nbsp;parity&nbsp;to&nbsp;other&nbsp;lakehouses&nbsp;like
&gt;&nbsp;paimon&nbsp;or&nbsp;iceberg&nbsp;e.g.&nbsp;batch&nbsp;or&nbsp;union&nbsp;read&nbsp;without&nbsp;lance&nbsp;feature&nbsp;such&nbsp;as
&gt;&nbsp;vector&nbsp;search.&nbsp;As&nbsp;such,&nbsp;I&nbsp;believe&nbsp;these&nbsp;can&nbsp;be&nbsp;decoupled&nbsp;and&nbsp;we&nbsp;can&nbsp;have&nbsp;a
&gt;&nbsp;separate&nbsp;effort,&nbsp;gathering&nbsp;input&nbsp;from&nbsp;lance&nbsp;community&nbsp;and&nbsp;FIP&nbsp;proposal&nbsp;for
&gt;&nbsp;integrating&nbsp;vector&nbsp;search&nbsp;into&nbsp;feature&nbsp;such&nbsp;as&nbsp;union&nbsp;read.
&gt;
&gt;&nbsp;Let&nbsp;me&nbsp;know&nbsp;what&nbsp;your&nbsp;thoughts&nbsp;are&nbsp;on&nbsp;this.&nbsp;Thank&nbsp;you!
&gt;
&gt;&nbsp;Best&nbsp;regards
&gt;&nbsp;Keith&nbsp;Lee
&gt;
&gt;
&gt;&nbsp;On&nbsp;Sun,&nbsp;1&nbsp;Mar&nbsp;2026&nbsp;at&nbsp;10:20,&nbsp;Cheng&nbsp;Wang&nbsp;<348448...@qq.com&gt;&nbsp;wrote:
&gt;
&gt;&nbsp;&gt;&nbsp;Hello&nbsp;Keith,
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Regarding&nbsp;our&nbsp;plan&nbsp;to&nbsp;implement&nbsp;union&nbsp;read&nbsp;for&nbsp;Lance&nbsp;using&nbsp;Flink,&nbsp;might
&gt;&nbsp;it
&gt;&nbsp;&gt;&nbsp;be&nbsp;beneficial&nbsp;to&nbsp;first&nbsp;gather&nbsp;input&nbsp;from&nbsp;the&nbsp;Lance&nbsp;community?
&gt;&nbsp;Understanding
&gt;&nbsp;&gt;&nbsp;the&nbsp;primary&nbsp;scenarios&nbsp;where&nbsp;union&nbsp;read&nbsp;would&nbsp;help&nbsp;in&nbsp;the&nbsp;machine&nbsp;learning
&gt;&nbsp;&gt;&nbsp;scenario,&nbsp;along&nbsp;with&nbsp;the&nbsp;most&nbsp;popular&nbsp;execution&nbsp;engine&nbsp;in&nbsp;Lance
&gt;&nbsp;ecosystem,
&gt;&nbsp;&gt;&nbsp;could&nbsp;ensure&nbsp;we're&nbsp;building&nbsp;the&nbsp;right&nbsp;integration&nbsp;to&nbsp;maximize&nbsp;its
&gt;&nbsp;adoption.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Regards,
&gt;&nbsp;&gt;&nbsp;Cheng&nbsp;Wang
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;------------------&amp;nbsp;Original&amp;nbsp;------------------
&gt;&nbsp;&gt;&nbsp;From:
&gt;&nbsp;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"dev"
&gt;&nbsp;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<
&gt;&nbsp;&gt;&nbsp;ipolyzos...@gmail.com&amp;gt;;
&gt;&nbsp;&gt;&nbsp;Date:&amp;nbsp;Sat,&nbsp;Feb&nbsp;28,&nbsp;2026&nbsp;11:20&nbsp;PM
&gt;&nbsp;&gt;&nbsp;To:&amp;nbsp;"dev"<dev@fluss.apache.org&amp;gt;;
&gt;&nbsp;&gt;&nbsp;Cc:&amp;nbsp;"Cheng&nbsp;Wang"<348448...@qq.com&amp;gt;;"forwardxu"<
&gt;&nbsp;&gt;&nbsp;forwar...@apache.org&amp;gt;;
&gt;&nbsp;&gt;&nbsp;Subject:&amp;nbsp;Re:&nbsp;Analysis&nbsp;of&nbsp;Lance&nbsp;storage&nbsp;format&nbsp;support
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;This&nbsp;is&nbsp;extremely&nbsp;helpful,&nbsp;thanks&nbsp;for&nbsp;putting&nbsp;this&nbsp;together.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Maybe&nbsp;we&nbsp;can&nbsp;create&nbsp;an&nbsp;umbrella&nbsp;ticket&nbsp;on&nbsp;GitHub&nbsp;to&nbsp;keep&nbsp;track&nbsp;on&nbsp;these
&gt;&nbsp;and
&gt;&nbsp;&gt;&nbsp;open&nbsp;individual&nbsp;tasks,&nbsp;for&nbsp;tracking.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Best,
&gt;&nbsp;&gt;&nbsp;Giannis
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;On&nbsp;Sat,&nbsp;28&nbsp;Feb&nbsp;2026&nbsp;at&nbsp;3:52 
PM,&nbsp;Keith&nbsp;Lee&nbsp;<leekeiabstract...@gmail.com
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;wrote:
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Hello,
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;As&nbsp;discussed&nbsp;on&nbsp;community&nbsp;sync&nbsp;yesterday&nbsp;on&nbsp;analysing&nbsp;where&nbsp;we&nbsp;are
&gt;&nbsp;at
&gt;&nbsp;&gt;&nbsp;the
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;moment&nbsp;in&nbsp;terms&nbsp;of&nbsp;Lance&nbsp;format&nbsp;support.
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Here&nbsp;are&nbsp;my&nbsp;findings&nbsp;as&nbsp;part&nbsp;of&nbsp;working&nbsp;on&nbsp;Lance&nbsp;QuickStart
&gt;&nbsp;&gt;&nbsp;documentation
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[1].&nbsp;Lance&nbsp;lake&nbsp;tiering&nbsp;works&nbsp;in&nbsp;general,&nbsp;however&nbsp;there&nbsp;are&nbsp;some
&gt;&nbsp;gaps
&gt;&nbsp;&gt;&nbsp;that
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;to&nbsp;be&nbsp;addressed&nbsp;to&nbsp;bring&nbsp;Lance&nbsp;format&nbsp;support&nbsp;in&nbsp;parity&nbsp;with&nbsp;Paimon
&gt;&nbsp;/
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Iceberg.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;-&nbsp;(Merged)&nbsp;Support&nbsp;for&nbsp;Arrow&nbsp;FixedSizeList&nbsp;to&nbsp;enable&nbsp;pylance&nbsp;native
&gt;&nbsp;&gt;&nbsp;vector
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;search&nbsp;[2]
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;-&nbsp;(In&nbsp;progress)&nbsp;Support&nbsp;Flink&nbsp;SQL&nbsp;Union&nbsp;Read&nbsp;query&nbsp;against&nbsp;Lance
&gt;&nbsp;&gt;&nbsp;table&nbsp;[3]
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;-&nbsp;(Open)&nbsp;Support&nbsp;Flink&nbsp;SQL&nbsp;batch&nbsp;query&nbsp;against&nbsp;Lance&nbsp;table&nbsp;[4]
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;-&nbsp;(Blocked)&nbsp;Primary&nbsp;Key&nbsp;table&nbsp;support&nbsp;-&nbsp;I&nbsp;believe&nbsp;this&nbsp;is&nbsp;still
&gt;&nbsp;&gt;&nbsp;blocking&nbsp;on
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Lance&nbsp;format&nbsp;support&nbsp;for&nbsp;delete&nbsp;API&nbsp;[5]
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Finally&nbsp;there&nbsp;is&nbsp;also&nbsp;a&nbsp;gap&nbsp;in&nbsp;the&nbsp;ability&nbsp;of&nbsp;performing&nbsp;vector
&gt;&nbsp;&gt;&nbsp;search&nbsp;on
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;hot&nbsp;data&nbsp;/&nbsp;via&nbsp;union&nbsp;read.&nbsp;After&nbsp;discussion&nbsp;with&nbsp;Mehul,&nbsp;native
&gt;&nbsp;vector
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;indexing&nbsp;on&nbsp;hot&nbsp;data&nbsp;in&nbsp;Fluss&nbsp;would&nbsp;be&nbsp;a&nbsp;separate,&nbsp;bigger&nbsp;effort
&gt;&nbsp;that
&gt;&nbsp;&gt;&nbsp;we
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;can&nbsp;evolve&nbsp;towards&nbsp;if&nbsp;there's&nbsp;demand&nbsp;for&nbsp;it.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Appreciate&nbsp;feedback&nbsp;here&nbsp;from&nbsp;Cheng,&nbsp;Forward&nbsp;and&nbsp;anyone&nbsp;else&nbsp;with
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;familiarity&nbsp;around&nbsp;this&nbsp;area&nbsp;as&nbsp;I&nbsp;have&nbsp;only&nbsp;started&nbsp;dipping&nbsp;my&nbsp;toes
&gt;&nbsp;&gt;&nbsp;into
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Lance.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;*Additionally,&nbsp;if&nbsp;anyone&nbsp;wants&nbsp;to&nbsp;help&nbsp;contributing&nbsp;in&nbsp;this&nbsp;area,
&gt;&nbsp;&gt;&nbsp;please
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;reach&nbsp;out.&nbsp;*
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Best&nbsp;regards
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Keith&nbsp;Lee
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;Reference
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[1]&nbsp;https://github.com/apache/fluss/pull/2716
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[2]&nbsp;https://github.com/apache/fluss/issues/2706
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[3]&nbsp;https://github.com/apache/fluss/issues/2715
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[4]&nbsp;https://github.com/apache/fluss/issues/2751
&gt;&nbsp;&gt;&nbsp;&amp;gt;&nbsp;[5]&nbsp;https://github.com/lance-format/lance/issues/3961
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;


--&nbsp;
Lorenzo&nbsp;Affetti
Senior&nbsp;Software&nbsp;Engineer&nbsp;@&nbsp;Flink&nbsp;Team
Ververica&nbsp;<http://www.ververica.com&gt;