Hi Jark,
Thanks for your feedback.
1. The FIP design has been updated to indicate that the value of an
AUTO_INCREMENT column can only be implicitly assigned.
2. I agree that a high rate of server restarts could lead to sparse allocation
of AUTO_INCREMENT values and accelerate their exhaustion. To address this,
enabling a tablet server to resume from its cached values after a failure would
likely require implementing a sophisticated checkpoint & restore mechanism.
As far as I know, most database systems that rely on in-memory cached local
IDs??such as Doris and TiDB??typically discard these cached IDs during failover
for simplicity. I think we can refine our failure handling protocol in the
future. What do you think?
Regards,
Cheng
------------------ Original ------------------
From:
"dev"
<[email protected]>;
Date: Sun, Sep 21, 2025 09:51 PM
To: "dev"<[email protected]>;
Subject: Re: [DISCUSS] FIP-16: auto-increment column
Hi Cheng,
Thanks for starting this discussion. This is a very useful feature for
enabling roaringbitmaps use case.
I have two concerns regarding the current design:
*1) Explicit insertion into auto-increment columns*
I share Yuxia??s concern: allowing explicit inserts into auto-increment
columns breaks the fundamental guarantee of uniqueness. This could silently
corrupt roaringbitmaps results, and such issues would be extremely
difficult to debug.
I recommend disallowing explicit inserts unless we can guarantee
uniqueness. We can support explicit insert in the future when we have such
a solution.
*2) ID range persistence across restarts/failovers*
Currently, the ID range is cached only in memory at the tablet server. If
the server restarts or fails over, large gaps will appear in the ID
sequence. This leads to sparse roaringbitmaps and degraded the
roaringbitmaps performance. So I think we need a mechanism to persist and
restore the last allocated ID range to make the bitmap density.
Looking forward to your thoughts!
Best,
Jark
On Sun, 21 Sept 2025 at 04:56, Mehul Batra <[email protected]> wrote:
> Hi Cheng,
>
> Thanks for driving this, it's a needed feature to leap forward making the
> stack production ready for real-world scenarios.
> Design made sense to me, I have small questions:
>
> - *Cache Coordination*: When a tablet server fails and its cached IDs
> (e.g., 50,000-100,000) are lost, how does ZooKeeper ensure those IDs are
> never reused? Does it maintain a global highest allocated counter?
> - *Cross-bucket Dependencies*: In the example, bucket 1 gets [1-100,000]
> and bucket 2 gets [100,001-200,000]. What happens if tablet server
> containing bucket 1 goes down permanently? Will there always be gaps in the
> sequence?
> - *Race Conditions*: If two Flink workers simultaneously lookup the same
> non-existent primary key, could both trigger insertIfNotExists and create
> duplicate auto-increment IDs? How is this prevented?
> - How should users decide the right table.auto_inc_cache_size?
Should we
> put a max cap on this to avoid overburden
>
> Best Regards,
> Mehul Batra
>
> On Fri, Sep 19, 2025 at 5:24?6?2PM Yang Wang <[email protected]>
> wrote:
>
> > Hi Cheng,
> >
> > Thank you for driving this FIP. I think it is a nice and important
> feature
> > for many real-world business scenarios, and the overall design makes
> sense
> > to me. I have just one small question:
> > Regarding the client-side API design:
> > ```
> > Schema.newBuilder()
> > .column("uid", DataTypes.STRING())
> > .column("uid_int64",
DataTypes.BIGINT())
> > .enableAutoIncrement()
> > .primaryKey("uid")
> > .build();
> > ```
> > If there is more than one column with INT or BIGINT type, which one
would
> > be the auto-increment column?
> >
> > Best regards,
> > Yang
> >
> > Wang Cheng <[email protected]> ??2025??9??18??????
22:49??????
> >
> > > Hi all,
> > >
> > >
> > > Auto-increment column is a bread-and-butter feature for
improving data
> > > management efficiency. It is the bedrock of many features in
analytical
> > > workloads??such as those in real-time unique visitor (UV)
counting
> > scenarios.
> > >
> > >
> > > To implement this capability, I'd like to propose FIP-16:
> auto-increment
> > > column [1].
> > >
> > >
> > > Any feedback and suggestions on this proposal are welcome!
> > >
> > >
> > > [1]:
> > >
> >
>
https://cwiki.apache.org/confluence/display/FLUSS/FIP-16%3A+Auto-Increment+Column
> > >
> > >
> > >
> > >
> > > Regards,
> > > Cheng
> > >
> > >
> > >
> > > &nbsp;
> >
>
