arunkumarucet opened a new pull request, #19032:
URL: https://github.com/apache/pinot/pull/19032
## Problem
When a geo column with an H3 index (plus a transform such as `ST_Point(lat,
long)`) is added to a table's schema, segments created **before** the source
columns existed have no data to derive the geometry from. On reload, the
derived `BYTES` column is filled with its default null value — the empty byte
array (`FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_BYTES`).
Building the H3 index over those rows during segment reload called
`GeometrySerializer.deserialize()` directly on the empty bytes:
```
java.nio.BufferUnderflowException
at
org.apache.pinot.segment.local.utils.GeometrySerializer.readGeometry(GeometrySerializer.java:83)
at
org.apache.pinot.segment.local.utils.GeometrySerializer.deserialize(GeometrySerializer.java:68)
at
org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler.handleDictionaryBasedColumn(H3IndexHandler.java:218)
at ...H3IndexHandler.updateIndices(H3IndexHandler.java:147)
```
The exception propagated out of the Helix state transition and parked the
segment in an `ERROR` state, blocking upgrades (old segments never come online).
## Root cause
The segment-**creation** path builds the H3 index via the
`GeoSpatialIndexCreator` default `add(Object value, int dictId)`, which wraps
`deserialize()` in a try/catch and treats failures as `null` geometry (skipped
when `continueOnError` is enabled). The segment-**reload** path in
`H3IndexHandler` bypassed that — it called the static
`GeometrySerializer.deserialize(bytes)` directly and fed the result to
`add(Geometry)`, so an empty/default value threw before the `add()` guard could
skip it.
## Fix
Route both `H3IndexHandler` reload paths (`handleDictionaryBasedColumn`,
`handleNonDictionaryBasedColumn`) through the same tolerant `add(value,
dictId)` the creation path uses. Empty/default geometry is now skipped when
`continueOnError` is enabled, consistent with ingestion, instead of crashing
the reload.
## Test
Added `SegmentPreProcessorTest#testH3IndexCreationOnEmptyDefaultValue` (v1
and v3), which adds a derived H3 column whose default null value is the empty
byte array and then builds the H3 index over it. Verified as a real regression
guard: it fails with `BufferUnderflowException` without the fix and passes with
it. `H3IndexTest` and the existing H3 preprocessor tests remain green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
