cypherean opened a new issue, #17233:
URL: https://github.com/apache/pinot/issues/17233
The recently added `check.crc.on.segment.load` flag (#16432) has been
helpful for handling realtime segments with mismatched CRCs despite identical
data. However, during testing we observed a couple of side effects when this
flag is enabled at the server level:
Offline table backfill:
In our setup (Uber), offline segment names are date based. When backfilling
for an older date, the uploaded segment has the same name but a different CRC.
With `check.crc.on.segment.load` disabled, the controller updates to the new
CRC, but the server does not load the updated segment.
```
// Steps to replicate:
// 1. Start quickstart batch with config set to false
// 2. update the crc of a completed segment locally
// 3. use the upload segment script to re-upload the existing segment
Handling message: ZnRecord=57211ff0-eb7d-4227-86dd-1d634a37a961,
{CREATE_TIMESTAMP=1763398232543, EXECUTE_START_TIM ESTAMP=1763398232574,
MSG_ID=57211ff0-eb7d-4227-86dd-1d634a37a961, MSG_STATE=new,
MSG_SUBTYPE=REFRESH_SEGMENT, MSG_ TYPE=USER_DEFINE_MSG,
PARTITION_NAME=airlineStats_OFFLINE_16071_16071_0,
RESOURCE_NAME=airlineStats_OFFLINE, RETRY_ COUNT=0,
SRC_CLUSTER=QuickStartCluster, SRC_INSTANCE_TYPE=PARTICIPANT,
SRC_NAME=Controller_127.0.0.1_9002, TGT_NAME =Server_127.0.0.1_7050,
TGT_SESSION_ID=10056a43eb20018, TIMEOUT=-1,
segmentName=airlineStats_OFFLINE_16071_16071_0,
tableName=airlineStats_OFFLINE}H}, Stat=Stat {_version=0,
_creationTime=1763398232552,_modifiedTime=17633982325
52, _ephemeralOwner=0}
Replacing segment: airlineStats_OFFLINE_16071_16071_0 in table:
airlineStats_OFFLINE
Replacing segment: airlineStats_OFFLINE_16071_16071_0
.....
Waiting for lock to reload/refresh: airlineStats_OFFLINE_16071_16071_0,
queue-length: 0
Acquired lock to reload/refresh segment: airlineStats_OFFLINE_16071_16071_0
(lock-time=0ms, queue-length=0)
Skipping replacing segment: airlineStats_OFFLINE_16071_16071_0 even though
its CRC has changed from: 912377941 to:
912377942 because instance.check.crc.on.segment.load is set to false
Message 57211ff0-eb7d-4227-86dd-1d634a37a961 completed.
```
Upsert compaction:
When upsert compaction is enabled, Pinot attempts to commit a compacted
segment that retains the same name but has a different CRC. This segment is
also ignored by the server when the `check.crc.on.segment.load` flag is
disabled.
```
// With upsert compaction enabled on testing table
Skipping replacing segment: redacted_tablename__14__458__20251118T1630Z even
though its CRC has changed from: 3709360175 to: 533870250 because
instance.check.crc.on.segment.load is set to false
```
These behaviors suggest that skipping CRC check may inadvertently block
valid segment reloads in scenarios where segment names remain constant but CRCs
legitimately change (backfill, compaction).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
