tibrewalpratik17 commented on PR #14506: URL: https://github.com/apache/pinot/pull/14506#issuecomment-2492040031
> A more robust way is to allow server to change the crc when updating the download URL. So currently server uploads the file with a UUID suffix inside the segmentUploadDir and it is controller which moves the zip file to the final deepstore location and then controller updates the download URL. If we move the logic of updating ZK to server that might become more risky as controller <> deepstore may fail and we might be left with a non-empty downloadURL pointing to an empty path (may cause FileNotFound exception in other places). > Ideally we need to make text index deterministic. Having indeterministic crc can cause lots of problems. Re-downloading all segments during server startup is not acceptable Agreed! But this seems a larger scope problem to what we are doing here. Right now given we know replicas have divergent crc, deepstore-upload retry backing up segments from random replica seems to be problematic which we are trying to solve. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
