rajagopr opened a new pull request, #14429: URL: https://github.com/apache/pinot/pull/14429
Re-introducing [PR-12863](https://github.com/apache/pinot/pull/12863) post testing in an internal cluster. ### Testing **Minion Logs** ``` 2024/11/12 17:04:12.231 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Generated 1 segments with duration: 12261ms 2024/11/12 17:04:12.239 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Adding new segments: 1 created from multiple input files: 1 2024/11/12 17:04:12.273 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/suspects001/startDataIngestRequest?tableType=OFFLINE&taskType=FileIngestionTask to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown 2024/11/12 17:04:12.273 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Submitted checkpoint: FileIngestionTask_1731431035906_0 for table: suspects001_OFFLINE with new segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] 2024/11/12 17:04:14.949 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Total uncompressed segment size for task 60613536 bytes 2024/11/12 17:04:14.949 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Compressed segments with duration: 2676ms 2024/11/12 17:04:14.951 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploading compressed segment: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz with name: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0 2024/11/12 17:04:14.953 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Using push mode: METADATA to upload segment: /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz 2024/11/12 17:04:15.102 INFO [S3PinotFS] [TaskStateModelFactory-task_thread-0] Copy /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz from local to s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz 2024/11/12 17:04:15.411 INFO [IngestionTaskUtils] [TaskStateModelFactory-task_thread-0] Moved generated segment from: /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz to: s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz 2024/11/12 17:04:15.412 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Start pushing segment metadata: {s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz=/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz} to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@4d2aba53] for table: suspects001_OFFLINE 2024/11/12 17:04:15.413 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Checking if metadata tar gz file /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.metadata.tar.gz exists ``` ``` 2024/11/12 17:04:15.413 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to untar Metadata file from: [/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz] to [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a] ``` **[Metadata file getting generated from the local segment file]** ``` 2024/11/12 17:04:15.419 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to untar CreationMeta file from: [/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz] to [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a] 2024/11/12 17:04:15.574 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to tar segment metadata dir [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a] to [/tmp/segmentMetadata-18f05bf5-6295-4c98-b96b-f84efe53110a.tar.gz] 2024/11/12 17:04:15.580 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Pushing segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] to location: https://pinot-pinot-controller-headless.managed.svc.cluster.local:9000 for table: suspects001_OFFLINE 2024/11/12 17:04:15.832 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/batchUpload?tableName=suspects001_OFFLINE&tableType=OFFLINE to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown 2024/11/12 17:04:15.833 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Response for pushing table suspects001_OFFLINE segments [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] to location https://pinot-pinot-controller-headless.managed.svc.cluster.local:9000 - 200: {"status":"Successfully uploaded segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] of table: suspects001_OFFLINE in 206 ms"} 2024/11/12 17:04:15.833 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploaded compressed segment: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz with name: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0 2024/11/12 17:04:15.838 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploaded segments: 1 with duration: 886ms 2024/11/12 17:04:15.874 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/suspects001/endDataIngestRequest?tableType=OFFLINE&taskType=FileIngestionTask&checkpointEntryKey=FileIngestionTask_1731431035906_0 to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
