pengding-stripe commented on issue #14083: URL: https://github.com/apache/pinot/issues/14083#issuecomment-2450247669
@rohityadav1993 thanks for the response. I think spark-connector only supports read operations so everyone uses spark for ingestion is using the job spec (though I see there are some efforts to make spark-connector support ingestion were merged recently) > I think the spark jobs are not implemented in way to generate partitioned segments for upserts to work when uploaded I think we just need to fix this [part](https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java#L169-L182) to get these fields from job spec? Are you suggesting we should get from table config? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
