nagasrisai opened a new pull request, #64575:
URL: https://github.com/apache/airflow/pull/64575
While wrapping up #64519, eladkal pointed to `BigQueryInsertJobOperator` as
a reference for handling transient errors. This is that follow-up.
The problem is that Salesforce Bulk errors live at the record level, not
the API level. When Salesforce is under concurrent write load you can get
`UNABLE_TO_LOCK_ROW` back on individual records, but the API call itself
succeeded — so you just end up with `success=False` entries in the result list.
There's no built-in way to deal with that today without unwrapping the XCom,
filtering for those codes, building a new payload, and calling the operator
again yourself.
This adds three optional parameters:
- `max_retries` (default `0`) — how many times to re-submit records that
come back with a transient error. Defaults to zero so existing behaviour is
unchanged.
- `retry_delay` (default `5.0`) — seconds to wait before each retry.
- `transient_error_codes` — which Salesforce `statusCode` values count as
transient. Defaults to `{"UNABLE_TO_LOCK_ROW", "API_TEMPORARILY_UNAVAILABLE"}`.
When retrying, only the failing records are re-submitted, not the whole
payload. The retry results slot back into the original positions because
Salesforce always returns results in the same order as the input. Permanent
errors like `INVALID_FIELD` are not in the default set and are never retried.
Related to #64519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
