morningman opened a new pull request, #69749:
URL: https://github.com/apache/doris/pull/69749
### What problem does this PR solve?
Issue Number: None
Related PR: #67010
Problem Summary:
**In short.** This ports branch-4.1 #67010 (also on branch-4.2) to master:
CREATE / ALTER ROUTINE LOAD and async materialized views accept a
`compute_group` property in cloud mode, so a background job can be pinned to a
compute group instead of inheriting one implicitly, and the declared group is
re-checked before every task.
**Background.** In cloud mode every query runs in a compute group. A routine
load job snapshots the compute group of the session that created it, and an
async MV has no compute group of its own: an automatic refresh runs wherever
`admin` resolves to, and a manual `REFRESH` borrows the group of the session
that triggered it.
**The problem, and what it cost.** On master nothing lets a user choose
where a background job runs:
| | master today | with this PR |
|---|---|---|
| Routine load | runs in whatever group the creating session was on; ALTER
cannot move it | `"compute_group" = "cg_etl"` on CREATE or ALTER pins it |
| Async MV, automatic refresh | runs where `admin` resolves to | runs in the
declared group |
| Async MV, manual `REFRESH` | runs in the triggering session's group, so
the same MV refreshes in different places depending on who triggered it | runs
in the declared group; an MV that declares none keeps borrowing the session's
group |
| A pinned group is dropped, or the owner's USAGE on it is revoked | (no
pinning) | the next task fails with that reason instead of a later "no
available BE found" |
| `compute_group` in non-cloud mode, or `DEFAULT` | (no property) | rejected
at CREATE / ALTER |
Two smaller defects go with it on master: CREATE ROUTINE LOAD checks the
`workload_group` in the session's compute group namespace although the job runs
elsewhere, and `RoutineLoadJob.plan()` builds the sink's plan context from the
raw snapshot field rather than the effective group.
**How this PR fixes it.** Same design as #67010:
- The declaration is stored in the existing property maps
(`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`) and written only when
declared: no new persisted field, journal type or meta version, and an absent
key keeps today's implicit resolution, which is also how a later binding model
tells a pinned group from an inherited one. `DEFAULT` is rejected because that
model reserves it for "follow the owner's default group"; non-cloud mode
rejects the property, and `ConnectContext.setCloudCluster()` is a no-op outside
cloud mode.
- `ComputeGroupBindingUtil` holds the two checks:
`validateDeclaredComputeGroup` at CREATE / ALTER (USAGE for the current user,
then existence, the order of `USE @<group>`), and `checkComputeGroupBeforeTask`
before every task, against the identity the task runs as. A missing group keeps
`INTERNAL_ERR`, so a routine load job is paused retryably and resumes by itself
when the group comes back (a group scaled to zero backends leaves the cluster
map and returns); a revoked USAGE uses `CANNOT_RESUME_ERR`, so `ScheduleRule`
does not keep resuming it.
- Routine load resolves the effective group in
`RoutineLoadJob.getCloudCluster()`, which backend selection, the task sent to
BE, the workload group namespace, the plan context and the `ComputeGroup`
column of SHOW ROUTINE LOAD all use, so Kafka and Kinesis jobs both follow it.
`RoutineLoadTaskScheduler` re-checks the declared group before backend
allocation and before `beginTxn`. CREATE validates the workload group in the
declared group's namespace, and the statement re-parsed at metadata load does
not re-validate the declared group, because a failure there would cancel the
job for good.
- An async MV applies the declared group, then the session's group, on the
planning context of the task and on every refresh context, and re-checks the
declared group before each refresh.
Where master differs from branch-4.x:
- master's `MTMVTask` splits a refresh into the IVM incremental path and the
partition / complete path, each with its own context. `applyComputeGroup()` is
factored out of the existing `setupComputeGroup()`, and the re-check runs in
both paths.
- master's routine load also has a Kinesis source; it goes through the same
`getCloudCluster()` and scheduler code.
- The 4.x unit test for a task without a task context is not carried over:
`MTMVJob` always creates a task with a context, and master's
`setupComputeGroup()` already relies on it.
- The three unit test classes are converted to JUnit 5, as master requires,
and `MTMVComputeGroupTest` gains two cases for the master-only parts:
`setupComputeGroup()` applies and records the declared group, and the MV
re-check fails for a declared group that no longer exists while it ignores the
session's group.
- `ConnectContextTest.testThreadInfoUsesSessionCloudCluster` (#68111) sets
the session variable directly instead of calling
`ConnectContext.setCloudCluster()`, which is now a no-op outside cloud mode;
branch-4.x made the same change when it picked #68111 (#68948).
**Results.** The unit tests below pass locally. The regression suites need a
cloud cluster and were not run locally: `test_routine_load_compute_group` and
`test_mtmv_compute_group` are p0 suites, so P0 runs their non-cloud rejection
checks and cloud_p0 runs them in cloud mode (the routine load part also needs
Kafka); `test_compute_group_binding_isolation` is a docker suite that starts
its own two-group cloud cluster.
### Release note
Support declaring `compute_group` on routine load jobs and async
materialized views in cloud mode, so a background job can be pinned to a
specific compute group instead of inheriting one implicitly.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `cloud_p0/multi_cluster/test_compute_group_binding_isolation`,
`load_p0/routine_load/test_routine_load_compute_group`,
`mtmv_p0/test_mtmv_compute_group`, carried over from #67010 unchanged. Not run
locally (cloud cluster and Kafka needed).
- [x] Unit Test
- `RoutineLoadComputeGroupTest`, `MTMVComputeGroupTest`,
`ComputeGroupBindingUtilTest`, plus the existing tests of the changed classes
and of every `setCloudCluster()` caller (`mvn -pl :fe-core -am test`,
Checkstyle included): 22 classes, 361 tests, all pass.
- Each of five mutations makes a new test fail: the session's group
winning over an MV's declared one, `setupComputeGroup()` ignoring the
declaration, the MV re-check looking at the session's group, the routine load
scheduler skipping the re-check, and `setCloudCluster()` recording a group
outside cloud mode.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Only for jobs that declare `compute_group`: such an MV
refreshes in its declared group even for a manual `REFRESH` from another group,
and such a job fails its next task when the group is dropped or its owner's
USAGE on it is revoked. Outside cloud mode `ConnectContext.setCloudCluster()`
no longer records a value.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new `compute_group` property of routine load and async
MV: cloud mode only for now, `DEFAULT` reserved.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
