felixhzhu opened a new issue, #4431:
URL: https://github.com/apache/amoro/issues/4431
### What happened?
After upgrading AMS from **0.8.x to 0.9.x**, disabling self-optimizing no
longer closes the current optimizing process.
In 0.8.x, after `self-optimizing.enabled=false` the process became
**CLOSED**. In 0.9.x the row stays **RUNNING** in `table_process` (often for
days), the dashboard keeps showing RUNNING, and Dashboard **Cancel** does not
change it.
We also see `amoro_table_optimizing_process_failed_count_total` increase in
the same window. `close()` / failed commit both call `completeProcess(false)`,
so **turning optimizing off is counted as a process failure**. There is no
metric that means “CLOSED by user/config” vs “FAILED commit”.
**Expected**
- Setting `self-optimizing.enabled=false` should persist the current process
as `CLOSED` (same as 0.8 / AMORO-2250), including when the in-memory
`OptimizingProcess` is already gone.
- Cancel should close by `processId` from DB. If it cannot close, the HTTP
API should not return 200.
- Close/cancel should not increment `process_failed_count` (or should use a
separate reason/metric).
### Affects Versions
0.9.0-incubating (regression vs 0.8.x)
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS, Spark
### How to reproduce
1. Iceberg table with a MINOR process in `RUNNING`.
2. From Spark (Copy-on-Write upsert job):
```sql
ALTER TABLE catalog.db.table SET TBLPROPERTIES ('self-optimizing.enabled' =
'false');
```
3. Optionally run Spark CoW upsert on the same table (common production
pattern: disable optimizing, then rewrite).
4. Check AMS DB / Optimizing page:
```sql
SELECT process_id, status, process_type, create_time, finish_time
FROM table_process
WHERE table_id = ? AND status = 'RUNNING';
```
The process stays `RUNNING` (`finish_time` NULL). Repeating this daily
accumulates many orphan RUNNING rows (we had 15+ on one table).
`status='CLOSED'` / `'FAILED'` can be empty.
5. Open the process on the Optimizing page and click **Cancel**. UI still
shows RUNNING. HTTP cancel returns 200.
### Relevant log output
```shell
When Spark already rewrote the same files and AMS later tries to commit the
leftover process:
org.apache.amoro.exception.OptimizingCommitException: unexpected commit error
Caused by: org.apache.iceberg.exceptions.ValidationException: Missing
required files to delete: ...
at org.apache.iceberg.ManifestFilterManager.validateRequiredDeletes
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.rewriteFiles
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit
That path increments `process_failed_count`. A successful in-memory
`close(false)` also increments it via `persistAndSetCompleted(false)`.
```
### Anything else
This looks like a 0.9 process-framework regression of
[#2250](https://github.com/apache/amoro/issues/2250) (disable self-optimizing
should close the running process). 0.9 moved history to `table_process` and
only closes the **in-memory** process:
- `TableRuntimeRefreshExecutor.handleConfigChanged` closes only when config
flips true→false **and** `getOptimizingProcess() != null`. Later refreshes do
not retry. If the in-memory process is already null, the DB row stays RUNNING.
- `DefaultOptimizingService.cancelProcess` returns `false` when
`getOptimizingProcess()` is null or the processId does not match.
`TableController.cancelOptimizingProcess` still returns HTTP 200.
This is different from [#4172](https://github.com/apache/amoro/issues/4172)
(all tasks SUCCESS, stuck before COMMITTING). Here the process is an orphan
RUNNING after disable/cancel, and `failed_count` treats close as failure.
Suggested direction (no code in this issue, just for maintainers):
1. When `self-optimizing.enabled=false`, close **all** RUNNING processes for
that table from DB, not only the in-memory one; retry on every refresh while
disabled.
2. `cancelProcess(processId)` should persist-close the `table_process` row
if the in-memory process is missing; HTTP should fail when cancel did not
happen.
3. `completeProcess(false)` after user/config close should not increment
`process_failed_count`.
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
