felixhzhu opened a new issue, #4434:
URL: https://github.com/apache/amoro/issues/4434
### What happened?
After upgrading AMS from 0.8.x to **0.9.x**, table optimizing **status
duration / in-status gauges are stuck at IDLE**.
The Tables page correctly shows tables in `major` / `minor` / `pending` (for
example `t_entity_data` major for 44 minutes). But Prometheus still reports:
```
amoro_table_optimizing_status_idle_duration_mills{...} 1.56e9 # ~18 days
== AMS uptime
amoro_table_optimizing_status_executing_duration_mills{...} 0
amoro_table_optimizing_status_pending_duration_mills{...} 0
amoro_table_optimizing_status_in_executing{...} 0
```
So `executing_duration` cannot be used to alert on “table is optimizing”. In
0.8.x these gauges followed the real table status.
**Expected**
When table status is MINOR/MAJOR/FULL, `executing_duration_mills` /
`in_executing` should be non-zero. When PENDING, `pending_duration_mills` /
`in_pending` should be non-zero. Idle duration should reset when leaving IDLE.
### Affects Versions
0.9.0-incubating / 0.9.x (works on 0.8.x)
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
1. Run AMS 0.9, register Iceberg tables, start optimizers.
2. Wait until the Tables page shows a table in `minor` / `major` / `pending`.
3. Scrape AMS metrics:
```bash
curl -s localhost:9090/metrics | grep table_optimizing_status
```
4. Observe `idle_duration_mills` ≈ process uptime and
`executing_duration_mills=0` even for tables that are clearly optimizing.
### Relevant log output
```shell
Scraped while the UI showed `t_entity_data` major ~44m, `social_account`
minor ~18m, `event_record_*` minor ~6m:
amoro_table_optimizing_status_idle_duration_mills{catalog="saas_cluster8",database="cdp_entity",table="t_entity_data",group="k8s-amoro",}
1.568392645E9
amoro_table_optimizing_status_executing_duration_mills{catalog="saas_cluster8",database="cdp_entity",table="t_entity_data",group="k8s-amoro",}
0.0
amoro_table_optimizing_status_idle_duration_mills{catalog="saas_cluster8",database="beacon_olap",table="social_account",group="k8s-amoro",}
1.568392638E9
amoro_table_optimizing_status_executing_duration_mills{catalog="saas_cluster8",database="beacon_olap",table="social_account",group="k8s-amoro",}
0.0
```
### Anything else
This is a regression from
**[AMORO-3747](https://github.com/apache/amoro/issues/3747) /
[#3748](https://github.com/apache/amoro/pull/3748)** (new TableRuntime
storage). Dashboard still reads `table_runtime.status_code`, so the UI is
correct.
0.8 updated metrics on every status change:
```java
private void updateOptimizingStatus(OptimizingStatus status) {
this.optimizingStatus = status;
this.currentStatusStartTime = System.currentTimeMillis();
this.optimizingMetrics.statusChanged(status, currentStatusStartTime);
}
```
`#3748` removed `updateOptimizingStatus`.
`TableOptimizingMetrics.statusChanged(...)` still exists, but **nothing in the
tree calls it anymore**. The gauge stays at the constructor default
`OptimizingStatus.IDLE` and `statusSetTimestamp = now` at AMS start.
Suggested fix: after `store().updateStatusCode(...).commit()`, call
```java
optimizingMetrics.statusChanged(getOptimizingStatus(), /*
status_code_update_time */);
```
Also initialize from persisted `status_code` + `status_code_update_time`
when the runtime is created, so a restart does not show “idle since boot” while
the table is still MINOR.
This is separate from process rows stuck RUNNING after
`self-optimizing.enabled=false` (dashboard process list vs table header). Here
the **table header is already major/minor**; only Prometheus gauges are stale.
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
