[AMORO-4431][AMS] Close optimizing process instead of failing when self-optimizing is disabled

[zhang-arvin]

Root cause

Since 0.9, both the normal close path and a failed commit routed through completeProcess(false), so disabling self-optimizing (or cancelling) was recorded as a process failure. On top of that, when the in-memory OptimizingProcess was already gone (AMS restart / released process), the table_process row was never closed and stayed RUNNING forever:

• TableRuntimeRefreshExecutor.handleConfigChanged only closed the in-memory process, and only when getOptimizingProcess() != null. Later refreshes did not retry, so an orphan row stayed RUNNING.
• DefaultOptimizingService.cancelProcess returned false when the in-memory process was missing, but TableController.cancelOptimizingProcess still returned HTTP 200.
• OptimizingQueue...persistAndSetCompleted(false) / close() called completeProcess(false), which incremented amoro_table_optimizing_process_failed_count_total even for a user/config close.

Changes

• Distinguish CLOSED from FAILED — DefaultTableRuntime.completeProcess now takes the terminal ProcessStatus; TableOptimizingMetrics.processComplete(type, success, failed, planTime) only increments the failure counters for a genuine FAILED process. A CLOSED/CANCELED/KILLED process no longer counts as a failure.
• Close by process id from the store — new DefaultTableRuntime.closeProcessFromStore(processId) (and closeTrackedProcessFromStore()) persist-close a non-terminal table_process row owned by the table and reset the table to IDLE when no in-memory process exists.
• Retry while disabled — TableRuntimeRefreshExecutor calls the close on every refresh while self-optimizing stays disabled, so an orphan RUNNING row cannot survive.
• Cancel is no longer a silent no-op — DefaultOptimizingService.cancelProcess falls back to closing the tracked row by id; TableController.cancelOptimizingProcess fails (non-200) when cancel did not happen.
• Null-safety — completeProcess is now null-safe (previously NPE'd when the in-memory process was gone, as seen in OptimizingQueue.releaseTable).

Tests

TestDefaultOptimizingService:

• disabling self-optimizing with a live RUNNING process → row becomes CLOSED, table IDLE, process_failed_count unchanged;
• disabling self-optimizing when the in-memory process is already gone → orphan row becomes CLOSED, counter unchanged;
• cancel-by-id when the in-memory process is gone → returns true and closes the row.

mvn -pl amoro-ams -am test -Dtest=TestDefaultOptimizingService (33 tests) and the related TestIcebergV3OptimizingPlanning, TestTableRuntimeRefreshExecutor, TestTableSummary*, TestOverviewManager, TestTableManagerQuery suites pass; spotless:check passes.

Fixes #4431.

[j1wonpark]

Thanks for separating intentional closure from failure accounting.

question: Could we clarify how the reported process ends up with a RUNNING row but no in-memory OptimizingProcess? The log in #4431 shows an Iceberg commit validation failure, but does not establish that the runtime reference was missing. The new tests clear the reference via reflection, so they verify the fallback behavior rather than the failure path that creates this state.

I reproduced one possible path: if the outer transaction commit in persistAndSetCompleted fails after completeProcess has cleared the reference, the database rolls back to RUNNING while the reference remains null. This does not establish that the same failure occurred in the reported environment.

Could we add a regression test through the actual failure/recovery path, with the relevant logs or reproduction steps linking it to #4431? That would help us determine whether this needs a fix to state consistency, orphan cleanup, or both.