This is an automated email from the ASF dual-hosted git repository.
wenjin272 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 0cce08277 [api][plan] Abstain on runtime non-candidate model
selections (#1205)
0cce08277 is described below
commit 0cce0827731eeb616a8972b1faf8b9021702f3f6
Author: Sangkyoon Nam <[email protected]>
AuthorDate: Thu Oct 8 14:57:02 2026 +0900
[api][plan] Abstain on runtime non-candidate model selections (#1205)
Generated-by: Claude Code 2.1.292 (Claude Opus 5.5)
---
.../chat/model/routing/CustomRoutingExecutor.java | 6 +-
.../agents/api/chat/model/routing/ModelRouter.java | 3 +-
.../api/chat/model/routing/RoutingDecision.java | 8 +-
docs/content/docs/development/model_routing.md | 10 +-
.../flink/agents/plan/actions/ChatModelAction.java | 2 +-
.../plan/routing/LlmJudgeRoutingExecutor.java | 6 +-
.../agents/plan/routing/ModelRoutingResolver.java | 58 +++++--
.../plan/actions/ChatModelActionRoutingTest.java | 170 ++++++++++++++++++++-
8 files changed, 229 insertions(+), 34 deletions(-)
diff --git
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/CustomRoutingExecutor.java
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/CustomRoutingExecutor.java
index 6f1ec847c..d539e9b66 100644
---
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/CustomRoutingExecutor.java
+++
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/CustomRoutingExecutor.java
@@ -42,8 +42,10 @@ public interface CustomRoutingExecutor extends Serializable {
*
* @param strategy the declaration this executor was configured with (type
+ arguments)
* @param context the request messages, prompt args, and candidates
- * @return the routing decision (selected candidate or abstain)
- * @throws Exception if the executor fails
+ * @return the routing decision (selected candidate or abstain); a name
outside the candidates
+ * is treated as abstain
+ * @throws Exception if the executor fails; the request then fails and
does not fall back to the
+ * default model
*/
RoutingDecision route(RoutingStrategy strategy, RoutingContext context)
throws Exception;
}
diff --git
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/ModelRouter.java
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/ModelRouter.java
index a9de9b0c1..bdab0fad9 100644
---
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/ModelRouter.java
+++
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/ModelRouter.java
@@ -45,7 +45,8 @@ import java.util.Optional;
* engine's plan layer (the {@code RoutingExecutor} implementations).
*
* <p>Abstain ({@link RoutingDecision#abstain()}) → {@link
#getDefaultModel()}. A returned name that
- * is not a candidate is an invalid decision and is failed clearly by the
caller.
+ * is not a candidate is treated as abstain, with the rejected name kept in
the routing event's
+ * metadata. Declaration errors (a non-candidate default or rule target) still
fail at build time.
*/
public class ModelRouter extends Resource {
diff --git
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/RoutingDecision.java
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/RoutingDecision.java
index c711eea6a..c52cbfffd 100644
---
a/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/RoutingDecision.java
+++
b/api/src/main/java/org/apache/flink/agents/api/chat/model/routing/RoutingDecision.java
@@ -33,9 +33,11 @@ import java.util.Map;
* explanation (reason, score, metadata), or signals abstention so the router
falls back to its
* default model.
*
- * <p>Returning a name that is not one of the router's candidates is an
<b>invalid</b> decision and
- * is failed clearly by the router (not silently defaulted). Abstention
({@link #abstain()}) is the
- * intended way to defer to the default model.
+ * <p>Abstention ({@link #abstain()}) is the intended way to defer to the
default model. A name that
+ * is not one of the router's candidates is treated as abstention: the router
selects its default
+ * model and records the rejected name and reason in the routing event's
metadata ({@code
+ * rejected_model}, {@code rejected_reason}). This applies to fresh decisions
and to persisted
+ * decisions replayed after the candidate set changed.
*
* <p>JSON-serializable so it can be persisted/replayed as a durable {@code
"route"} call result.
*/
diff --git a/docs/content/docs/development/model_routing.md
b/docs/content/docs/development/model_routing.md
index 27864e32f..de706bc53 100644
--- a/docs/content/docs/development/model_routing.md
+++ b/docs/content/docs/development/model_routing.md
@@ -127,7 +127,7 @@ Resources can equally be registered on the execution
environment with `agentsEnv
## Routing Strategies
-A strategy is declared with a `Strategies` factory and travels with the agent
plan. Every strategy either selects a candidate or abstains. A rule or custom
strategy that selects a name that is not a candidate is an error: the request
produces a failed `ChatResponseEvent`, and never falls back to the default. The
LLM judge is the exception, because its reply is untrusted model output: a
verdict naming a non-candidate abstains.
+A strategy is declared with a `Strategies` factory and travels with the agent
plan. Every strategy either selects a candidate or abstains. A runtime result
that names a model outside the candidates is treated as abstention and lands on
the default model, whichever strategy produced it. A fresh judge verdict
outside the candidates abstains inside the judge with the reason `judge verdict
was not a candidate name`. Any other non-candidate decision is normalized by
the router: the routing ev [...]
### Rules
@@ -250,7 +250,7 @@ An attempt fails when the model call throws, when the
candidate does not resolve
Each candidate and the routing judge have up to `1 + max-retries` attempts,
with exponential backoff from `retry-wait-interval`. The default retry budget
is 0. Candidate fallback is independent of retries.
-A judge call that exhausts its retries, a throwing rule/custom strategy, or a
strategy selecting a non-candidate produces a failed `ChatResponseEvent`; these
failures do not count as abstention. A normal abstaining verdict still selects
the default model. If all attempted candidates fail, the terminal failed
response contains the last candidate's exception type and message. Earlier
failures remain available in diagnostics. The retry settings are the job-level
options in [Configuration]({ [...]
+A judge call that exhausts its retries or a throwing rule/custom strategy
produces a failed `ChatResponseEvent`; these failures do not count as
abstention. A normal abstaining verdict still selects the default model. If all
attempted candidates fail, the terminal failed response contains the last
candidate's exception type and message. Earlier failures remain available in
diagnostics. The retry settings are the job-level options in
[Configuration]({{< ref "docs/operations/configuration#c [...]
## Observability
@@ -262,7 +262,7 @@ Every accepted routing decision emits a
`ModelRoutingEvent`, event type `_model_
| `selected_model` | The model the decision selected. |
| `decision_source` | `strategy` for a rule or custom selection, `llm_judge`
for a judge verdict, `default` when the strategy abstained, `fallback` on the
second event. |
| `fallback_enabled` | Whether fallback is configured, not whether it
happened. |
-| `reason`, `score`, `metadata` | Set by the strategy. Rules record `matched
rule: <pattern>`; the judge records `judge_model`, the judge's token counts
when reported, and `judge_context_truncated` when history was dropped. |
+| `reason`, `score`, `metadata` | Set by the strategy. Rules record `matched
rule: <pattern>`; the judge records `judge_model`, the judge's token counts
when reported, and `judge_context_truncated` when history was dropped. When the
router normalizes a non-candidate decision to its default, `metadata` adds
`rejected_model` and, if the strategy gave one, `rejected_reason`; the
strategy's score is dropped. |
| `decision_ms` | Decision latency. For the judge it includes the judge call
and its retries. |
The final response message carries a `model_routing` map in its extra
arguments with the routing details and the fallback result: `final_model`, the
model that answered, `fallback_attempted`, and `fallback_models_tried`.
@@ -289,9 +289,9 @@ Decision latency is recorded as the
`action.chat_model_action.routingDecisionLat
## Advanced
-**Validation.** Declaration mistakes such as an invalid rule pattern, an
unknown candidate in a rule or description, or a missing strategy fail at the
builder calls. Whether the judge is a plain chat model and whether a custom
executor class can be loaded is checked when the plan is built, before any
record is processed. Whether the default model is a candidate is only checked
on the TaskManager, when the router first handles a request; an invalid router
configuration then produces a fai [...]
+**Validation.** Declaration mistakes such as an invalid rule pattern, an
unknown candidate in a rule or description, or a missing strategy fail at the
builder calls. Whether the judge is a plain chat model and whether a custom
executor class can be loaded is checked when the plan is built, before any
record is processed. Whether the default model is a candidate is checked at
`build()` and again when the plan is built, so a descriptor that skipped the
builder also fails before any record [...]
-**Recovery.** Routing runs once per request. When an [action state store]({{<
ref "docs/operations/configuration#action-state-store" >}}) is configured, the
routing decision, the judge call, and each candidate attempt of the initial
request are persisted and replayed on recovery, so a custom executor or judge
is not run again for a request that already has a decision. Without a store,
which is the default, the decision is recomputed on recovery, and a
non-deterministic strategy may pick [...]
+**Recovery.** Routing runs once per request. When an [action state store]({{<
ref "docs/operations/configuration#action-state-store" >}}) is configured, the
routing decision, the judge call, and each candidate attempt of the initial
request are persisted and replayed on recovery, so a rule or custom executor is
not run again for a request that already has a decision. The judge executor
does run again, but it reads the persisted judge call instead of invoking the
judge model, and the stor [...]
## Examples
diff --git
a/plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java
b/plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java
index 53d86c7f0..0e6df1ddb 100644
---
a/plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java
+++
b/plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java
@@ -60,7 +60,7 @@ import static
org.apache.flink.agents.api.agents.Agent.STRUCTURED_OUTPUT;
*
* <ol>
* <li><b>Decide</b> — {@link ModelRoutingResolver} runs the router's
strategy and normalizes the
- * result (abstain → default model; non-candidate → fail).
+ * result (abstain or non-candidate → default model; a strategy
exception → fail).
* <li><b>Durably</b> — the strategy runs inside a durable call ({@code
"route:<router>"};
* per-request uniqueness comes from the store's (key, sequence, event,
action) scoping, and
* the id must stay deterministic across recovery re-processing), so
recovery replays the
diff --git
a/plan/src/main/java/org/apache/flink/agents/plan/routing/LlmJudgeRoutingExecutor.java
b/plan/src/main/java/org/apache/flink/agents/plan/routing/LlmJudgeRoutingExecutor.java
index 864371e30..c25d2666c 100644
---
a/plan/src/main/java/org/apache/flink/agents/plan/routing/LlmJudgeRoutingExecutor.java
+++
b/plan/src/main/java/org/apache/flink/agents/plan/routing/LlmJudgeRoutingExecutor.java
@@ -163,9 +163,9 @@ final class LlmJudgeRoutingExecutor implements
RoutingExecutor {
}
return builder.build();
}
- // Persisted as a real abstain: replay resolves to the router's
*current* default, so
- // a candidate-set change across a restart degrades gracefully (like
the strategy
- // path) instead of failing the non-candidate guard.
+ // Persisted as a real abstain: replay resolves to the router's
*current* default. A
+ // stored verdict whose candidate was removed before a restart gets
the same treatment
+ // from the resolver's non-candidate normalization.
return new RoutingDecision(
null, true, abstainReason, null, new HashMap<>(judgeMetadata),
null);
}
diff --git
a/plan/src/main/java/org/apache/flink/agents/plan/routing/ModelRoutingResolver.java
b/plan/src/main/java/org/apache/flink/agents/plan/routing/ModelRoutingResolver.java
index c79047d43..977a82a3d 100644
---
a/plan/src/main/java/org/apache/flink/agents/plan/routing/ModelRoutingResolver.java
+++
b/plan/src/main/java/org/apache/flink/agents/plan/routing/ModelRoutingResolver.java
@@ -29,6 +29,7 @@ import
org.apache.flink.agents.api.metrics.FlinkAgentsMetricGroup;
import org.apache.flink.agents.api.resource.ResourceType;
import org.apache.flink.agents.plan.actions.ChatModelInvoker;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
@@ -55,6 +56,15 @@ public final class ModelRoutingResolver {
return "route:" + model;
}
+ /**
+ * Decision-metadata key naming the model a strategy selected outside the
router's candidates
+ * when that selection was normalized to abstain.
+ */
+ public static final String REJECTED_MODEL_KEY = "rejected_model";
+
+ /** Decision-metadata key carrying the strategy's own reason for a
rejected selection. */
+ public static final String REJECTED_REASON_KEY = "rejected_reason";
+
private ModelRoutingResolver() {}
/** An ordinary routing failure captured at the strategy boundary. */
@@ -91,9 +101,9 @@ public final class ModelRoutingResolver {
/**
* If {@code model} names a {@link ModelRouter}, execute its declared
strategy (persisted under
* the durable {@code "route"} call so the decision replays
deterministically on recovery),
- * normalize the result (abstain -> default model, non-candidate -> fail
clearly), emit an
- * observability-only {@link ModelRoutingEvent}, and return the selected
concrete model.
- * Otherwise returns a direct selection.
+ * normalize the result (abstain or non-candidate -> default model), emit
an observability-only
+ * {@link ModelRoutingEvent}, and return the selected concrete model.
Otherwise returns a direct
+ * selection.
*
* <p>Routing runs once for the initial chat request; tool-call rounds
reuse the selected
* concrete model because it is saved in the tool-request context (see
{@code
@@ -231,10 +241,13 @@ public final class ModelRoutingResolver {
}
/**
- * Shared post-durable handling for all executors: abstain resolves to the
router's
- * <i>current</i> default (so a persisted abstain replays gracefully
across candidate changes),
- * a concrete selection is guarded against the current candidate set, and
the observability
- * event and resolved route are emitted.
+ * Shared post-durable handling for all executors, applied identically to
fresh and replayed
+ * decisions against the router's <i>current</i> candidates and default. A
concrete selection
+ * outside the candidate set is normalized to abstain, the same contract
the judge applies to a
+ * non-candidate verdict; abstain resolves to the current default, so a
persisted decision
+ * replays gracefully across candidate changes. The rejected name and the
strategy's reason are
+ * kept in the decision metadata for the routing event. Normalization
happens after persistence,
+ * so the stored record keeps the strategy's original decision.
*/
private static ResolvedModelRoute normalizeAndFinish(
UUID requestId,
@@ -243,6 +256,9 @@ public final class ModelRoutingResolver {
RoutingDecision decision,
String concreteSource,
RunnerContext ctx) {
+ if (!decision.isAbstain() &&
!router.isCandidate(decision.getSelectedModel())) {
+ decision = rejectNonCandidate(decision);
+ }
String selectedModel;
String decisionSource;
if (decision.isAbstain()) {
@@ -250,18 +266,32 @@ public final class ModelRoutingResolver {
decisionSource = ModelRoutingEvent.SOURCE_DEFAULT;
} else {
selectedModel = decision.getSelectedModel();
- if (!router.isCandidate(selectedModel)) {
- throw new RoutingFailure(
- String.format(
- "Routing decision for router '%s' selected
non-candidate model '%s'; candidates are %s.",
- model, selectedModel,
router.getCandidateNames()));
- }
decisionSource = concreteSource;
}
return finish(requestId, model, router, decision, selectedModel,
decisionSource, ctx);
}
- /** Records the decision latency histogram sample (also for decisions the
guards reject). */
+ /**
+ * An abstaining copy of a non-candidate decision. The score is dropped
because it rated the
+ * rejected selection, not the default; the latency and the strategy's
metadata are kept.
+ */
+ private static RoutingDecision rejectNonCandidate(RoutingDecision
decision) {
+ String rejected = decision.getSelectedModel();
+ Map<String, Object> metadata = new HashMap<>(decision.getMetadata());
+ metadata.put(REJECTED_MODEL_KEY, rejected);
+ if (decision.getReason() != null) {
+ metadata.put(REJECTED_REASON_KEY, decision.getReason());
+ }
+ return new RoutingDecision(
+ null,
+ true,
+ "selected model '" + rejected + "' is not a candidate",
+ null,
+ metadata,
+ decision.getDecisionMs());
+ }
+
+ /** Records the decision latency histogram sample (also for decisions
normalized to abstain). */
private static void recordDecisionLatency(RunnerContext ctx,
RoutingDecision decision) {
Double decisionMs = decision.getDecisionMs();
FlinkAgentsMetricGroup actionMetrics = ctx.getActionMetricGroup();
diff --git
a/plan/src/test/java/org/apache/flink/agents/plan/actions/ChatModelActionRoutingTest.java
b/plan/src/test/java/org/apache/flink/agents/plan/actions/ChatModelActionRoutingTest.java
index 73e37a750..fb79bfee9 100644
---
a/plan/src/test/java/org/apache/flink/agents/plan/actions/ChatModelActionRoutingTest.java
+++
b/plan/src/test/java/org/apache/flink/agents/plan/actions/ChatModelActionRoutingTest.java
@@ -47,6 +47,7 @@ import
org.apache.flink.agents.api.resource.ResourceDescriptor;
import org.apache.flink.agents.api.resource.ResourceType;
import org.apache.flink.agents.api.tools.ToolResponse;
import org.apache.flink.agents.plan.AgentConfiguration;
+import org.apache.flink.agents.plan.routing.ModelRoutingResolver;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Histogram;
import org.junit.jupiter.api.Test;
@@ -70,13 +71,17 @@ import static org.mockito.Mockito.when;
/** Integration tests for model routing inside {@link ChatModelAction}. */
public class ChatModelActionRoutingTest {
- /** An executor that returns a name that is not a candidate (to exercise
the invalid path). */
+ /** An executor that returns a name that is not a candidate (to exercise
the abstain path). */
public static class SelectsUnknownStrategy implements
CustomRoutingExecutor {
public SelectsUnknownStrategy() {}
@Override
public RoutingDecision route(RoutingStrategy strategy, RoutingContext
context) {
- return RoutingDecision.of("nonexistent");
+ return RoutingDecision.builder("nonexistent")
+ .reason("tenant override")
+ .score(0.9)
+ .metadata("tenant", "t1")
+ .build();
}
}
@@ -445,8 +450,12 @@ public class ChatModelActionRoutingTest {
.isEqualTo(ModelRoutingEvent.SOURCE_STRATEGY);
}
+ /**
+ * A custom strategy naming a non-candidate abstains to the default, like
a non-candidate judge
+ * verdict; the rejected name and the strategy's own reason stay visible
on the routing event.
+ */
@Test
- void invalidCandidateFailsClearly() throws Exception {
+ void customNonCandidateSelectionAbstainsToDefault() throws Exception {
ModelRouter router =
new ModelRouter(
ModelRouter.of("small", "big")
@@ -458,8 +467,41 @@ public class ChatModelActionRoutingTest {
ChatModelAction.processChatRequestOrToolResponse(
new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hi"))),
ctx);
- assertThat(ctx.chatResponse().isFailed()).isTrue();
- assertThat(ctx.chatResponse().getError()).contains("non-candidate");
+
+ assertThat(ctx.chatResponse().isFailed()).isFalse();
+ ModelRoutingEvent event = ctx.routingEvent();
+ assertThat(event.getSelectedModel()).isEqualTo("small");
+
assertThat(event.getDecisionSource()).isEqualTo(ModelRoutingEvent.SOURCE_DEFAULT);
+ assertThat(event.getReason()).contains("not a candidate");
+ assertThat(event.getMetadata())
+ .containsEntry(ModelRoutingResolver.REJECTED_MODEL_KEY,
"nonexistent")
+ .containsEntry(ModelRoutingResolver.REJECTED_REASON_KEY,
"tenant override")
+ .containsEntry("tenant", "t1");
+ // the score belonged to the rejected selection, not to the default
+ assertThat(event.getScore()).isNull();
+ assertThat(event.getDecisionMs()).isNotNull();
+ assertThat(ctx.resolvedChatModels).containsExactly("small");
+ }
+
+ /** Without a configured default, a non-candidate selection lands on the
first candidate. */
+ @Test
+ void customNonCandidateSelectionWithoutDefaultUsesFirstCandidate() throws
Exception {
+ ModelRouter router =
+ new ModelRouter(
+ ModelRouter.of("small", "big")
+
.strategy(Strategies.custom(SelectsUnknownStrategy.class))
+ .build(),
+ null);
+ FakeRunnerContext ctx = new FakeRunnerContext(router);
+ ChatModelAction.processChatRequestOrToolResponse(
+ new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hi"))),
+ ctx);
+
+ assertThat(ctx.chatResponse().isFailed()).isFalse();
+ assertThat(ctx.routingEvent().getSelectedModel()).isEqualTo("small");
+ assertThat(ctx.routingEvent().getMetadata())
+ .containsEntry(ModelRoutingResolver.REJECTED_MODEL_KEY,
"nonexistent");
+ assertThat(ctx.resolvedChatModels).containsExactly("small");
}
@Test
@@ -577,6 +619,11 @@ public class ChatModelActionRoutingTest {
assertThat(ctx.routingEvent().getSelectedModel()).isEqualTo("small");
assertThat(ctx.routingEvent().getDecisionSource())
.isEqualTo(ModelRoutingEvent.SOURCE_DEFAULT);
+ assertThat(ctx.routingEvent().getReason())
+ .isEqualTo("judge verdict was not a candidate name");
+ // the judge abstains on its own; the resolver's non-candidate
normalization is not involved
+ assertThat(ctx.routingEvent().getMetadata())
+ .doesNotContainKey(ModelRoutingResolver.REJECTED_MODEL_KEY);
}
@Test
@@ -1273,6 +1320,119 @@ public class ChatModelActionRoutingTest {
assertThat(ctx.resolvedChatModels).containsExactly("big");
}
+ /**
+ * A stored concrete decision whose model was removed from the candidates
before recovery is
+ * normalized like a fresh non-candidate selection: it abstains to the
router's current default,
+ * keeps its original latency, and records the rejected name. The strategy
is not re-run.
+ */
+ @Test
+ void storedDecisionForRemovedCandidateAbstainsToCurrentDefault() throws
Exception {
+ ModelRouter router =
+ new ModelRouter(
+ ModelRouter.of("small", "medium")
+
.strategy(Strategies.custom(ExplodingStrategy.class))
+ .defaultModel("medium")
+ .build(),
+ null);
+ RoutingDecision stored =
+
RoutingDecision.builder("big").reason("stored").build().withDecisionMs(42.0);
+ FakeRunnerContext ctx = new
FakeRunnerContext(router).seedDurable("route:router", stored);
+
+ ChatModelAction.processChatRequestOrToolResponse(
+ new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hi"))),
+ ctx);
+
+ assertThat(ctx.chatResponse().isFailed()).isFalse();
+ ModelRoutingEvent event = ctx.routingEvent();
+ assertThat(event.getSelectedModel()).isEqualTo("medium");
+
assertThat(event.getDecisionSource()).isEqualTo(ModelRoutingEvent.SOURCE_DEFAULT);
+ assertThat(event.getMetadata())
+ .containsEntry(ModelRoutingResolver.REJECTED_MODEL_KEY, "big")
+ .containsEntry(ModelRoutingResolver.REJECTED_REASON_KEY,
"stored");
+ assertThat(event.getDecisionMs()).isEqualTo(42.0);
+ assertThat(ctx.resolvedChatModels).containsExactly("medium");
+ }
+
+ /**
+ * The same stored non-candidate decision follows whatever default the
router has at recovery: a
+ * changed default, or the first candidate when none is configured.
+ */
+ @Test
+ void storedNonCandidateDecisionFollowsCurrentDefault() throws Exception {
+ RoutingDecision stored = RoutingDecision.of("removed");
+
+ ModelRouter changedDefault =
+ new ModelRouter(
+ ModelRouter.of("small", "big")
+
.strategy(Strategies.custom(ExplodingStrategy.class))
+ .defaultModel("big")
+ .build(),
+ null);
+ FakeRunnerContext first =
+ new
FakeRunnerContext(changedDefault).seedDurable("route:router", stored);
+ ChatModelAction.processChatRequestOrToolResponse(
+ new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hi"))),
+ first);
+ assertThat(first.chatResponse().isFailed()).isFalse();
+ assertThat(first.routingEvent().getSelectedModel()).isEqualTo("big");
+ assertThat(first.routingEvent().getMetadata())
+ .containsEntry(ModelRoutingResolver.REJECTED_MODEL_KEY,
"removed");
+ assertThat(first.resolvedChatModels).containsExactly("big");
+
+ ModelRouter noDefault =
+ new ModelRouter(
+ ModelRouter.of("small", "big")
+
.strategy(Strategies.custom(ExplodingStrategy.class))
+ .build(),
+ null);
+ FakeRunnerContext second =
+ new FakeRunnerContext(noDefault).seedDurable("route:router",
stored);
+ ChatModelAction.processChatRequestOrToolResponse(
+ new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hi"))),
+ second);
+ assertThat(second.chatResponse().isFailed()).isFalse();
+
assertThat(second.routingEvent().getSelectedModel()).isEqualTo("small");
+ assertThat(second.resolvedChatModels).containsExactly("small");
+ }
+
+ /**
+ * A stored judge verdict naming a candidate that was removed before
recovery gets the same
+ * treatment as a custom strategy's: abstain to the current default,
rejected name recorded. The
+ * judge chat replays from its own record and the judge model is not
re-invoked.
+ */
+ @Test
+ void storedJudgeVerdictForRemovedCandidateAbstainsToCurrentDefault()
throws Exception {
+ ModelRouter router =
+ new ModelRouter(
+ ModelRouter.of("small", "medium")
+ .strategy(Strategies.llm("judge"))
+ .defaultModel("medium")
+ .build(),
+ null);
+ RoutingDecision stored = RoutingDecision.builder("big").reason("llm
judge verdict").build();
+ FakeRunnerContext ctx =
+ new FakeRunnerContext(router)
+ .register(
+ "judge",
+ new FakeChatModel(
+ new RuntimeException("judge re-invoked
on replay")))
+ .seedDurable(
+ "judge:router",
+ new ChatMessage(MessageRole.ASSISTANT,
"{\"model\": \"big\"}"))
+ .seedDurable("route:router", stored);
+ ChatModelAction.processChatRequestOrToolResponse(
+ new ChatRequestEvent("router", List.of(new
ChatMessage(MessageRole.USER, "hard"))),
+ ctx);
+
+ assertThat(ctx.chatResponse().isFailed()).isFalse();
+ ModelRoutingEvent event = ctx.routingEvent();
+ assertThat(event.getSelectedModel()).isEqualTo("medium");
+
assertThat(event.getDecisionSource()).isEqualTo(ModelRoutingEvent.SOURCE_DEFAULT);
+ assertThat(event.getMetadata())
+ .containsEntry(ModelRoutingResolver.REJECTED_MODEL_KEY, "big")
+ .containsEntry(ModelRoutingResolver.REJECTED_REASON_KEY, "llm
judge verdict");
+ }
+
/**
* Replay for a judge router (review: both replay tests seeded a CUSTOM
strategy, which takes
* the simple durable path). The judge path persists two records — the
judge chat under {@code
