This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 3d61a3223961 CAMEL-26942: design - CLI short commands: telling an
agent to check is not enough, the checks run for it (#29904)
3d61a3223961 is described below
commit 3d61a32239616711eac483cda44ab30562d4e817
Author: Claus Ibsen <[email protected]>
AuthorDate: Sun Nov 15 13:11:47 2026 +0100
CAMEL-26942: design - CLI short commands: telling an agent to check is not
enough, the checks run for it (#29904)
## What
A follow-up to #29890 (design: Camel CLI short commands without a JVM
start). It adds the result of the benchmark that the design asked for. Docs
only.
The local-model benchmark (qwen3.6:35b, 53 beginner questions about 5
example apps, 3 rounds, 159 answers per run) ran without and with an
instruction at the start of each conversation: look options up with
`camel_catalog_doc`, and check every snippet with `camel_validate_source`
before you answer.
| per run of 159 answers | without | with |
|---|---|---|
| pass | 130 | 128 |
| "what does this app do" (always the first question) | 8/15 | 12/15 |
| "how can I make it better" | 12/15 | 5/15 |
| code blocks in the answers | 84 | 221 |
| snippets validated | 0 | 0 |
| options that do not exist | 6 in 4 answers | 32 in 14 answers |
| compactions of the history | 6 | 13 |
| model time per answer | 5.8 s | 8.1 s |
The instruction faded after the first question. Most of the gain on the
first question is one app: in the first run the model answered from the
AI-written summary that the readme tool returned then (fixed by CAMEL-26946,
#29891).
## Change
- **Motivation:** the two runs, with the caveat that this is one model and
one run each.
- **Summary and recommendation:** telling an agent to check is not enough,
so the checks run without its choice.
- The AI panel and `camel ask` mark the mistakes in the code of an answer
(CAMEL-26960).
- The MCP server already validates a file before `camel_write_file`
writes it.
- An agent that edits files with its own tools needs a hook that runs
`camel validate yaml` on each edited route file. That hook runs on every edit,
so it is where the daemon saves the most.
- **Measure the agents:** the next runs are a line in the system prompt of
the AI panel (two jars that differ only in that line) and a run with the marks
of CAMEL-26960.
- Links CAMEL-26961 (the daemon itself) and CAMEL-26960.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
design/cli-short-commands.adoc | 62 ++++++++++++++++++++++++++++++++++--------
1 file changed, 51 insertions(+), 11 deletions(-)
diff --git a/design/cli-short-commands.adoc b/design/cli-short-commands.adoc
index 4794abd5b3ed..3205f27f6eb3 100644
--- a/design/cli-short-commands.adoc
+++ b/design/cli-short-commands.adoc
@@ -5,10 +5,12 @@ authors:
reviewers: []
approvers: []
creation-date: 2026-11-13
-last-updated: 2026-11-13
+last-updated: 2026-11-14
status: proposed
see-also:
- "CAMEL-26942"
+ - "CAMEL-26961"
+ - "CAMEL-26960"
- "CAMEL-26935"
- "CAMEL-26937"
- "CAMEL-26938"
@@ -38,8 +40,11 @@ against 10-40 ms estimated), for a fraction of the build and
release work.
The seconds a daemon saves are small next to the turns of the model. The
reason to build it is that a command that
answers at once can be called after every step. An agent makes fewer mistakes
when it validates after every edit, looks
an option up with `camel doc` instead of guessing it, and checks `camel
status` after every reload. Speed alone
-does not make an agent do that, it has to be told (see the motivation), but
speed makes it cheap to do. With the Java
-client the daemon answers in 50-65 ms, under the 0.1 s at which a response
feels instant, like a native command.
+does not make an agent do that, and telling it to is not enough. In a
benchmark with a local model, an instruction to
+check made the model read more, but it checked none of the code it wrote (see
the motivation). The checks have to run
+whether the model asks for them or not, for example in a hook of the agent
that validates every edited file. A check
+that runs on every edit is where a fast command pays the most. With the Java
client the daemon answers in 50-65 ms,
+under the 0.1 s at which a response feels instant, like a native command.
== Motivation
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc
and got it right. It s
exist, and validated nothing in 9 answers. A lookup also costs context: the
doc of the file component is 17K
characters, and three docs in one answer forced a compaction of the history.
+Two full runs on the night of 2026-11-13 then asked the same model 53
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was
always "what does this app do". The
+ model read more for it (3.9 tool calls per answer instead of 2.1) and passed
it 12 of 15 times instead of 8. Most
+ of that gain is one app. In the first run, the model answered from the
AI-written summary that the readme tool
+ returned, and left a step out. Since CAMEL-26946 that tool returns the facts
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no
validation per answer. Yet the model wrote
+ 2.6 times as much code in all (221 code blocks instead of 84) and validated
none of it. Its 3 calls of
+ `camel_validate_source`, all on the first question, checked the route file
of the app, not a snippet. Options that
+ do not exist went from 6 in 4 answers to 32 in 14, all after the first
question. "How can I make it better" passed
+ 5 of 15 times instead of 12.
+- It cost context and time: 13 compactions of the history instead of 6, and
8.1 s of model time per answer instead
+ of 5.8 s.
+- In all, 128 answers passed instead of 130: 4 more on the first question, 6
fewer on the others.
+
+This is one model and one run each. The changes in code, invented options,
compactions and time are well beyond the
+variance between the rounds. Most changes in the pass rate of a single
question are not.
+
+Telling the model to check made it read more and write more, not check what it
wrote. The check of the code in an
+answer has to run whether the model asks for it or not. CAMEL-26960 proposes
that the AI panel and `camel ask` mark
+the mistakes in the code of an answer. One such mistake is an option that does
not exist.
+
== Goals
- The short commands an agent runs most answer in under 0.1 s, the usual limit
for a response to feel instant, as a
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the
start of the daemon itself.
readiness pushed over the WebSocket transport.
. Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a
response feels instant, so the commands
already feel native. A native client for the main platforms, with the Java
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so
what agents read (the Camel CLI pages,
- the instructions of the MCP server, the system prompt of the AI panel) says
that the short commands are cheap to
- call: look an option up instead of guessing it, asking only for the options
needed (the `optionsFilter` of
- `camel_catalog_doc`) as a whole doc page costs context, and validate every
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an
agent uses the CLI. The Camel CLI pages,
+ the MCP server's instructions and the AI panel's system prompt say that the
short commands are cheap to call. They
+ tell an agent to look an option up instead of guessing it. They also tell it
to ask only for the options it needs
+ (the `optionsFilter` of `camel_catalog_doc`), since a whole doc page costs
context. Telling is not enough for the
+ checks: told to validate every snippet, the local model wrote more code and
validated none of it (see the
+ motivation). So the checks run without the agent's choice. The AI panel and
`camel ask` mark the mistakes in the
+ code of an answer (CAMEL-26960). The MCP server already validates a file
before `camel_write_file` writes it. An
+ agent that edits files with its own tools needs a hook that runs `camel
validate yaml` on each edited route file.
+ The Camel CLI pages show one. That hook runs on every edit, so it is where
the daemon saves the most.
. Measure the agents, not only the commands. The local-model benchmark runs in
the AI panel of the TUI, whose tools
- run in the process, so it measures the instruction, not the daemon: run it
with and without the instruction, and
- compare the pass rate, the calls and the invented options per answer, the
compactions and the time of the model.
- The daemon needs a benchmark that drives `camel` through a shell. Make the
daemon the default only after a release
- as opt-in, and when that benchmark shows the gain.
+ run in the process, so it measures the instruction and the checks, not the
daemon. It compares the pass rate, the
+ calls and the invented options per answer, the compactions and the time of
the model. The first run had the
+ instruction in the first message (see the motivation). The next run puts
such a line in the system prompt of the
+ AI panel, where it may last past the first question. It compares two jars
that differ only in that line. A run
+ after that uses the marks of CAMEL-26960 and counts the invented options
shown without a mark. The daemon needs a
+ benchmark that drives `camel` through a shell. Make the daemon the default
only after a release as opt-in, and when
+ that benchmark shows the gain.
. Do not build the native front. A daemon with a native client is in the same
range (21-37 ms measured, against
10-40 ms estimated), and the native front costs the most: the base classes
of the commands change, GraalVM metadata
must follow every command and option, aesh needs a change, and each release
needs a binary per platform, all before
@@ -348,12 +386,14 @@ measure.
== Related tickets
- CAMEL-26942: this design.
+- CAMEL-26961: the opt-in resident daemon of this design, in its stages.
- CAMEL-26935: make the Camel CLI faster for AI agents, the parent of the
sub-tasks below.
- CAMEL-26936: `camel run --background` learns at once that Camel has started.
- CAMEL-26937: faster startup of the short commands (no JLine terminal or
log4j when not needed, C1).
- CAMEL-26938: shorter action polls.
- CAMEL-26940: a JDK 25 AOT cache instead of a CDS archive per command.
- CAMEL-26941: `camel status`, the state of an integration in one call.
+- CAMEL-26960: mark the mistakes in the code of AI answers, such as options
that do not exist.
- CAMEL-26913: CLI startup and config (CDS archives shared between installs).
- CAMEL-25388: the Camel CLI reliable for AI agents and scripts.
- CAMEL-23810: the port of the CLI to aesh.