This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/main by this push:
     new 3d61a3223961 CAMEL-26942: design - CLI short commands: telling an 
agent to check is not enough, the checks run for it (#29904)
3d61a3223961 is described below

commit 3d61a32239616711eac483cda44ab30562d4e817
Author: Claus Ibsen <[email protected]>
AuthorDate: Sun Nov 15 13:11:47 2026 +0100

    CAMEL-26942: design - CLI short commands: telling an agent to check is not 
enough, the checks run for it (#29904)
    
    ## What
    
    A follow-up to #29890 (design: Camel CLI short commands without a JVM 
start). It adds the result of the benchmark that the design asked for. Docs 
only.
    
    The local-model benchmark (qwen3.6:35b, 53 beginner questions about 5 
example apps, 3 rounds, 159 answers per run) ran without and with an 
instruction at the start of each conversation: look options up with 
`camel_catalog_doc`, and check every snippet with `camel_validate_source` 
before you answer.
    
    | per run of 159 answers | without | with |
    |---|---|---|
    | pass | 130 | 128 |
    | "what does this app do" (always the first question) | 8/15 | 12/15 |
    | "how can I make it better" | 12/15 | 5/15 |
    | code blocks in the answers | 84 | 221 |
    | snippets validated | 0 | 0 |
    | options that do not exist | 6 in 4 answers | 32 in 14 answers |
    | compactions of the history | 6 | 13 |
    | model time per answer | 5.8 s | 8.1 s |
    
    The instruction faded after the first question. Most of the gain on the 
first question is one app: in the first run the model answered from the 
AI-written summary that the readme tool returned then (fixed by CAMEL-26946, 
#29891).
    
    ## Change
    
    - **Motivation:** the two runs, with the caveat that this is one model and 
one run each.
    - **Summary and recommendation:** telling an agent to check is not enough, 
so the checks run without its choice.
      - The AI panel and `camel ask` mark the mistakes in the code of an answer 
(CAMEL-26960).
      - The MCP server already validates a file before `camel_write_file` 
writes it.
      - An agent that edits files with its own tools needs a hook that runs 
`camel validate yaml` on each edited route file. That hook runs on every edit, 
so it is where the daemon saves the most.
    - **Measure the agents:** the next runs are a line in the system prompt of 
the AI panel (two jars that differ only in that line) and a run with the marks 
of CAMEL-26960.
    - Links CAMEL-26961 (the daemon itself) and CAMEL-26960.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
 design/cli-short-commands.adoc | 62 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 51 insertions(+), 11 deletions(-)

diff --git a/design/cli-short-commands.adoc b/design/cli-short-commands.adoc
index 4794abd5b3ed..3205f27f6eb3 100644
--- a/design/cli-short-commands.adoc
+++ b/design/cli-short-commands.adoc
@@ -5,10 +5,12 @@ authors:
 reviewers: []
 approvers: []
 creation-date: 2026-11-13
-last-updated: 2026-11-13
+last-updated: 2026-11-14
 status: proposed
 see-also:
   - "CAMEL-26942"
+  - "CAMEL-26961"
+  - "CAMEL-26960"
   - "CAMEL-26935"
   - "CAMEL-26937"
   - "CAMEL-26938"
@@ -38,8 +40,11 @@ against 10-40 ms estimated), for a fraction of the build and 
release work.
 The seconds a daemon saves are small next to the turns of the model. The 
reason to build it is that a command that
 answers at once can be called after every step. An agent makes fewer mistakes 
when it validates after every edit, looks
 an option up with `camel doc` instead of guessing it, and checks `camel 
status` after every reload. Speed alone
-does not make an agent do that, it has to be told (see the motivation), but 
speed makes it cheap to do. With the Java
-client the daemon answers in 50-65 ms, under the 0.1 s at which a response 
feels instant, like a native command.
+does not make an agent do that, and telling it to is not enough. In a 
benchmark with a local model, an instruction to
+check made the model read more, but it checked none of the code it wrote (see 
the motivation). The checks have to run
+whether the model asks for them or not, for example in a hook of the agent 
that validates every edited file. A check
+that runs on every edit is where a fast command pays the most. With the Java 
client the daemon answers in 50-65 ms,
+under the 0.1 s at which a response feels instant, like a native command.
 
 == Motivation
 
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc 
and got it right. It s
 exist, and validated nothing in 9 answers. A lookup also costs context: the 
doc of the file component is 17K
 characters, and three docs in one answer forced a compaction of the history.
 
+Two full runs on the night of 2026-11-13 then asked the same model 53 
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look 
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of 
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was 
always "what does this app do". The
+  model read more for it (3.9 tool calls per answer instead of 2.1) and passed 
it 12 of 15 times instead of 8. Most
+  of that gain is one app. In the first run, the model answered from the 
AI-written summary that the readme tool
+  returned, and left a step out. Since CAMEL-26946 that tool returns the facts 
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no 
validation per answer. Yet the model wrote
+  2.6 times as much code in all (221 code blocks instead of 84) and validated 
none of it. Its 3 calls of
+  `camel_validate_source`, all on the first question, checked the route file 
of the app, not a snippet. Options that
+  do not exist went from 6 in 4 answers to 32 in 14, all after the first 
question. "How can I make it better" passed
+  5 of 15 times instead of 12.
+- It cost context and time: 13 compactions of the history instead of 6, and 
8.1 s of model time per answer instead
+  of 5.8 s.
+- In all, 128 answers passed instead of 130: 4 more on the first question, 6 
fewer on the others.
+
+This is one model and one run each. The changes in code, invented options, 
compactions and time are well beyond the
+variance between the rounds. Most changes in the pass rate of a single 
question are not.
+
+Telling the model to check made it read more and write more, not check what it 
wrote. The check of the code in an
+answer has to run whether the model asks for it or not. CAMEL-26960 proposes 
that the AI panel and `camel ask` mark
+the mistakes in the code of an answer. One such mistake is an option that does 
not exist.
+
 == Goals
 
 - The short commands an agent runs most answer in under 0.1 s, the usual limit 
for a response to feel instant, as a
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the 
start of the daemon itself.
    readiness pushed over the WebSocket transport.
 . Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a 
response feels instant, so the commands
   already feel native. A native client for the main platforms, with the Java 
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so 
what agents read (the Camel CLI pages,
-  the instructions of the MCP server, the system prompt of the AI panel) says 
that the short commands are cheap to
-  call: look an option up instead of guessing it, asking only for the options 
needed (the `optionsFilter` of
-  `camel_catalog_doc`) as a whole doc page costs context, and validate every 
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an 
agent uses the CLI. The Camel CLI pages,
+  the MCP server's instructions and the AI panel's system prompt say that the 
short commands are cheap to call. They
+  tell an agent to look an option up instead of guessing it. They also tell it 
to ask only for the options it needs
+  (the `optionsFilter` of `camel_catalog_doc`), since a whole doc page costs 
context. Telling is not enough for the
+  checks: told to validate every snippet, the local model wrote more code and 
validated none of it (see the
+  motivation). So the checks run without the agent's choice. The AI panel and 
`camel ask` mark the mistakes in the
+  code of an answer (CAMEL-26960). The MCP server already validates a file 
before `camel_write_file` writes it. An
+  agent that edits files with its own tools needs a hook that runs `camel 
validate yaml` on each edited route file.
+  The Camel CLI pages show one. That hook runs on every edit, so it is where 
the daemon saves the most.
 . Measure the agents, not only the commands. The local-model benchmark runs in 
the AI panel of the TUI, whose tools
-  run in the process, so it measures the instruction, not the daemon: run it 
with and without the instruction, and
-  compare the pass rate, the calls and the invented options per answer, the 
compactions and the time of the model.
-  The daemon needs a benchmark that drives `camel` through a shell. Make the 
daemon the default only after a release
-  as opt-in, and when that benchmark shows the gain.
+  run in the process, so it measures the instruction and the checks, not the 
daemon. It compares the pass rate, the
+  calls and the invented options per answer, the compactions and the time of 
the model. The first run had the
+  instruction in the first message (see the motivation). The next run puts 
such a line in the system prompt of the
+  AI panel, where it may last past the first question. It compares two jars 
that differ only in that line. A run
+  after that uses the marks of CAMEL-26960 and counts the invented options 
shown without a mark. The daemon needs a
+  benchmark that drives `camel` through a shell. Make the daemon the default 
only after a release as opt-in, and when
+  that benchmark shows the gain.
 . Do not build the native front. A daemon with a native client is in the same 
range (21-37 ms measured, against
   10-40 ms estimated), and the native front costs the most: the base classes 
of the commands change, GraalVM metadata
   must follow every command and option, aesh needs a change, and each release 
needs a binary per platform, all before
@@ -348,12 +386,14 @@ measure.
 == Related tickets
 
 - CAMEL-26942: this design.
+- CAMEL-26961: the opt-in resident daemon of this design, in its stages.
 - CAMEL-26935: make the Camel CLI faster for AI agents, the parent of the 
sub-tasks below.
 - CAMEL-26936: `camel run --background` learns at once that Camel has started.
 - CAMEL-26937: faster startup of the short commands (no JLine terminal or 
log4j when not needed, C1).
 - CAMEL-26938: shorter action polls.
 - CAMEL-26940: a JDK 25 AOT cache instead of a CDS archive per command.
 - CAMEL-26941: `camel status`, the state of an integration in one call.
+- CAMEL-26960: mark the mistakes in the code of AI answers, such as options 
that do not exist.
 - CAMEL-26913: CLI startup and config (CDS archives shared between installs).
 - CAMEL-25388: the Camel CLI reliable for AI agents and scripts.
 - CAMEL-23810: the port of the CLI to aesh.

Reply via email to