This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/camel-performance-tests.git
commit 26fffb2a29509ca029aa9bfd21cdbeefabffdd99 Author: Claus Ibsen <[email protected]> AuthorDate: Thu Nov 5 11:07:31 2026 +0100 ai-benchmark: BENCH_CATALOG_CAP is 18000, the longest answer camel_catalog_doc gives The server cuts catalog answers at 18000 characters since CAMEL-24961 (Oct 23, CatalogDocs.MAX_ANSWER_LENGTH), up from 16000. The harness still cut them at 16000, so the model never saw the end of 53 answers in r27n and 15 in r27l (the longest was 17,810), nor the moreHeaders list that comes only in an answer the server cut. In r27n-4, aws s2 asked for aws2-s3 includeHeaders 21 times: the answer was 17,705 characters, and the model saw 16,000 of them. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016Y4dDi5iQNgtHrDdz2NpwF --- ai-benchmark/agent_mcp_stepwise.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/ai-benchmark/agent_mcp_stepwise.py b/ai-benchmark/agent_mcp_stepwise.py index 0c53dfa..1839460 100755 --- a/ai-benchmark/agent_mcp_stepwise.py +++ b/ai-benchmark/agent_mcp_stepwise.py @@ -49,9 +49,11 @@ TOOL_GROUPS = os.environ.get("BENCH_TOOL_GROUPS") == "1" BARE = os.environ.get("BENCH_BARE") == "1" TOOL_RESULT_CAP = 6000 # camel_catalog_doc answers are cut later: at 6000 the model never saw most of a component's options (file: include, -# move and sortBy sit past 11000 characters), and r14b asked for all of them 75 times to get the same first 6000. 16000 -# is the longest answer camel_catalog_doc gives since CAMEL-25541. BENCH_CATALOG_CAP=6000 gives the r1-r14 behaviour -CATALOG_RESULT_CAP = int(os.environ.get("BENCH_CATALOG_CAP", "16000")) +# move and sortBy sit past 11000 characters), and r14b asked for all of them 75 times to get the same first 6000. 18000 +# is the longest answer camel_catalog_doc gives since CAMEL-24961 (Oct 23 2026; 16000 since CAMEL-25541): the 16000 +# here cut 53 answers of r27n, among them the aws2-s3 one of 17,705 that r27n-4 asked for 21 times. +# BENCH_CATALOG_CAP=6000 gives the r1-r14 behaviour, 16000 that of Oct 16 to Nov 5 2026 +CATALOG_RESULT_CAP = int(os.environ.get("BENCH_CATALOG_CAP", "18000")) SHARED = ["camel_catalog_doc", "camel_catalog_find", "camel_catalog_sample", "camel_validate_source", "camel_get_files", "camel_write_file", "camel_edit_file",
