davsclaus commented on code in PR #27656: URL: https://github.com/apache/camel/pull/27656#discussion_r4237444329
########## components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAuditInputConfiguration.java: ########## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.semantic; + +import java.util.Objects; + +/** Per-expert opt-in to retain the selected evaluation input, independently of telemetry. */ +public final class SemanticAuditInputConfiguration { + public static final int DEFAULT_MAX_CHARS = 4096; + public static final int MAX_CHARS = 100000; Review Comment: This bounds a single record. The memory store holds up to `capacity` records (and `queueCapacity` pending ones), both up to 100000, so the input kept on the heap can reach `capacity × maxChars` code points; even the default capacity of 1000 with `maxChars=100000` is ~100M chars. The PR description says the memory backend remains bounded. Could the memory store get a total input budget (evicting input, not the record, when exceeded), or at least document the worst case? ########## dsl/camel-jbang/camel-jbang-plugin-tui/src/main/java/org/apache/camel/dsl/jbang/core/commands/tui/SemanticAuditView.java: ########## @@ -211,8 +239,14 @@ void refresh() { } boolean ensureLoaded() { + if (!Objects.equals(pid, currentPid())) { + reset(); + pid = currentPid(); + } refresh(); - if (page == null && pending == null && error == null) { + if (pending == null && error == null && (page == null + || pendingDetails == null && !paused && cursor == null && !inputActive() + && nanoTime.getAsLong() - lastRefresh >= 1_000_000_000L)) { Review Comment: With the live refresh this fetches 50 records (plus the selected record) every second, and `SemanticAuditConsole` returns `toMap()` for each row, input included, although the table never shows it. Suggest returning `input` only for an `eventId` lookup and leaving it out of list responses. ########## components/camel-ai/camel-semantic/src/main/java/org/apache/camel/language/semantic/SemanticLanguage.java: ########## @@ -720,6 +722,7 @@ private Object evaluate(Exchange exchange, boolean asPredicate) { active.addAll(invocations.values()); failureReason = "invalid_input"; Object state = current.state.evaluate(exchange, Object.class); + invocations.values().forEach(invocation -> invocation.captureInput(state)); Review Comment: In a batch each invocation snapshots and runs the redactor again on the same `state`. Could it be done once per expert (or once per batch when the redactor is the same)? ########## components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc: ########## @@ -1138,13 +1138,97 @@ Decision records carry explicit action, operation, target, optional namespace/co and policy identifiers, reason code and evidence IDs. They never derive application correlation or tenant identity from arbitrary inbound headers. -Input, body, headers, variables, arbitrary provider metadata and exception messages are -omitted before dispatch to any backend or observer. Caller-authored score descriptions -are also omitted; the UI does not substitute today's definition for missing historical -semantics. Strings are limited to 256 code points and collections to 100 entries. -Oversized result snapshots are omitted with `resultOmitted: snapshot_limit`, preserving -the execution status without displaying altered labels or partial probabilities. These records are evidence about -an evaluation and declared action, not a copy of the submitted content. +Input is omitted by default. Headers, variables, arbitrary provider metadata and exception +messages are never copied automatically. Caller-authored score descriptions are also omitted; +static capability meanings remain available. Result and metadata strings are limited to +256 code points, and their collections to 100 entries. An oversized result is represented by +`resultOmitted: snapshot_limit` without changing the evaluation outcome. + +==== Storing evaluation input + +Enable input capture explicitly for each expert whose submitted state should be retained: + +[source,yaml] +---- +- semantic: + audit: + enabled: true + experts: + security: + enabled: true + input: + enabled: true + maxChars: 4096 + # redactor: auditInputRedactor + decisions: + enabled: false + sinks: [memory] + reader: memory +---- + +`input.enabled` defaults to `false` and does not enable auditing itself. It captures the +selected state passed to the expert (`state`, including a selected header), before provider +validation or execution. It does not take a copy of the entire exchange. Named, batch, direct +and console evaluations use the same settings. Decisions link to their evaluation records; +input is stored on the evaluation rather than duplicated on the decision. + +`maxChars` defaults to 4096 and accepts 1 through 100000. It bounds the total Unicode code +points in strings, map keys and scalar representations. Snapshots accept text, JSON-compatible +maps/lists and scalar values, with at most eight nested levels and 1000 nodes (including map +keys). They are immutable; later changes to the submitted state cannot change history. +Oversized, cyclic or unsupported input is omitted with +`inputOmitted: snapshot_limit_or_unsupported_type`. Input is never truncated, and streams or +arbitrary objects are not converted to text. `inputOmitted: unavailable` means the selected +state was not obtained, for example because the state expression failed. + +An optional `redactor` names a registry bean implementing `SemanticAuditInputRedactor`. +It receives a bounded, immutable copy on the evaluation thread and returns a sanitized value. Review Comment: The size limit is applied before the redactor runs, so a state larger than `maxChars` is dropped even when the redactor would reduce it to a few fields (as in the example that keeps only `question`). Is that intended? If yes, please say so here. ########## dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelSemantic.java: ########## @@ -94,26 +95,101 @@ protected void render(JsonObject response) { new Column().header("REFERENCE").dataAlign(HorizontalAlign.LEFT).with(r -> text(r, "reference")), new Column().header("NAME").dataAlign(HorizontalAlign.LEFT).with(r -> text(r, "name")), new Column().header("ERROR").dataAlign(HorizontalAlign.LEFT).with(r -> text(r, "error"))))); - printer().println("Default expert: " + text(response, "defaultExpert")); - if (response.get("defaultError") != null) { - printer().println("Default expert error: " + response.get("defaultError")); - } + String defaultExpert = text(response, "defaultExpert"); + printer().println("Default expert: " + (defaultExpert.isEmpty() ? "none" : defaultExpert)); Review Comment: The text output now drops `defaultError` entirely. That hides the expected "exactly one eligible expert" case, but also a default expert that exists and fails `describeExpert`, which now prints just `Default expert: none`. Could only the expected case be suppressed? ########## components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc: ########## @@ -1138,13 +1138,97 @@ Decision records carry explicit action, operation, target, optional namespace/co and policy identifiers, reason code and evidence IDs. They never derive application correlation or tenant identity from arbitrary inbound headers. -Input, body, headers, variables, arbitrary provider metadata and exception messages are -omitted before dispatch to any backend or observer. Caller-authored score descriptions -are also omitted; the UI does not substitute today's definition for missing historical -semantics. Strings are limited to 256 code points and collections to 100 entries. -Oversized result snapshots are omitted with `resultOmitted: snapshot_limit`, preserving -the execution status without displaying altered labels or partial probabilities. These records are evidence about -an evaluation and declared action, not a copy of the submitted content. +Input is omitted by default. Headers, variables, arbitrary provider metadata and exception +messages are never copied automatically. Caller-authored score descriptions are also omitted; +static capability meanings remain available. Result and metadata strings are limited to +256 code points, and their collections to 100 entries. An oversized result is represented by +`resultOmitted: snapshot_limit` without changing the evaluation outcome. + +==== Storing evaluation input + +Enable input capture explicitly for each expert whose submitted state should be retained: + +[source,yaml] +---- +- semantic: + audit: + enabled: true + experts: + security: + enabled: true + input: + enabled: true + maxChars: 4096 + # redactor: auditInputRedactor + decisions: + enabled: false + sinks: [memory] + reader: memory +---- + +`input.enabled` defaults to `false` and does not enable auditing itself. It captures the +selected state passed to the expert (`state`, including a selected header), before provider +validation or execution. It does not take a copy of the entire exchange. Named, batch, direct +and console evaluations use the same settings. Decisions link to their evaluation records; +input is stored on the evaluation rather than duplicated on the decision. + +`maxChars` defaults to 4096 and accepts 1 through 100000. It bounds the total Unicode code +points in strings, map keys and scalar representations. Snapshots accept text, JSON-compatible +maps/lists and scalar values, with at most eight nested levels and 1000 nodes (including map +keys). They are immutable; later changes to the submitted state cannot change history. +Oversized, cyclic or unsupported input is omitted with +`inputOmitted: snapshot_limit_or_unsupported_type`. Input is never truncated, and streams or +arbitrary objects are not converted to text. `inputOmitted: unavailable` means the selected +state was not obtained, for example because the state expression failed. + +An optional `redactor` names a registry bean implementing `SemanticAuditInputRedactor`. +It receives a bounded, immutable copy on the evaluation thread and returns a sanitized value. +The return value is bounded again; a null return omits input (`inputOmitted: redacted`). +Successful redaction adds `inputRedacted: true`. Exceptions or invalid redactor output omit +input (`inputOmitted: redaction_failed`) without recording exception text or affecting the +expert result. Redactors must be thread-safe and should not block. Unknown bean references +fail startup. For example, a redactor can remove confidential fields from structured state: + +[source,java] +---- +context.getRegistry().bind("auditInputRedactor", (SemanticAuditInputRedactor) input -> { + if (input instanceof Map<?, ?> state) { + return Map.of("question", state.get("question")); + } + return null; +}); +---- + +Java uses the additional input settings map in `SemanticAuditConfiguration`: + +[source,java] +---- +new SemanticAuditConfiguration(true, Map.of("security", true, "decisions", false), + List.of("memory"), "memory", 1000, 1000, + Map.of("security", new SemanticAuditInputConfiguration(true, 4096, "auditInputRedactor"))); +---- + +The equivalent XML expert entry is: + +[source,xml] +---- +<expert name="security" enabled="true" inputEnabled="true" + inputMaxChars="4096" inputRedactor="auditInputRedactor"/> +---- + +Input settings follow the same restart requirement as the other audit settings. +Captured input goes to every configured audit sink, including `log` when selected, and is Review Comment: Captured input is also served by the dev console and returned to the AI agent by `tui_get_audit` (up to 200 records with raw input). When the audit records detected prompt injections, these payloads are handed to the operator's coding agent as tool output. Could MCP list results leave `input` out (it is available per `eventId`), or wrap it as untrusted data? A sentence here that the dev console carries captured input and must stay on a trusted network would also help, and the `log` sink writing up to 100k chars at INFO deserves a note on log retention/shipping. ########## components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc: ########## @@ -1138,13 +1138,97 @@ Decision records carry explicit action, operation, target, optional namespace/co and policy identifiers, reason code and evidence IDs. They never derive application correlation or tenant identity from arbitrary inbound headers. -Input, body, headers, variables, arbitrary provider metadata and exception messages are -omitted before dispatch to any backend or observer. Caller-authored score descriptions -are also omitted; the UI does not substitute today's definition for missing historical -semantics. Strings are limited to 256 code points and collections to 100 entries. -Oversized result snapshots are omitted with `resultOmitted: snapshot_limit`, preserving -the execution status without displaying altered labels or partial probabilities. These records are evidence about -an evaluation and declared action, not a copy of the submitted content. +Input is omitted by default. Headers, variables, arbitrary provider metadata and exception +messages are never copied automatically. Caller-authored score descriptions are also omitted; +static capability meanings remain available. Result and metadata strings are limited to +256 code points, and their collections to 100 entries. An oversized result is represented by +`resultOmitted: snapshot_limit` without changing the evaluation outcome. + +==== Storing evaluation input + +Enable input capture explicitly for each expert whose submitted state should be retained: + +[source,yaml] +---- +- semantic: + audit: + enabled: true + experts: + security: + enabled: true + input: + enabled: true + maxChars: 4096 + # redactor: auditInputRedactor + decisions: + enabled: false + sinks: [memory] + reader: memory +---- + +`input.enabled` defaults to `false` and does not enable auditing itself. It captures the +selected state passed to the expert (`state`, including a selected header), before provider +validation or execution. It does not take a copy of the entire exchange. Named, batch, direct +and console evaluations use the same settings. Decisions link to their evaluation records; +input is stored on the evaluation rather than duplicated on the decision. + +`maxChars` defaults to 4096 and accepts 1 through 100000. It bounds the total Unicode code Review Comment: If no total budget is added, please document here that the memory store can hold `capacity × maxChars` code points of input. ########## components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticXmlRoutesBuilderLoader.java: ########## @@ -181,17 +181,27 @@ private void declarations(Element semantic, SemanticEvaluationsBuilder evaluatio private SemanticAuditConfiguration audit(Element element) { attributes(element, Set.of("enabled", "reader", "capacity", "queueCapacity")); Map<String, Boolean> experts = new LinkedHashMap<>(); + Map<String, SemanticAuditInputConfiguration> inputs = new LinkedHashMap<>(); List<String> sinks = new ArrayList<>(); for (Element child : children(element)) { if ("expert".equals(child.getLocalName())) { - attributes(child, Set.of("name", "enabled")); - String name = child.getAttribute("name"); + attributes(child, Set.of("name", "enabled", "inputEnabled", "inputMaxChars", "inputRedactor")); + String name = getCamelContext().resolvePropertyPlaceholders(child.getAttribute("name")); Review Comment: XML now resolves property placeholders in the expert `name`, but YAML expert names (map keys) are not resolved, so the two DSLs behave differently. This also looks unrelated to input capture. Either do the same in YAML or leave it out of this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
