This is an automated email from the ASF dual-hosted git repository.
wenjin272 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 582d7f15 [docs] Clarify Tool Use injection and parallel-call guidance
(#1261)
582d7f15 is described below
commit 582d7f15974a39b39c6fac8d1a0c977d5d48c452
Author: daierenao <[email protected]>
AuthorDate: Fri Oct 16 14:27:14 2026 +0800
[docs] Clarify Tool Use injection and parallel-call guidance (#1261)
Generated-by: Codex CLI 0.162.0-alpha.17.2 (GPT-6)
---
docs/content/docs/development/calling-llms/chat.md | 2 +-
.../docs/development/calling-llms/tool_use.md | 181 +++++++++++++++------
docs/content/docs/development/create-agent/yaml.md | 2 +
docs/content/docs/operations/configuration.md | 8 +-
4 files changed, 134 insertions(+), 59 deletions(-)
diff --git a/docs/content/docs/development/calling-llms/chat.md
b/docs/content/docs/development/calling-llms/chat.md
index eb6a00f0..b9216033 100644
--- a/docs/content/docs/development/calling-llms/chat.md
+++ b/docs/content/docs/development/calling-llms/chat.md
@@ -297,7 +297,7 @@ the text blocks and exclude media.
## Tool Calling {#built-in-events-and-actions}
-A chat model can use tools to perform operations while answering a request. To
enable this, define or register the tools and reference their names in the
model setup's `tools` list. See [Tool Use]({{< ref
"docs/development/calling-llms/tool_use" >}}) for configuration and examples.
+A chat model can use tools to perform operations while answering a request. To
enable this, define or register the tools and reference their names in the
model setup's `tools` list. See [Local Function as Tool]({{< ref
"docs/development/calling-llms/tool_use#local-function-as-tool" >}}) for
declaration and registration examples, [Tool Parameter Injection]({{< ref
"docs/development/calling-llms/tool_use#tool-parameter-injection" >}}) for
framework-provided arguments, and [Parallel tool-ca [...]
Send a `ChatRequestEvent` as usual. The model decides whether to answer
directly or request tool calls:
diff --git a/docs/content/docs/development/calling-llms/tool_use.md
b/docs/content/docs/development/calling-llms/tool_use.md
index dc7fef4e..e4035fa0 100644
--- a/docs/content/docs/development/calling-llms/tool_use.md
+++ b/docs/content/docs/development/calling-llms/tool_use.md
@@ -26,19 +26,36 @@ under the License.
## Overview
-Flink Agents provides a flexible and extensible tool use mechanism. Developers
can define the tool as a local Python function, or they can integrate with a
remote MCP server to use the tools provided by the MCP server.
+Tools let a model request work from a Java or Python function, or from an
+[MCP server]({{< ref "docs/development/calling-llms/mcp" >}}). This page
explains how
+to declare function tools, supply runtime parameters, and configure tool
execution.
+For the chat request, tool-call, and response flow, see
+[Tool Calling in Chat with Models]({{< ref
"docs/development/calling-llms/chat#built-in-events-and-actions" >}}).
## Local Function as Tool
-Developer can define the tool as a local Python/Java function, and there are
two ways to define and register a local function as a tool:
+Declare a tool as a static method in an Agent class, or register it on the
+execution environment to share it across agents. In both cases, reference the
+registered tool name in the chat model setup's `tools` list.
+
+The following snippets show the tool declaration and registration parts of a
job.
+They assume a connection named `ollamaChatModelConnection` has been registered
as
+explained in [Declare the Resources]({{< ref
"docs/development/calling-llms/chat-models" >}}).
+The notification helper is application code; the complete
+{{< gh_link
file="python/flink_agents/examples/quickstart/react_agent_example.py"
name="Python example" >}}
+and {{< gh_link
file="examples/src/main/java/org/apache/flink/agents/examples/ReActAgentExample.java"
name="Java example" >}}
+show how to connect a tool to a ReAct Agent and a Flink job.
{{< hint info >}}
-Flink Agents uses the docstring of the python tool function to generate the
tool metadata. The docstring of the python function should accurately describe
the tool's purpose, parameters, and return value, so that the LLM can
understand the tool and use it effectively.
+Flink Agents uses a Python function's docstring to describe the tool to the
model.
+Describe its purpose, parameters, and return value accurately. In Java, use
+`@Tool(description = "...")` and `@ToolParam` to describe the function and
name its parameters.
{{< /hint >}}
### Define Tool as Static Method in Agent Class
-Developer can define the tool as a static method in the agent class while
defining the Agent, and use the `@tool` decorator to mark the function as a
tool in python (or `@Tool` annotation in java). The tool can be referenced by
its name in the `tools` list of the `ResourceDescriptor` when creating the chat
model in the agent.
+Use `@tool` in Python or `@Tool` in Java to declare a static method as a tool.
+The chat model setup references that method by name:
{{< tabs "Define Tool as Static Method in Agent Class" >}}
@@ -67,7 +84,8 @@ class ReviewAnalysisAgent(Agent):
"""ChatModel which focus on review analysis."""
return ResourceDescriptor(
clazz=ResourceName.ChatModel.OLLAMA_SETUP,
- ...,
+ connection="ollamaChatModelConnection",
+ model="qwen3:8b",
tools=["notify_shipping_manager"], # reference the tool by its name
)
@@ -89,12 +107,10 @@ public class ReviewAnalysisAgent extends Agent {
public static ResourceDescriptor reviewAnalysisModel() {
return
ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP)
.addInitialArgument("connection", "ollamaChatModelConnection")
- ...
+ .addInitialArgument("model", "qwen3:8b")
.addInitialArgument("tools",
Collections.singletonList("notifyShippingManager")) // reference the tool by
its name
.build();
}
-
- ...
}
```
{{< /tab >}}
@@ -108,7 +124,8 @@ public class ReviewAnalysisAgent extends Agent {
### Register Tool to Execution Environment
-Developer can register the tool to the execution environment, and then
reference the tool by its name. This allows the tool to be reused by multiple
agents.
+Register a tool on the execution environment when multiple agents need it.
+Each agent references the registered resource name in its chat model setup.
{{< tabs "Register Tool to Execution Environment" >}}
@@ -140,9 +157,10 @@ agents_env.add_resource(
review_analysis_react_agent = ReActAgent(
chat_model=ResourceDescriptor(
clazz=ResourceName.ChatModel.OLLAMA_SETUP,
+ connection="ollamaChatModelConnection",
+ model="qwen3:8b",
tools=["notify_shipping_manager"], # reference the tool by its name
),
- ...
)
```
{{< /tab >}}
@@ -152,7 +170,7 @@ review_analysis_react_agent = ReActAgent(
@Tool(description = "Notify the shipping manager when product received a
negative review due to shipping damage.")
public static void notifyShippingManager(
@ToolParam(name = "id") String id, @ToolParam(name = "review") String
review) {
- ...
+ CustomTypesAndResources.notifyShippingManager(id, review);
}
// Add notify shipping manager tool to the execution environment.
@@ -167,11 +185,13 @@ agentsEnv
// Create react agent with notify shipping manager tool.
ReActAgent reviewAnalysisReactAgent = new ReActAgent(
ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP)
+ .addInitialArgument("connection", "ollamaChatModelConnection")
+ .addInitialArgument("model", "qwen3:8b")
.addInitialArgument(
"tools",
Collections.singletonList("notifyShippingManager")) // reference the tool by
its name
- ...
.build(),
- ...);
+ null,
+ null);
```
{{< /tab >}}
@@ -225,6 +245,29 @@ Some tools need runtime data that should not be chosen by
the model, such as a t
The model only sees and provides normal tool parameters. The built-in
`tool_call_action` resolves injected values and overwrites any model-supplied
values for the same parameters before calling the tool.
+Choose the source according to where the value is available when the tool runs:
+
+| Source | Runtime value | Python declaration |
+| --- | --- | --- |
+| `config` | A value in the agent execution environment configuration |
`InjectedArg.from_config("tenant_id")` |
+| `sensory_memory` | A value at a path in the current run's sensory memory |
`InjectedArg.from_sensory_memory("request.trace_id")` |
+| `short_term_memory` | A value at a path in the agent's short-term memory |
`InjectedArg.from_short_term_memory("session.tenant_id")` |
+
+For memory sources, write the value to the selected path before requesting the
+tool call. The path must exist and refer to a value, rather than a nested
memory
+object. The injection key identifies the configuration entry or memory path;
+it can differ from the tool parameter name.
+
+In Java annotations, set `@ToolParam(injected = true)` to hide the parameter
from
+the model, and use `source` and `key` to bind its value. The default source is
+`sensory_memory`; an omitted key uses the parameter name. For YAML
declarations,
+see the [`injected_args` reference]({{< ref
"docs/development/create-agent/yaml#resources" >}}),
+which covers the configuration shape, defaults, and examples.
+
+For the following example, set `tenant_id` on the agent execution environment
+with `agents_env.get_config().set_str("tenant_id", "tenant-a")` in Python or
+`agentsEnv.getConfig().setStr("tenant_id", "tenant-a")` in Java.
+
{{< tabs "Inject Tool Parameters" >}}
{{< tab "Python" >}}
@@ -271,7 +314,7 @@ public class OrderAgent extends Agent {
{{< /tabs >}}
-Configure the injected value on the agent execution environment. Python uses
`agents_env.get_config().set_str("tenant_id", "tenant-a")`; Java uses
`agentsEnv.getConfig().setStr("tenant_id", "tenant-a")`.
+The model supplies only `order_id`; the framework supplies `tenant_id`.
Injected values can also come from memory when the value is attached to the
current request or session instead of static environment configuration:
@@ -311,61 +354,91 @@ public static String lookupOrder(
{{< /tabs >}}
-The supported sources are `config`, `sensory_memory`, and `short_term_memory`.
-If `source` is omitted, it defaults to `sensory_memory`.
-For Java annotation-based tools, `@ToolParam(injected = true)` is what marks
the
-parameter as framework-injected and hidden from the model; `source` and `key`
only
-configure where the value is read from.
-If the same injected parameter is declared both on the tool function and by a
-descriptor such as YAML, the declarations must resolve to the same source and
key;
-conflicting declarations fail during agent plan construction.
+### Overrides and failures
+
+The built-in `tool_call_action` resolves injection before invoking a function
tool:
-Injected parameters are part of the tool execution contract, not the model
contract:
+- Injected parameters are excluded from the model-facing JSON schema.
+- A framework value overrides any model-supplied value for the same parameter.
+ The original `ToolRequestEvent` arguments remain unchanged.
+- A missing configuration entry, missing memory path, or invalid injected value
+ fails the tool call without invoking the function. Model-supplied values are
+ never used as a fallback.
+- If the function and a descriptor such as YAML both declare an injected
+ parameter, their source and key must agree. A conflict fails during agent
plan
+ construction, before the tool can run.
-- They are not included in the JSON schema sent to the model.
-- They are not written back to the original `ToolRequestEvent`.
-- If a model supplies an argument with the same name, the framework-provided
value takes precedence.
+This resolution is part of the built-in tool-call flow. When calling a function
+tool directly, supply its required arguments yourself.
## Parallel tool-call batches
-When `tool-call.async` is enabled, the built-in `tool_call_action` runs all
tool calls from one
-`ToolRequestEvent` as a single durable batch. Set `async.batch.parallelism` to
control concurrency:
+A model may request several tool calls in one `ToolRequestEvent`. The built-in
+`tool_call_action` can execute the tools registered by the agent, possibly
+including ordinary functions or MCP tool calls, as parallel operations. Use the
+following settings on the agent execution environment:
-- `1` — serial execution (one tool at a time).
-- `> 1` — parallel batch with a sliding window of at most that many in-flight
tool calls.
+| Option | Default | Effect on tool execution |
+| --- | --- | --- |
+| `tool-call.async` | `true` | Enables durable async tool calls. With `false`,
tools execute synchronously, one at a time. |
+| `async.batch.parallelism` | Available CPU count | Maximum concurrent calls
in one async batch. `1` executes tools one at a time while keeping async
execution enabled. |
-Java and Python support concurrent tool-call batches on all supported JDK
versions.
+For example, allow up to four concurrent calls per batch:
-Tool batches share a `num-async-threads` pool **per operator subtask** (every
key routed to
-that subtask). On JDK 21+ and in Python, chat and RAG calls also use this
pool. On older JDKs,
-Java batches use a separate pool from the Action workers. The default batch
parallelism is
-the host CPU count.
+{{< tabs "Configure Tool Parallelism" >}}
+{{< tab "Python" >}}
+```python
+from flink_agents.api.core_options import AgentExecutionOptions
-Parallel tool calls share execution capacity with other external calls. A
large tool
-batch may delay other model calls or retrieval requests. Use
`async.batch.parallelism`
-to limit how many tools run concurrently.
+agents_env.get_config().set(AgentExecutionOptions.TOOL_CALL_ASYNC, True)
+agents_env.get_config().set(AgentExecutionOptions.ASYNC_BATCH_PARALLELISM, 4)
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+import org.apache.flink.agents.api.agents.AgentExecutionOptions;
-{{< hint warning >}}
-**Cross-key impact:** with defaults (`num-async-threads = 2× cores`,
`async.batch.parallelism =
-cores`), one full tool batch can use up to half the subtask pool; several hot
keys can saturate it.
-If your job mixes heavy tool batches with chat/RAG on the same subtask, lower
-`async.batch.parallelism` or increase `num-async-threads`.
-{{< /hint >}}
+agentsEnv.getConfig().set(AgentExecutionOptions.TOOL_CALL_ASYNC, true);
+agentsEnv.getConfig().set(AgentExecutionOptions.ASYNC_BATCH_PARALLELISM, 4);
+```
+{{< /tab >}}
+{{< /tabs >}}
-`async.batch.timeout.ms` applies to the whole batch. On timeout, completed
slots keep their
-outcome; slots that started but did not finish are recorded as failures; slots
that never started
-executing (for example, queued in a saturated pool) stay pending, so they are
re-executed after
-recovery instead of recording a false failure. Timeout cancellation is
best-effort; side-effecting
-tools should be idempotent or provide a reconciler.
+With async execution enabled and `async.batch.parallelism` greater than `1`,
+multiple tool calls can run concurrently. This limit also applies to batches
+created with `ctx.gather(...)`. The `num-async-threads` setting further limits
+available concurrency. Tool batches share execution capacity with other
external
+calls, so a large batch may delay other model calls or retrieval requests.
Lower
+the batch limit to leave capacity for other calls or avoid overwhelming an
+external service. Actual concurrency can be lower than the batch limit.
{{< hint warning >}}
-**Thread reclamation:** a timeout unblocks the batch but cannot interrupt a
tool that is still
-running. Its worker thread stays in the shared `num-async-threads` pool until
the tool returns on
-its own, so a tool that never returns permanently reduces pool capacity. Bound
blocking work inside
-the tool (for example an HTTP client read timeout) rather than relying on this
batch timeout to free
-the thread.
+Calls in a parallel batch must be independent: a tool must not wait for another
+call in the same batch to complete. Concurrency is an optimization and does not
+guarantee that calls overlap or finish in request order. If one operation
depends
+on another's result, request it in a later step instead of the same batch.
{{< /hint >}}
+### Timeouts and external effects
+
+`async.batch.timeout.ms` sets an overall timeout for a parallel batch of
ordinary
+tool calls. Its default is `-1`; non-positive values disable it. When it
expires,
+completed calls keep their results and unfinished calls are reported to the
model
+as failed tool calls.
+
+For requests containing only ordinary tools, the batch timeout is not used when
+`tool-call.async` is `false`, when `async.batch.parallelism` is `1` or less,
or when
+the request has a single tool call. Configure timeouts for blocking operations
+inside each tool, such as an HTTP client read timeout.
+
+A batch timeout does not undo external effects or guarantee that an already
+running tool stops. Tools that change external systems should be idempotent,
+because a call may run again after recovery. See
+[Configuration]({{< ref "docs/operations/configuration" >}}) for the full
async batch
+and execution-capacity options, and
+[Durable Execution]({{< ref
"docs/development/create-agent/actions#durable-execution" >}})
+for recovery guarantees.
+
## Tool response content and metadata
For simple tools, return a value and the framework converts it to response
text.
diff --git a/docs/content/docs/development/create-agent/yaml.md
b/docs/content/docs/development/create-agent/yaml.md
index cb18bda1..b00b0f65 100644
--- a/docs/content/docs/development/create-agent/yaml.md
+++ b/docs/content/docs/development/create-agent/yaml.md
@@ -179,6 +179,8 @@ agents:
The supported `source` values are `config`, `sensory_memory`, and
`short_term_memory`. If omitted, `source` defaults to `sensory_memory`. For
`config`, `key` is read from the agent execution environment configuration. For
memory sources, `key` is the memory path.
+See [Tool Parameter Injection]({{< ref
"docs/development/calling-llms/tool_use#tool-parameter-injection" >}}) for
runtime value resolution, overriding model-supplied arguments, and
missing-value behavior.
+
For Java tools loaded from YAML, the `injected_args` entry itself marks the
parameter
as framework-injected and hides it from the model-facing schema. The Java
method may
use `@ToolParam(name = "...")` to provide stable parameter names, but it does
not need
diff --git a/docs/content/docs/operations/configuration.md
b/docs/content/docs/operations/configuration.md
index d9bc526b..a934ca92 100644
--- a/docs/content/docs/operations/configuration.md
+++ b/docs/content/docs/operations/configuration.md
@@ -130,11 +130,11 @@ Here is the list of all built-in core configuration
options.
| `max-retries` | 0 | int
| Number of additional attempts per model call, including routing judge
calls. Defaults to 0 (no retries).
|
| `retry-wait-interval` | 1 | int
| Base wait interval in seconds between retries. Uses exponential backoff:
the actual wait time for the Nth retry is `retry-wait-interval * 2^(N-1)`
seconds. For example, with default 1s, waits are 1s, 2s, 4s, etc. Retry count
and total wait time are reported in `ChatResponseEvent` and recorded as metrics
(`retryCount`, `retryWaitSec`) under the configured ChatModel resource name. |
| `chat.async` | true | boolean
| Whether chat asynchronously for built-in chat action.
|
-| `tool-call.async` | true | boolean
| Whether the built-in tool-call action runs each tool via durable async
execution.
|
-| `async.batch.parallelism` | os cpu count | int | Maximum concurrent calls in
one `ctx.gather(...)` batch, including ordinary calls, durable calls, and
built-in async tool calls. `1` runs calls sequentially. Actual concurrency is
also limited by `num-async-threads`; lower this per-batch limit to leave
capacity for other calls on the same subtask. Java and Python support batch
concurrency on all supported JDK versions. |
-| `async.batch.timeout.ms` | -1 (disabled) | long (milliseconds) | Overall
timeout for one async batch, including ordinary calls, durable calls, and
built-in async tool calls. Non-positive values disable it. Completed calls keep
their outcomes; unfinished calls receive timeout failures. For durable calls,
started calls are recorded as failures while unstarted calls remain pending for
recovery. Running callbacks may continue after timeout, occupy workers, and
produce external effects, so [...]
+| `tool-call.async` | true | boolean
| Whether the built-in tool-call action runs each tool via durable async
execution. See [Parallel tool-call batches]({{< ref
"docs/development/calling-llms/tool_use#parallel-tool-call-batches" >}}) for
how this switch works with the batch settings below. |
+| `async.batch.parallelism` | os cpu count | int | Maximum concurrent calls in
one async batch, including batches created with `ctx.gather(...)`. `1` runs
calls sequentially. Actual concurrency is also limited by `num-async-threads`.
See [Parallel tool-call batches]({{< ref
"docs/development/calling-llms/tool_use#parallel-tool-call-batches" >}}). |
+| `async.batch.timeout.ms` | -1 (disabled) | long (milliseconds) | Overall
timeout for one async batch. Non-positive values disable it. A timeout does not
guarantee that running calls stop or undo their external effects. See [Timeouts
and external effects]({{< ref
"docs/development/calling-llms/tool_use#timeouts-and-external-effects" >}}) for
tool-call behavior and constraints. |
| `rag.async` | true | boolean
| Whether retrieve context asynchronously for built-in context retrieval
action.
|
-| `num-async-threads` | os cpu count * 2 | int
| Size of the async worker pool shared by all keys on an operator subtask. On
JDK 21+ and in Python, chat, RAG, and batches use this pool. On older JDKs,
Java uses a separate batch pool of this size in addition to its Action workers.
Default `async.batch.parallelism` is `os cpu count`, so one full batch can
occupy up to half of this pool; multiple keys running large batches can
saturate it. |
+| `num-async-threads` | os cpu count * 2 | int | Controls execution capacity
for asynchronous calls. `async.batch.parallelism` separately limits concurrency
within each batch. See [Parallel tool-call batches]({{< ref
"docs/development/calling-llms/tool_use#parallel-tool-call-batches" >}}) for
tool-call concurrency guidance. |
| `parallel-execution.enabled` | true | boolean
| Whether plans with only **Java Actions** run Actions concurrently on the
parallel execution engine when the continuation-based engine is unavailable
(**JDK < 21**). Plans declaring Python Resources additionally require a Flink
version supporting [cross-language async calls]({{< ref
"docs/development/create-agent/actions#cross-language-async-requirements" >}}).
Same-key input records commit in input order, [...]
| `max-in-flight-input-records` | 100 | int
| Maximum number of input records that may be in flight concurrently. Only
enforced by the **JDK < 21** parallel execution engine for plans with only
**Java Actions**; the JDK 21 coroutine engine and plans containing **Python**
actions ignore it. Every admitted record consumes one unit of budget; at the
cap, admission of further records blocks until an in-flight record retires. |
| `job-identifier` | none | String
| The unique identifier of job, remaining consistent after restoring from a
savepoint. If not set, uses flink job id.
|
