mirror of
https://github.com/ZhuLinsen/daily_stock_analysis
synced 2026-09-20 02:43:35 +08:00
This commit is contained in:
@@ -507,6 +507,15 @@ AGENT_SKILLS=
|
||||
# AGENT_PORTFOLIO_AGENT_TIMEOUT_S=0
|
||||
# AGENT_SKILL_AGENT_TIMEOUT_S=0
|
||||
|
||||
# Agent 工具按类别默认超时(秒,0=关闭;未配置时回退到全局 tool_call_timeout_seconds 预算)
|
||||
# 有效超时按 first-wins 解析:显式 per-run tool_call_timeout_seconds > 单工具显式 timeout_seconds > 类别默认值 > 无限制;
|
||||
# 剩余 wall-clock 预算仅作不可突破的外层 cap;inf/nan/负数视为无限制。
|
||||
# market 类工具(get_market_indices / get_sector_rankings 等网络数据调用)复用 AGENT_DATA_TOOL_TIMEOUT_S,无独立开关。
|
||||
# AGENT_DATA_TOOL_TIMEOUT_S=0
|
||||
# AGENT_SEARCH_TOOL_TIMEOUT_S=0
|
||||
# AGENT_ANALYSIS_TOOL_TIMEOUT_S=0
|
||||
# AGENT_ACTION_TOOL_TIMEOUT_S=0
|
||||
|
||||
# 策略专家并发数(仅 specialist 模式生效;范围 1-4,默认 3)
|
||||
# AGENT_SKILL_CONCURRENCY=3
|
||||
|
||||
|
||||
@@ -15,6 +15,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||
- [修复] 单股推送模式在未配置通知渠道时仍会落盘本地个股报告;CLI 启动分析若因空股票列表、个股结果全失败或本地报告保存失败而未生成报告,会显式返回失败并记录原因。
|
||||
- [修复] 合并推送模式下即使个股汇总报告落盘失败,仍会先发送已有的合并通知;仅启用大盘复盘但最终未生成任何复盘内容时,分析任务会显式返回失败。
|
||||
|
||||
- [新功能] Agent 工具调用支持按类别(data/search/analysis/action/market)配置默认超时,并允许单工具声明 `timeout_seconds`;有效超时按 first-wins 优先级解析(显式 per-run `tool_call_timeout_seconds` > 单工具显式 `timeout_seconds` > 类别默认 > 无限制),剩余 wall-clock 预算仅作不可突破的外层 cap,超时后返回结构化 `{"timeout": true}` 错误(标记 `retriable: false` 并写入 `non_retriable_tool_results` 防重试重复执行)供 Agent 继续执行而非中断循环(fixes #1890)。
|
||||
- [修复] Agent 工具注册表(`src/agent/factory.get_tool_registry`)由模块级缓存改为按「类别超时映射的值」比对失效,规避 CPython 回收对象后地址复用(`id(config)` 相同)导致配置 reload 后的 `Config` 被误判为未变、沿用过期超时的真 bug;新增 `_coerce_config_timeout` 类型白名单,使调用方传入 `MagicMock` / 缺属性 stub / 脏字符串(如 `float(MagicMock())` 静默得到 1.0)时降级为「无类别限制」而非崩溃或强加 1 秒超时;`build_agent_executor(config)` / `build_agent_chat_executor(config)` 现已把调用方 `config` 透传给 `get_tool_registry(config)`(不再无参调用冻结首构 registry);`main._reload_runtime_config` 与 `SystemConfigService._reload_runtime_singletons`(及 `update()`→`reload_now` 路径)在配置热重载时调用 `reset_tool_registry()` 强制重建;回归测试补充「传入新 config 后 registry 重建」「reload 后新超时应生效」及「builder 透传 config」三类场景(#1890 的 review follow-up,闭环 OR-COM-dd1e8fa7 / OR-COM-bff42110)
|
||||
- [修复] Agent 工具超时 review 闭环(fixes #1890 的 4 个 blocker):超时解析由 min 契约改为 first-wins(显式 per-run `tool_call_timeout_seconds` > 单工具 `ToolDefinition.timeout_seconds` > 类别默认 > 无限制,剩余 wall-clock 预算只作不可突破的外层 cap;research 路径不再传 `tool_call_timeout_seconds` 以免覆盖类别限制);超时结果标记 `retriable: false` 并写入 `non_retriable_tool_results` 阻断 LLM 同调用重试重入,且超时触发时为仍在后台运行的 handler 武装协作取消信号(`is_tool_cancellation_requested()` 与既有 `check_tool_execution()` 检查点均响应,handler 从不轮询则行为不变),作为 review 要求的「handler 内协作取消」缓解,规避 Python 线程无法 force-stop 导致的重复执行与副作用;`_coerce_config_timeout` 对 `inf`/`nan`/负数降级为「无限制」,根绝 `future.result(timeout=inf)` 触发 `OverflowError`;`get_tool_registry` / `reset_tool_registry` 加 `threading.Lock` 双检锁,且重建后返回本次构建的局部 registry(而非全局缓存),消除并发重建竞态与跨调用超时串扰;`@tool` 装饰器将 `ToolPolicy.timeout_seconds` 折叠进 `ToolDefinition` 单一来源;统一单/并行工具超时包装(单一 executor + deadline 驱动的 wait loop,消除并行路径嵌套 executor 与线程翻倍,duration 精确到各工具自身超时值),并新增快慢工具混合并行回归;同步 `docs/full-guide_EN.md` 的超时环境变量文档;测试覆盖 first-wins、non-retriable、协作取消接线、finite 校验、缓存线程安全与快慢混合并行。
|
||||
- [修复] 按最新 review 复核收敛 3 处正确性问题(OR-COM-7f3d3f5b / 3d6b61f8 / a1e8b0c2):`BaseAgent._filtered_registry()` 携带源 registry 的类别超时映射(工具子集仍生效类别上限,不再绕过 #1890 类别超时);并行批次 >5 时排队调用的 per-tool 超时自 worker 实际开始起算(不再提交即烧预算导致对未启动调用的假超时);`get_tool_registry()` 缓存命中快路径在锁内读取一致对(消除与 `reset_tool_registry()` 竞态返回 `None` 或错配 registry)。新增对应回归测试。
|
||||
|
||||
## [3.30.0] - 2026-08-09
|
||||
|
||||
### 发布亮点
|
||||
|
||||
@@ -246,6 +246,10 @@ daily_stock_analysis/
|
||||
| `AGENT_BACKEND` | 现有问股 Chat 的运行方式:`auto`(推荐,保持默认模型)、`litellm` 或 `codex_app_server`(实验,仅 single-agent Chat) | `auto` | 否 |
|
||||
| `AGENT_GENERATION_BACKEND` | Agent Chat 生成后端;Web 设置页仅暴露 `auto|litellm`,手写 local CLI backend 会返回 unsupported tool-calling 诊断 | `auto` | 否 |
|
||||
| `AGENT_SKILL_CONCURRENCY` | `specialist` 模式策略专家 worker 并发上限,范围 `1-4`;最多选择 4 个策略,默认 3 个并发,第 4 个进入下一批次并共享整体超时预算 | `3` | 否 |
|
||||
| `AGENT_DATA_TOOL_TIMEOUT_S` | Agent `data` 类工具默认超时秒数;同时作为 `market` 类工具(`get_market_indices` / `get_sector_rankings` 等网络数据调用)的类别默认;`0` 表示关闭,回退到全局预算;有效超时按 first-wins 解析:显式 per-run `tool_call_timeout_seconds` > 单工具显式 `timeout_seconds` > 类别默认 > 无限制,剩余 wall-clock 预算仅作不可突破的外层 cap,`inf`/`nan`/负数降级为「无限制」;超时为 best-effort 软中断:Python 线程无法被强制停止,handler 可能在超时后继续运行,超时结果标记 `retriable: false` 并写入 `non_retriable_tool_results` 阻断重试,且为仍在后台的 handler 武装协作取消信号(`is_tool_cancellation_requested()` 与既有 `check_tool_execution()` 检查点均响应,handler 从不轮询则行为不变)以尽量减小副作用 | `0` | 否 |
|
||||
| `AGENT_SEARCH_TOOL_TIMEOUT_S` | Agent `search` 类工具默认超时秒数;`0` 表示关闭,回退到全局预算 | `0` | 否 |
|
||||
| `AGENT_ANALYSIS_TOOL_TIMEOUT_S` | Agent `analysis` 类工具默认超时秒数;`0` 表示关闭,回退到全局预算 | `0` | 否 |
|
||||
| `AGENT_ACTION_TOOL_TIMEOUT_S` | Agent `action` 类工具默认超时秒数;`0` 表示关闭,回退到全局预算 | `0` | 否 |
|
||||
| `LITELLM_MODEL` | 主模型,格式 `provider/model`(如 `gemini/gemini-3.1-pro-preview`),推荐优先使用 | - | 否 |
|
||||
| `AGENT_LITELLM_MODEL` | 「默认模型」问股的主模型(可选);留空继承主模型,无 provider 前缀按 `openai/<model>` 解析;Codex 不使用此项 | - | 否 |
|
||||
| `AGENT_CONTEXT_COMPRESSION_ENABLED` | 「默认模型」问股可见历史的 LLM 压缩开关;Codex 使用最近 20 条可见对话且保留该配置 | `false` | 否 |
|
||||
|
||||
@@ -217,6 +217,10 @@ Default schedule: Every weekday at **18:00 (Beijing Time)** automatic execution.
|
||||
| `AGENT_BACKEND` | Runtime for the existing ask-stock Chat: `auto` (recommended, preserves the default model), `litellm`, or `codex_app_server` (experimental, single-agent Chat only) | `auto` | No |
|
||||
| `AGENT_GENERATION_BACKEND` | Agent Chat generation backend. Web settings only expose `auto|litellm`; hand-written local CLI backends return an unsupported tool-calling diagnostic | `auto` | No |
|
||||
| `AGENT_SKILL_CONCURRENCY` | Specialist-mode strategy worker concurrency cap, range `1-4`. Up to four strategies are selected; the default runs three concurrently and queues the fourth under the shared pipeline budget | `3` | No |
|
||||
| `AGENT_DATA_TOOL_TIMEOUT_S` | Default timeout (seconds) for Agent `data`-category tools; also the category default for `market` tools (`get_market_indices` / `get_sector_rankings` and other network-backed data calls); `0` disables and falls back to the global budget. The effective timeout is resolved first-wins: explicit per-run `tool_call_timeout_seconds` > per-tool `timeout_seconds` > category default > no limit, with the remaining wall-clock budget as an unbreakable outer cap; `inf`/`nan`/negative degrade to "no limit". Timeout is a best-effort soft interrupt: Python threads cannot be force-stopped, so a handler may keep running after the timeout; the timed-out result is marked `retriable: false` and recorded in `non_retriable_tool_results` to block immediate retries, and the runner arms a cooperative-cancel signal (`is_tool_cancellation_requested()` and the existing `check_tool_execution()` checkpoints both honor it) to reduce side effects | `0` | No |
|
||||
| `AGENT_SEARCH_TOOL_TIMEOUT_S` | Default timeout (seconds) for Agent `search`-category tools; `0` disables and falls back to the global budget | `0` | No |
|
||||
| `AGENT_ANALYSIS_TOOL_TIMEOUT_S` | Default timeout (seconds) for Agent `analysis`-category tools; `0` disables and falls back to the global budget | `0` | No |
|
||||
| `AGENT_ACTION_TOOL_TIMEOUT_S` | Default timeout (seconds) for Agent `action`-category tools; `0` disables and falls back to the global budget | `0` | No |
|
||||
| `LITELLM_MODEL` | Primary model, format `provider/model` (e.g. `gemini/gemini-3.1-pro-preview`), recommended | - | No |
|
||||
| `AGENT_LITELLM_MODEL` | Optional primary model for **Default model** ask-stock; empty inherits the primary model and bare names become `openai/<model>`; Codex does not use this setting | - | No |
|
||||
| `AGENT_CONTEXT_COMPRESSION_ENABLED` | LLM compression for visible **Default model** ask-stock history; Codex uses the 20 most recent visible messages and retains this setting | `false` | No |
|
||||
|
||||
16
main.py
16
main.py
@@ -1326,7 +1326,21 @@ def _reload_runtime_config() -> Config:
|
||||
"""Reload config from the latest persisted `.env` values for scheduled runs."""
|
||||
_reload_env_file_values_preserving_overrides()
|
||||
Config.reset_instance()
|
||||
return get_config()
|
||||
new_config = get_config()
|
||||
|
||||
# Drop the module-level ToolRegistry so the next
|
||||
# ``build_agent_executor`` / ``build_agent_chat_executor`` call rebuilds
|
||||
# it against the freshly-loaded Config and picks up new
|
||||
# ``AGENT_*_TOOL_TIMEOUT_S`` overrides (Issue #1890). Wrap in try/except
|
||||
# so a future runtime reset helper cannot crash scheduled-job bootstrap.
|
||||
try:
|
||||
from src.agent.factory import reset_tool_registry
|
||||
|
||||
reset_tool_registry()
|
||||
except Exception as exc: # pragma: no cover - defensive guard
|
||||
logger.warning("Failed to reset tool registry during config reload: %s", exc)
|
||||
|
||||
return new_config
|
||||
|
||||
|
||||
def _build_schedule_time_provider(default_schedule_time: str):
|
||||
|
||||
@@ -257,7 +257,9 @@ class BaseAgent(ABC):
|
||||
return self.tool_registry
|
||||
|
||||
from src.agent.tools.registry import ToolRegistry as TR
|
||||
filtered = TR()
|
||||
# Carry the source registry's category-timeout map so the filtered subset
|
||||
# still enforces the per-category ceilings (review OR-COM-7f3d3f5b).
|
||||
filtered = TR(category_timeout_map=self.tool_registry.category_timeout_map)
|
||||
for name in self.tool_names:
|
||||
tool_def = self.tool_registry.get(name)
|
||||
if tool_def:
|
||||
|
||||
@@ -26,6 +26,8 @@ Usage::
|
||||
|
||||
import copy
|
||||
import logging
|
||||
import math
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from typing import List, Optional
|
||||
|
||||
@@ -37,6 +39,17 @@ logger = logging.getLogger(__name__)
|
||||
# Module-level caches
|
||||
# ---------------------------------------------------------------------------
|
||||
_TOOL_REGISTRY = None
|
||||
# Track the per-category timeout mapping ``_TOOL_REGISTRY`` was built from, so
|
||||
# callers passing a freshly-reloaded ``Config`` always observe the right
|
||||
# ceilings instead of inheriting the first build's values.
|
||||
#
|
||||
# Deliberately keyed by *value* rather than ``id(config)``: CPython reuses the
|
||||
# address of a freed object, so a ``Config.reset_instance()`` + ``get_config()``
|
||||
# round-trip can hand back a new instance with the *same* id() as the collected
|
||||
# one, which would silently mark a changed config as "unchanged". This mirrors
|
||||
# the ``_SKILL_MANAGER_CUSTOM_DIR`` value-comparison used by ``get_skill_manager``
|
||||
# below. Pair with :func:`reset_tool_registry` for runtime config reload paths.
|
||||
_CACHED_TIMEOUT_MAP: Optional[dict] = None
|
||||
_SKILL_MANAGER_PROTOTYPE = None
|
||||
# Sentinel used as initial value so None (i.e. no custom dir) compares as "changed"
|
||||
# on the very first call, forcing a build rather than accidentally skipping it.
|
||||
@@ -45,6 +58,20 @@ _SENTINEL = object()
|
||||
# the cache if AGENT_SKILL_DIR changes at runtime (e.g. via config reload).
|
||||
_SKILL_MANAGER_CUSTOM_DIR: object = _SENTINEL
|
||||
|
||||
# Guards the module-level ToolRegistry cache against concurrent rebuild/reset.
|
||||
# ``get_tool_registry`` builds the registry *outside* this lock (see below) and
|
||||
# only takes it to atomically assign the shared ``_TOOL_REGISTRY`` /
|
||||
# ``_CACHED_TIMEOUT_MAP`` pair, so two requests with different ``Config``
|
||||
# objects can never observe a registry built for the other's timeouts.
|
||||
_tool_registry_lock = threading.Lock()
|
||||
|
||||
# Safety ceiling for any single per-category timeout value sourced from
|
||||
# ``AGENT_*_TOOL_TIMEOUT_S``. ``inf``/``nan``/negative already degrade to
|
||||
# "no limit"; this clips absurd-but-finite values so they can never reach
|
||||
# ``future.result(timeout=...)`` and stall a request. The global
|
||||
# ``tool_call_timeout_seconds`` budget is *not* subject to this clip.
|
||||
_MAX_TOOL_TIMEOUT_S = 3600.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class SkillPromptState:
|
||||
@@ -190,12 +217,144 @@ def _should_use_legacy_default_prompt(
|
||||
return getattr(bull_trend_skill, "source", None) == "builtin"
|
||||
|
||||
|
||||
def get_tool_registry():
|
||||
"""Return a cached ToolRegistry (built once, shared across requests)."""
|
||||
global _TOOL_REGISTRY
|
||||
if _TOOL_REGISTRY is not None:
|
||||
return _TOOL_REGISTRY
|
||||
def _coerce_config_timeout(config, field_name: str) -> float:
|
||||
"""Read a timeout field off ``config`` as a float, defaulting to ``0.0``.
|
||||
|
||||
``get_tool_registry`` builds its category map from the *caller-supplied*
|
||||
config, so it must tolerate partial config objects and test doubles: a
|
||||
missing attribute raises ``AttributeError`` and ``MagicMock() > 0`` raises
|
||||
``TypeError``. A plain ``float()`` conversion is not enough either --
|
||||
``float(MagicMock())`` silently yields ``1.0``, which would quietly impose
|
||||
a bogus 1-second ceiling on every tool. Only genuine numbers (or numeric
|
||||
strings, since ``.env`` values arrive as text) are accepted; anything else
|
||||
degrades to "no category limit" and defers to the global
|
||||
``tool_call_timeout_seconds`` budget.
|
||||
"""
|
||||
raw_value = getattr(config, field_name, None)
|
||||
|
||||
if isinstance(raw_value, bool) or raw_value is None:
|
||||
# ``True`` would coerce to a nonsensical 1-second ceiling.
|
||||
return 0.0
|
||||
if isinstance(raw_value, (int, float)):
|
||||
value = float(raw_value)
|
||||
if not math.isfinite(value):
|
||||
# ``inf``/``nan`` are not valid ceilings; an ``inf`` would later
|
||||
# raise ``OverflowError`` at ``future.result(timeout=inf)``.
|
||||
logger.warning(
|
||||
"[AgentFactory] Non-finite value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
raw_value,
|
||||
)
|
||||
return 0.0
|
||||
if value < 0:
|
||||
logger.warning(
|
||||
"[AgentFactory] Negative value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
raw_value,
|
||||
)
|
||||
return 0.0
|
||||
if value == 0.0:
|
||||
# Explicitly disabled (the default) — silent, not a warning.
|
||||
return 0.0
|
||||
if value > _MAX_TOOL_TIMEOUT_S:
|
||||
logger.warning(
|
||||
"[AgentFactory] Clamping %s=%r to safety ceiling %ss",
|
||||
field_name, value, _MAX_TOOL_TIMEOUT_S,
|
||||
)
|
||||
return float(_MAX_TOOL_TIMEOUT_S)
|
||||
return value
|
||||
if isinstance(raw_value, str):
|
||||
try:
|
||||
value = float(raw_value.strip())
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
"[AgentFactory] Invalid value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
raw_value,
|
||||
)
|
||||
return 0.0
|
||||
if not math.isfinite(value):
|
||||
logger.warning(
|
||||
"[AgentFactory] Non-finite value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
raw_value,
|
||||
)
|
||||
return 0.0
|
||||
if value < 0:
|
||||
logger.warning(
|
||||
"[AgentFactory] Negative value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
raw_value,
|
||||
)
|
||||
return 0.0
|
||||
if value == 0.0:
|
||||
return 0.0
|
||||
if value > _MAX_TOOL_TIMEOUT_S:
|
||||
return float(_MAX_TOOL_TIMEOUT_S)
|
||||
return value
|
||||
|
||||
# Mocks / arbitrary objects: never let them shape real timeout behaviour.
|
||||
logger.debug(
|
||||
"[AgentFactory] Non-numeric value for %s: %r, treating as no category limit",
|
||||
field_name,
|
||||
type(raw_value).__name__,
|
||||
)
|
||||
return 0.0
|
||||
|
||||
|
||||
def _build_category_timeout_map(config) -> dict:
|
||||
"""Map tool categories to their default timeout (seconds) from config.
|
||||
|
||||
Only explicit overrides (value > 0) are carried; 0.0 means "no category
|
||||
limit, fall back to the global tool_call_timeout_seconds budget".
|
||||
|
||||
``market`` tools (get_market_indices / get_sector_rankings) are
|
||||
network-backed data calls, so they share the ``data`` ceiling.
|
||||
"""
|
||||
data_timeout = _coerce_config_timeout(config, "agent_data_tool_timeout_s")
|
||||
return {
|
||||
k: v
|
||||
for k, v in {
|
||||
"data": data_timeout,
|
||||
"search": _coerce_config_timeout(config, "agent_search_tool_timeout_s"),
|
||||
"analysis": _coerce_config_timeout(config, "agent_analysis_tool_timeout_s"),
|
||||
"action": _coerce_config_timeout(config, "agent_action_tool_timeout_s"),
|
||||
"market": data_timeout,
|
||||
}.items()
|
||||
if v > 0
|
||||
}
|
||||
|
||||
|
||||
def reset_tool_registry() -> None:
|
||||
"""Drop the cached :class:`ToolRegistry` so the next access rebuilds it.
|
||||
|
||||
Intended for runtime config reload paths
|
||||
(``main._reload_runtime_config`` /
|
||||
``SystemConfigService._reload_runtime_singletons``) so that newly-loaded
|
||||
``AGENT_*_TOOL_TIMEOUT_S`` overrides actually take effect on subsequent
|
||||
:func:`build_agent_executor` / :func:`build_agent_chat_executor` calls,
|
||||
instead of being silently shadowed by the registry built at first import.
|
||||
|
||||
Guarded by ``_tool_registry_lock`` so a concurrent :func:`get_tool_registry`
|
||||
cannot read a half-reset cache (``None`` registry with a stale map, or vice
|
||||
versa).
|
||||
"""
|
||||
global _TOOL_REGISTRY, _CACHED_TIMEOUT_MAP
|
||||
with _tool_registry_lock:
|
||||
if _TOOL_REGISTRY is not None:
|
||||
logger.info("[AgentFactory] ToolRegistry cache cleared; will rebuild on next access")
|
||||
_TOOL_REGISTRY = None
|
||||
_CACHED_TIMEOUT_MAP = None
|
||||
|
||||
|
||||
def _build_tool_registry(category_timeout_map):
|
||||
"""Construct and populate a :class:`ToolRegistry` from the tool modules.
|
||||
|
||||
Extracted from :func:`get_tool_registry` so the (potentially heavy) tool
|
||||
module imports and registrations happen *outside* the cache lock — holding
|
||||
the lock during imports would serialise startup and risk a deadlock if any
|
||||
imported module ever imported this module back.
|
||||
"""
|
||||
from src.agent.tools.registry import ToolRegistry
|
||||
from src.agent.tools.data_tools import ALL_DATA_TOOLS
|
||||
from src.agent.tools.analysis_tools import ALL_ANALYSIS_TOOLS
|
||||
@@ -203,13 +362,91 @@ def get_tool_registry():
|
||||
from src.agent.tools.market_tools import ALL_MARKET_TOOLS
|
||||
from src.agent.tools.backtest_tools import ALL_BACKTEST_TOOLS
|
||||
|
||||
registry = ToolRegistry()
|
||||
for tool_fn in ALL_DATA_TOOLS + ALL_ANALYSIS_TOOLS + ALL_SEARCH_TOOLS + ALL_MARKET_TOOLS + ALL_BACKTEST_TOOLS:
|
||||
registry = ToolRegistry(category_timeout_map=category_timeout_map or None)
|
||||
for tool_fn in (
|
||||
ALL_DATA_TOOLS
|
||||
+ ALL_ANALYSIS_TOOLS
|
||||
+ ALL_SEARCH_TOOLS
|
||||
+ ALL_MARKET_TOOLS
|
||||
+ ALL_BACKTEST_TOOLS
|
||||
):
|
||||
registry.register(tool_fn)
|
||||
return registry
|
||||
|
||||
_TOOL_REGISTRY = registry
|
||||
logger.info("[AgentFactory] ToolRegistry cached (%d tools)", len(registry._tools) if hasattr(registry, "_tools") else -1)
|
||||
return _TOOL_REGISTRY
|
||||
|
||||
def get_tool_registry(config=None):
|
||||
"""Return a :class:`ToolRegistry` bound to the lifetime of ``config``.
|
||||
|
||||
The first call (or the first call after :func:`reset_tool_registry`)
|
||||
builds and caches a registry from ``config``. The cache is keyed by the
|
||||
resolved per-category timeout *values*, so a config carrying different
|
||||
``AGENT_*_TOOL_TIMEOUT_S`` settings (e.g. a freshly-reloaded ``Config``
|
||||
singleton) rebuilds the registry, while an equivalent config reuses the
|
||||
cached instance and avoids re-registering the whole tool surface.
|
||||
|
||||
Value comparison is intentional: ``id(config)`` is not a safe key because
|
||||
CPython reuses the address of a collected object, so the instance returned
|
||||
after ``Config.reset_instance()`` can share the previous instance's id and
|
||||
would be misread as "unchanged".
|
||||
|
||||
When ``config`` is ``None`` the function falls back to ``get_config()``,
|
||||
preserving the legacy contract for callers that do not yet know about
|
||||
config reload.
|
||||
|
||||
Thread-safety: the fast-path cache hit reads both globals under the lock
|
||||
(so the check and the return observe one consistent pair — see
|
||||
``reset_tool_registry``), and the registry build runs outside the lock with
|
||||
only the final assignment of the shared ``_TOOL_REGISTRY`` /
|
||||
``_CACHED_TIMEOUT_MAP`` pair serialised (double-checked locking), so two
|
||||
requests with different ``Config`` objects can never observe a registry built
|
||||
for the other's timeouts. On a (re)build the function returns the registry it
|
||||
just constructed for the caller's timeout map, so a concurrent rebuild cannot
|
||||
leak a mismatched registry into this call's result.
|
||||
"""
|
||||
global _TOOL_REGISTRY, _CACHED_TIMEOUT_MAP
|
||||
|
||||
if config is None:
|
||||
from src.config import get_config
|
||||
|
||||
config = get_config()
|
||||
|
||||
category_timeout_map = _build_category_timeout_map(config)
|
||||
# Fast path: take the lock so the cache-hit check and the return observe one
|
||||
# consistent (registry, timeout-map) pair — an unlocked check-then-use can
|
||||
# race with ``reset_tool_registry()`` and return ``None`` or a registry built
|
||||
# for the wrong timeout map (review OR-COM-a1e8b0c2).
|
||||
with _tool_registry_lock:
|
||||
if _TOOL_REGISTRY is not None and _CACHED_TIMEOUT_MAP == category_timeout_map:
|
||||
return _TOOL_REGISTRY
|
||||
|
||||
# Build *outside* the lock (see ``_build_tool_registry`` rationale above).
|
||||
registry = _build_tool_registry(category_timeout_map)
|
||||
with _tool_registry_lock:
|
||||
# Re-check under the lock: another thread may have installed an
|
||||
# equivalent registry while we were importing the tool modules.
|
||||
if _TOOL_REGISTRY is not None and _CACHED_TIMEOUT_MAP == category_timeout_map:
|
||||
return _TOOL_REGISTRY
|
||||
invalidated = _TOOL_REGISTRY is not None
|
||||
_TOOL_REGISTRY = registry
|
||||
_CACHED_TIMEOUT_MAP = category_timeout_map
|
||||
if invalidated:
|
||||
logger.info(
|
||||
"[AgentFactory] ToolRegistry rebuilt for new category timeouts: %s",
|
||||
category_timeout_map or "none (global budget only)",
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"[AgentFactory] ToolRegistry cached (%d tools, category timeouts: %s)",
|
||||
len(registry._tools) if hasattr(registry, "_tools") else -1,
|
||||
category_timeout_map or "none (global budget only)",
|
||||
)
|
||||
# Return the registry built for *this* call's timeout map (not the shared
|
||||
# ``_TOOL_REGISTRY`` global). Under a concurrent rebuild for a different
|
||||
# ``Config``/timeout map, this guarantees the caller gets the registry that
|
||||
# matches the timeouts it requested, instead of a registry built for another
|
||||
# request. The global is still assigned above so equivalent subsequent calls
|
||||
# hit the cache fast path.
|
||||
return registry
|
||||
|
||||
|
||||
def get_skill_manager(config=None):
|
||||
@@ -336,7 +573,11 @@ def build_agent_executor(config=None, skills: Optional[List[str]] = None):
|
||||
|
||||
from src.agent.llm_adapter import LLMToolAdapter
|
||||
|
||||
registry = get_tool_registry()
|
||||
# Pass ``config`` explicitly so a reloaded ``Config`` instance observes
|
||||
# the new per-category timeout map (Issue #1890 compatibility concern:
|
||||
# the module-level ToolRegistry would otherwise keep the first build's
|
||||
# timeouts even after ``_reload_runtime_config``).
|
||||
registry = get_tool_registry(config)
|
||||
prompt_state = resolve_skill_prompt_state(config, skills=skills)
|
||||
skill_manager = prompt_state.skill_manager
|
||||
logger.info(
|
||||
@@ -406,7 +647,10 @@ def build_agent_chat_executor(config=None, skills: Optional[List[str]] = None):
|
||||
if backend_id == "litellm" and arch == "multi":
|
||||
return build_agent_executor(config, skills=skills)
|
||||
|
||||
registry = get_tool_registry()
|
||||
# Pass ``config`` explicitly so a reloaded ``Config`` instance observes
|
||||
# the new per-category timeout map (see ``build_agent_executor`` for the
|
||||
# same rationale).
|
||||
registry = get_tool_registry(config)
|
||||
prompt_state = resolve_skill_prompt_state(config, skills=skills)
|
||||
if backend_id == "litellm":
|
||||
from src.agent.llm_adapter import LLMToolAdapter
|
||||
|
||||
@@ -379,7 +379,10 @@ Token budget remaining: ~{remaining_budget}
|
||||
llm_adapter=self.llm_adapter,
|
||||
max_steps=4,
|
||||
max_wall_clock_seconds=timeout_seconds,
|
||||
tool_call_timeout_seconds=timeout_seconds,
|
||||
# NOTE: do not pass tool_call_timeout_seconds here — under the
|
||||
# first-wins contract it would override the per-tool / category
|
||||
# limits, letting one tool consume the whole sub-question budget.
|
||||
# The remaining wall-clock budget alone keeps tools bounded.
|
||||
)
|
||||
if not result.success and self._looks_like_timeout_error(result.error):
|
||||
return {
|
||||
|
||||
@@ -17,10 +17,12 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
import time
|
||||
import threading
|
||||
import contextvars
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError, as_completed
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
@@ -30,6 +32,7 @@ from src.agent.protocols import StageFailureReason
|
||||
from src.agent.stream_events import stream_event
|
||||
from src.agent.tools.registry import ToolRegistry
|
||||
from src.agent.tools.execution import (
|
||||
TOOL_CANCEL_EVENT,
|
||||
_build_tool_cache_key,
|
||||
_guard_tool_stock_scope,
|
||||
_is_non_retriable_tool_result,
|
||||
@@ -348,7 +351,10 @@ def run_agent_loop(
|
||||
progress_callback: Optional callback receiving progress dicts.
|
||||
thinking_labels: Override map of tool_name → friendly label.
|
||||
max_wall_clock_seconds: Optional overall timeout budget for the loop.
|
||||
tool_call_timeout_seconds: Optional timeout for one parallel tool batch.
|
||||
tool_call_timeout_seconds: Optional explicit per-run tool-call timeout.
|
||||
Highest-priority (first-wins) in the per-tool timeout chain —
|
||||
overrides per-tool declarations and category defaults — but is
|
||||
capped by ``max_wall_clock_seconds`` when both are set.
|
||||
emit_stage_events: Whether to emit the synthetic ``agent_loop``
|
||||
stage lifecycle. Orchestrated business stages disable this so
|
||||
``stage_start`` / ``stage_done`` only describe real stages.
|
||||
@@ -509,13 +515,10 @@ def run_agent_loop(
|
||||
assistant_msg["provider_blocks"] = response.provider_blocks
|
||||
messages.append(assistant_msg)
|
||||
|
||||
# Execute tools (parallel when > 1)
|
||||
effective_tool_timeout = tool_call_timeout_seconds
|
||||
if remaining_timeout is not None:
|
||||
effective_tool_timeout = min(
|
||||
remaining_timeout,
|
||||
tool_call_timeout_seconds if tool_call_timeout_seconds and tool_call_timeout_seconds > 0 else remaining_timeout,
|
||||
)
|
||||
# Execute tools (parallel when > 1). ``tool_call_timeout_seconds`` is
|
||||
# the caller's explicit per-run override — highest priority in the
|
||||
# first-wins chain (Issue #1890 contract) — while ``remaining_timeout``
|
||||
# is the unbreakable outer wall-clock cap for this batch.
|
||||
tool_results = _execute_tools(
|
||||
response.tool_calls,
|
||||
tool_registry,
|
||||
@@ -523,7 +526,8 @@ def run_agent_loop(
|
||||
progress_callback,
|
||||
tool_calls_log,
|
||||
non_retriable_tool_results,
|
||||
tool_wait_timeout_seconds=effective_tool_timeout,
|
||||
tool_call_timeout_seconds=tool_call_timeout_seconds,
|
||||
tool_wait_timeout_seconds=remaining_timeout,
|
||||
stock_scope=stock_scope,
|
||||
)
|
||||
|
||||
@@ -601,6 +605,107 @@ def run_agent_loop(
|
||||
# Internal tool execution
|
||||
# ============================================================
|
||||
|
||||
def _coerce_positive_timeout(value) -> Optional[float]:
|
||||
"""Coerce a timeout candidate to a positive finite float, else ``None``.
|
||||
|
||||
``None`` / non-numeric / non-positive / ``inf`` / ``nan`` all map to
|
||||
``None`` ("no limit at this level"). Rejecting non-finite values prevents
|
||||
an ``OverflowError`` from ``future.result(timeout=inf)`` and avoids the
|
||||
undefined ordering a ``nan`` would produce.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
v = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not math.isfinite(v) or v <= 0:
|
||||
return None
|
||||
return v
|
||||
|
||||
|
||||
def _build_timeout_result_payload(
|
||||
tool_name: str,
|
||||
arguments: Dict[str, Any],
|
||||
timeout_s: float,
|
||||
non_retriable_tool_results: Optional[Dict[str, str]],
|
||||
) -> str:
|
||||
"""Build the timeout-shaped result string and record it as non-retriable.
|
||||
|
||||
A timed-out call is marked ``"retriable": False`` and inserted into
|
||||
``non_retriable_tool_results`` so the LLM's retry of the *same* call reuses
|
||||
the cached failure instead of spinning up a second (possibly side-effecting)
|
||||
execution — a best-effort guard against duplicate work, since Python cannot
|
||||
forcibly cancel an already-started tool thread.
|
||||
"""
|
||||
label = f"{timeout_s:.2f}s"
|
||||
result_str = json.dumps({
|
||||
"error": f"Tool execution timed out after {label}",
|
||||
"timeout": True,
|
||||
"retriable": False,
|
||||
})
|
||||
if non_retriable_tool_results is not None:
|
||||
cache_key = _build_tool_cache_key(tool_name, arguments)
|
||||
# Non-dict / missing ``arguments`` yields ``None`` here; skip the write so
|
||||
# unrelated no-arg tool calls cannot collide on a shared ``None`` key.
|
||||
if cache_key:
|
||||
non_retriable_tool_results[cache_key] = result_str
|
||||
return result_str
|
||||
|
||||
|
||||
def _resolve_per_tool_timeout(
|
||||
tool_call,
|
||||
tool_registry: Optional[ToolRegistry],
|
||||
explicit_timeout: Optional[float] = None,
|
||||
wall_clock_budget: Optional[float] = None,
|
||||
) -> Optional[float]:
|
||||
"""Resolve the effective timeout for a single tool call (Issue #1890).
|
||||
|
||||
Precedence is *first-wins*, exactly as confirmed in the issue contract:
|
||||
|
||||
1. explicit per-run timeout (``tool_call_timeout_seconds``)
|
||||
2. per-tool declaration (``ToolDefinition.timeout_seconds``)
|
||||
3. category default (``AGENT_*_TOOL_TIMEOUT_S``)
|
||||
4. none (no per-tool limit)
|
||||
|
||||
An earlier level is never lowered by a later one: an explicit
|
||||
``tool_call_timeout_seconds`` can *relax* a stricter per-tool or category
|
||||
default (the previous ``min()`` across all levels made that impossible), and
|
||||
a per-tool declaration is never overridden by a smaller category default.
|
||||
|
||||
``wall_clock_budget`` (the remaining overall loop budget) is then applied as
|
||||
an *unbreakable outer cap*: the winner can never exceed it, so a long
|
||||
explicit timeout cannot blow past the caller's overall wall-clock budget.
|
||||
"""
|
||||
explicit = _coerce_positive_timeout(explicit_timeout)
|
||||
budget = _coerce_positive_timeout(wall_clock_budget)
|
||||
|
||||
if tool_registry is None:
|
||||
base = explicit
|
||||
else:
|
||||
tool_def = tool_registry.get(tool_call.name)
|
||||
if tool_def is None:
|
||||
base = explicit
|
||||
else:
|
||||
per_tool = _coerce_positive_timeout(getattr(tool_def, "timeout_seconds", None))
|
||||
category = _coerce_positive_timeout(
|
||||
tool_registry.category_default_timeout(tool_def.category)
|
||||
)
|
||||
# First-wins: explicit per-run > per-tool declaration > category default.
|
||||
base = (
|
||||
explicit
|
||||
if explicit is not None
|
||||
else (per_tool if per_tool is not None else category)
|
||||
)
|
||||
|
||||
if base is None:
|
||||
# No explicit/per-tool/category limit; the wall-clock budget governs.
|
||||
return budget
|
||||
if budget is not None:
|
||||
return min(base, budget)
|
||||
return base
|
||||
|
||||
|
||||
def _execute_tools(
|
||||
tool_calls,
|
||||
tool_registry: ToolRegistry,
|
||||
@@ -608,12 +713,23 @@ def _execute_tools(
|
||||
progress_callback: Optional[Callable],
|
||||
tool_calls_log: List[Dict[str, Any]],
|
||||
non_retriable_tool_results: Optional[Dict[str, str]] = None,
|
||||
tool_call_timeout_seconds: Optional[float] = None,
|
||||
tool_wait_timeout_seconds: Optional[float] = None,
|
||||
stock_scope: Optional[StockScope] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Execute one or more tool calls, returning ordered result dicts.
|
||||
|
||||
Single tools run inline; multiple tools run in parallel threads.
|
||||
A single tool with no resolved timeout runs inline (no thread); otherwise
|
||||
all tools share one executor whose per-tool timeouts are enforced by a
|
||||
deadline-driven wait loop — there are no per-tool nested pools, so a batch
|
||||
of N tools uses at most ``min(N, 5)`` threads (review: unify the single and
|
||||
parallel timeout wrapping). ``tool_wait_timeout_seconds`` also caps the
|
||||
whole batch as an outer ceiling.
|
||||
|
||||
``tool_call_timeout_seconds`` is the caller's explicit per-run override
|
||||
(highest priority, first-wins) and ``tool_wait_timeout_seconds`` the
|
||||
remaining wall-clock budget (outer cap) — both feed
|
||||
:func:`_resolve_per_tool_timeout`.
|
||||
"""
|
||||
|
||||
def _exec_single(tc_item):
|
||||
@@ -624,50 +740,39 @@ def _execute_tools(
|
||||
non_retriable_tool_results=non_retriable_tool_results,
|
||||
)
|
||||
|
||||
results: List[Dict[str, Any]] = []
|
||||
def _exec_with_deadline(tc_item, per_tool_timeout, deadline_holder):
|
||||
"""Run one tool and record its per-tool deadline when the worker actually
|
||||
starts. The batch pool is capped at ``min(N, 5)`` workers, so calls that
|
||||
queue behind a full pool must not burn their timeout before execution
|
||||
begins (review OR-COM-3d6b61f8) — otherwise a batch larger than five can
|
||||
falsely time out a tool that never got a worker.
|
||||
"""
|
||||
if per_tool_timeout and per_tool_timeout > 0:
|
||||
deadline_holder[0] = time.monotonic() + per_tool_timeout
|
||||
return _exec_single(tc_item)
|
||||
|
||||
if len(tool_calls) == 1:
|
||||
tc = tool_calls[0]
|
||||
if progress_callback:
|
||||
progress_callback(stream_event("tool_start", step=step, tool=tc.name))
|
||||
timeout_triggered = False
|
||||
if tool_wait_timeout_seconds and tool_wait_timeout_seconds > 0:
|
||||
pool = ThreadPoolExecutor(max_workers=1)
|
||||
ctx = contextvars.copy_context()
|
||||
try:
|
||||
future = pool.submit(ctx.run, _exec_single, tc)
|
||||
try:
|
||||
_, result_str, success, dur, cached, guard_result = future.result(timeout=tool_wait_timeout_seconds)
|
||||
except FuturesTimeoutError:
|
||||
timeout_triggered = True
|
||||
future.cancel()
|
||||
timeout_label = f"{tool_wait_timeout_seconds:.2f}s"
|
||||
logger.warning("Tool '%s' timed out after %s at step %d", tc.name, timeout_label, step)
|
||||
result_str = json.dumps({
|
||||
"error": f"Tool execution timed out after {timeout_label}",
|
||||
"timeout": True,
|
||||
})
|
||||
success = False
|
||||
dur = round(tool_wait_timeout_seconds, 2)
|
||||
cached = False
|
||||
guard_result = None
|
||||
finally:
|
||||
pool.shutdown(wait=not timeout_triggered, cancel_futures=timeout_triggered)
|
||||
def _record(tc_item, *, timed_out, timeout_s=None, out=None, guard_result=None):
|
||||
"""Emit ``tool_done``, build the log entry, and append to ``results``."""
|
||||
if out is not None:
|
||||
_, result_str, success, dur, cached, guard_result = out
|
||||
else:
|
||||
_, result_str, success, dur, cached, guard_result = _exec_single(tc)
|
||||
result_str = _build_timeout_result_payload(
|
||||
tc_item.name, tc_item.arguments, timeout_s, non_retriable_tool_results,
|
||||
)
|
||||
success = False
|
||||
dur = round(timeout_s, 2)
|
||||
cached = False
|
||||
if progress_callback:
|
||||
progress_callback(stream_event("tool_done", step=step, tool=tc.name, success=success, duration=dur))
|
||||
progress_callback(stream_event(
|
||||
"tool_done", step=step, tool=tc_item.name, success=success, duration=dur,
|
||||
))
|
||||
log_entry = {
|
||||
"step": step, "tool": tc.name, "arguments": tc.arguments,
|
||||
"step": step, "tool": tc_item.name, "arguments": tc_item.arguments,
|
||||
"success": success, "duration": dur, "result_length": len(result_str),
|
||||
"cached": cached,
|
||||
}
|
||||
if tool_wait_timeout_seconds and tool_wait_timeout_seconds > 0 and not success:
|
||||
try:
|
||||
if json.loads(result_str).get("timeout") is True:
|
||||
log_entry["timeout"] = True
|
||||
except (TypeError, ValueError, json.JSONDecodeError):
|
||||
pass
|
||||
if timed_out:
|
||||
log_entry["timeout"] = True
|
||||
if guard_result is not None:
|
||||
log_entry.update({
|
||||
"guarded": True,
|
||||
@@ -676,74 +781,109 @@ def _execute_tools(
|
||||
"allowed_stock_codes": guard_result.get("allowed_stock_codes", []),
|
||||
})
|
||||
tool_calls_log.append(log_entry)
|
||||
results.append({"tc": tc, "result_str": result_str})
|
||||
else:
|
||||
for tc in tool_calls:
|
||||
if progress_callback:
|
||||
progress_callback(stream_event("tool_start", step=step, tool=tc.name))
|
||||
results.append({"tc": tc_item, "result_str": result_str})
|
||||
|
||||
pool = ThreadPoolExecutor(max_workers=min(len(tool_calls), 5))
|
||||
timeout_triggered = False
|
||||
try:
|
||||
futures = {pool.submit(contextvars.copy_context().run, _exec_single, tc): tc for tc in tool_calls}
|
||||
pending = set(futures)
|
||||
for future in as_completed(
|
||||
futures,
|
||||
timeout=tool_wait_timeout_seconds if tool_wait_timeout_seconds and tool_wait_timeout_seconds > 0 else None,
|
||||
):
|
||||
pending.discard(future)
|
||||
tc_item, result_str, success, dur, cached, guard_result = future.result()
|
||||
if progress_callback:
|
||||
progress_callback(stream_event("tool_done", step=step, tool=tc_item.name, success=success, duration=dur))
|
||||
log_entry = {
|
||||
"step": step, "tool": tc_item.name, "arguments": tc_item.arguments,
|
||||
"success": success, "duration": dur, "result_length": len(result_str),
|
||||
"cached": cached,
|
||||
}
|
||||
if guard_result is not None:
|
||||
log_entry.update({
|
||||
"guarded": True,
|
||||
"expected_stock_code": guard_result.get("expected_stock_code"),
|
||||
"requested_stock_code": guard_result.get("requested_stock_code"),
|
||||
"allowed_stock_codes": guard_result.get("allowed_stock_codes", []),
|
||||
})
|
||||
tool_calls_log.append(log_entry)
|
||||
results.append({"tc": tc_item, "result_str": result_str})
|
||||
except FuturesTimeoutError:
|
||||
timeout_triggered = True
|
||||
timeout_label = (
|
||||
f"{tool_wait_timeout_seconds:.2f}s"
|
||||
if tool_wait_timeout_seconds is not None
|
||||
else "the configured limit"
|
||||
results: List[Dict[str, Any]] = []
|
||||
if not tool_calls:
|
||||
return results
|
||||
|
||||
plan = []
|
||||
for tc in tool_calls:
|
||||
per_tool_timeout = _resolve_per_tool_timeout(
|
||||
tc, tool_registry, tool_call_timeout_seconds, tool_wait_timeout_seconds,
|
||||
)
|
||||
plan.append((tc, per_tool_timeout))
|
||||
if progress_callback:
|
||||
progress_callback(stream_event("tool_start", step=step, tool=tc.name))
|
||||
|
||||
# Fast path: a single tool with no resolved timeout runs inline (no thread).
|
||||
if len(plan) == 1 and not (plan[0][1] and plan[0][1] > 0):
|
||||
_record(plan[0][0], timed_out=False, out=_exec_single(plan[0][0]))
|
||||
return results
|
||||
|
||||
pool = ThreadPoolExecutor(max_workers=min(len(plan), 5))
|
||||
futures: Dict[Any, Any] = {}
|
||||
cancel_of: Dict[Any, threading.Event] = {}
|
||||
# ``deadline_of`` maps a Future to a single-element holder list; the worker
|
||||
# writes the deadline into it when it *starts* running (see
|
||||
# ``_exec_with_deadline``). ``None`` until then means "still queued / no
|
||||
# per-tool timeout" and is never treated as expired.
|
||||
deadline_of: Dict[Any, List[Optional[float]]] = {}
|
||||
timeout_of: Dict[Any, float] = {}
|
||||
timeout_triggered = False
|
||||
try:
|
||||
# One executor for the whole batch. Each future gets its own cancel
|
||||
# event (keyed by Future, not by tool name, so two parallel calls to the
|
||||
# same tool cannot shadow each other) and its own per-tool deadline.
|
||||
for tc, per_tool_timeout in plan:
|
||||
cancel_event = threading.Event()
|
||||
ctx = contextvars.copy_context()
|
||||
ctx.run(TOOL_CANCEL_EVENT.set, cancel_event)
|
||||
deadline_holder: List[Optional[float]] = [None]
|
||||
fut = pool.submit(
|
||||
ctx.run, _exec_with_deadline, tc, per_tool_timeout, deadline_holder,
|
||||
)
|
||||
logger.warning("Tool batch timed out after %s at step %d", timeout_label, step)
|
||||
for future, tc_item in futures.items():
|
||||
if future in pending:
|
||||
future.cancel()
|
||||
result_str = json.dumps({
|
||||
"error": f"Tool execution timed out after {timeout_label}",
|
||||
"timeout": True,
|
||||
})
|
||||
if progress_callback:
|
||||
progress_callback(stream_event(
|
||||
"tool_done",
|
||||
step=step,
|
||||
tool=tc_item.name,
|
||||
success=False,
|
||||
duration=round(tool_wait_timeout_seconds or 0.0, 2),
|
||||
))
|
||||
tool_calls_log.append({
|
||||
"step": step,
|
||||
"tool": tc_item.name,
|
||||
"arguments": tc_item.arguments,
|
||||
"success": False,
|
||||
"duration": round(tool_wait_timeout_seconds or 0.0, 2),
|
||||
"result_length": len(result_str),
|
||||
"cached": False,
|
||||
"timeout": True,
|
||||
})
|
||||
results.append({"tc": tc_item, "result_str": result_str})
|
||||
finally:
|
||||
pool.shutdown(wait=not timeout_triggered, cancel_futures=timeout_triggered)
|
||||
futures[fut] = tc
|
||||
cancel_of[fut] = cancel_event
|
||||
timeout_of[fut] = per_tool_timeout or 0.0
|
||||
deadline_of[fut] = deadline_holder
|
||||
|
||||
batch_deadline = (
|
||||
time.monotonic() + tool_wait_timeout_seconds
|
||||
if tool_wait_timeout_seconds and tool_wait_timeout_seconds > 0
|
||||
else None
|
||||
)
|
||||
pending = set(futures)
|
||||
|
||||
while pending:
|
||||
now = time.monotonic()
|
||||
deadlines = []
|
||||
for f in pending:
|
||||
holder = deadline_of.get(f)
|
||||
if holder is not None and holder[0] is not None:
|
||||
deadlines.append(holder[0])
|
||||
if batch_deadline is not None:
|
||||
deadlines.append(batch_deadline)
|
||||
next_deadline = min(deadlines) if deadlines else None
|
||||
if next_deadline is not None:
|
||||
wait_timeout = max(0.0, next_deadline - now)
|
||||
elif any(timeout_of.get(f, 0.0) > 0 for f in pending):
|
||||
# A pending call has a per-tool timeout but its worker has not
|
||||
# started yet (it is queued behind the capped pool). Poll briefly
|
||||
# so a tool that starts and then hangs still gets its own timeout
|
||||
# (review OR-COM-3d6b61f8) instead of being forgotten.
|
||||
wait_timeout = 0.01
|
||||
else:
|
||||
# No deadline and no pending call has a timeout — nothing can time
|
||||
# out, so block until a call completes (efficient for the default
|
||||
# no-timeout path).
|
||||
wait_timeout = None
|
||||
done, _ = wait(pending, timeout=wait_timeout, return_when=FIRST_COMPLETED)
|
||||
for fut in done:
|
||||
pending.discard(fut)
|
||||
_record(futures[fut], timed_out=False, out=fut.result())
|
||||
|
||||
now = time.monotonic()
|
||||
for fut in list(pending):
|
||||
holder = deadline_of.get(fut)
|
||||
deadline = holder[0] if holder is not None else None
|
||||
expired = (deadline is not None and now >= deadline) or (
|
||||
batch_deadline is not None and now >= batch_deadline
|
||||
)
|
||||
if not expired:
|
||||
continue
|
||||
pending.discard(fut)
|
||||
cancel_of[fut].set()
|
||||
timeout_triggered = True
|
||||
timeout_s = timeout_of.get(fut) or (tool_wait_timeout_seconds or 0.0)
|
||||
logger.warning(
|
||||
"Tool '%s' timed out after %.2fs at step %d",
|
||||
futures[fut].name, timeout_s, step,
|
||||
)
|
||||
_record(futures[fut], timed_out=True, timeout_s=timeout_s)
|
||||
finally:
|
||||
# Do not wait when a timeout fired — the timed-out handler may still be
|
||||
# running in a worker and would otherwise block this call indefinitely.
|
||||
pool.shutdown(wait=not timeout_triggered, cancel_futures=True)
|
||||
|
||||
return results
|
||||
|
||||
@@ -21,6 +21,39 @@ from src.agent.tools.registry import ToolRegistry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cooperative cancellation signal (best-effort timeouts, Issue #1890 / review)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python cannot forcibly stop an already-started tool thread, so when a tool
|
||||
# times out its handler may keep running in the background. To give long or
|
||||
# side-effecting handlers a way to honour the timeout, the runner arms a
|
||||
# per-call ``threading.Event`` on timeout and publishes it through this
|
||||
# contextvar for the duration of the call. Handlers MAY poll
|
||||
# ``is_tool_cancellation_requested`` and abort early; the same signal is also
|
||||
# honoured by the existing ``check_tool_execution()`` checkpoint that data /
|
||||
# backtest / tool-surface handlers already call, so a timed-out handler stops at
|
||||
# its next safe boundary even without opting into the helper. The signal is
|
||||
# strictly opt-in by checkpoint: handlers that never call either are completely
|
||||
# unaffected, and the runner only sets the event after a timeout has already
|
||||
# fired — never during normal completion. This is the "in-handler cooperative
|
||||
# cancel" mitigation requested in review, layered on top of the non-retriable
|
||||
# cache (which blocks the LLM from re-launching the same call).
|
||||
TOOL_CANCEL_EVENT: "contextvars.ContextVar[Optional[threading.Event]]" = contextvars.ContextVar(
|
||||
"tool_cancel_event", default=None
|
||||
)
|
||||
|
||||
|
||||
def is_tool_cancellation_requested() -> bool:
|
||||
"""Return True when the current tool call has been asked to cancel.
|
||||
|
||||
Tool handlers may call this cheaply inside long loops or before performing
|
||||
an irreversible side effect, to honour a best-effort timeout. Returns False
|
||||
unless the runner has armed the cancellation event for the running call.
|
||||
"""
|
||||
event = TOOL_CANCEL_EVENT.get()
|
||||
return event is not None and event.is_set()
|
||||
|
||||
|
||||
_SUMMARY_LIMIT = 500
|
||||
_TOKEN_PATTERN = re.compile(
|
||||
r"(?i)\b(?:sk|pk|ghp|gho|github_pat|xox[baprs]?|bearer)[-_a-z0-9]{12,}\b"
|
||||
@@ -100,12 +133,20 @@ def reset_tool_execution_context(token: contextvars.Token) -> None:
|
||||
def check_tool_execution() -> None:
|
||||
"""Stop at a safe handler boundary when cancellation or deadline is reached."""
|
||||
context = _ACTIVE_TOOL_CONTEXT.get()
|
||||
if context is None:
|
||||
return
|
||||
if context.cancel_event is not None and context.cancel_event.is_set():
|
||||
raise ToolExecutionCancelled("Tool execution was cancelled")
|
||||
if context.deadline is not None and time.monotonic() >= context.deadline:
|
||||
raise ToolExecutionDeadlineExceeded("Tool execution deadline was exceeded")
|
||||
if context is not None:
|
||||
if context.cancel_event is not None and context.cancel_event.is_set():
|
||||
raise ToolExecutionCancelled("Tool execution was cancelled")
|
||||
if context.deadline is not None and time.monotonic() >= context.deadline:
|
||||
raise ToolExecutionDeadlineExceeded("Tool execution deadline was exceeded")
|
||||
# Runner-armed cooperative cancel (Issue #1890): when a per-tool timeout
|
||||
# fires, the runner arms ``TOOL_CANCEL_EVENT`` for the running call. Real
|
||||
# tool handlers (data/backtest/tool-surface) poll this checkpoint rather than
|
||||
# the opt-in ``is_tool_cancellation_requested()`` helper, so honour the signal
|
||||
# here too — a still-running handler then aborts before its next side-effecting
|
||||
# step instead of running to completion in the background thread.
|
||||
cancel_event = TOOL_CANCEL_EVENT.get()
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
raise ToolExecutionCancelled("Tool execution timed out (cooperative cancel)")
|
||||
|
||||
|
||||
def serialize_tool_result(result: Any) -> str:
|
||||
|
||||
@@ -44,6 +44,7 @@ class ToolPolicy:
|
||||
policy_status: str = "unknown"
|
||||
scope_dimensions: List[str] = field(default_factory=list)
|
||||
cancellation_safe: bool = False
|
||||
timeout_seconds: Optional[float] = None
|
||||
|
||||
@classmethod
|
||||
def unknown(cls) -> "ToolPolicy":
|
||||
@@ -58,6 +59,7 @@ class ToolPolicy:
|
||||
permissions: Optional[List[str]] = None,
|
||||
scope_dimensions: Optional[List[str]] = None,
|
||||
cancellation_safe: bool = False,
|
||||
timeout_seconds: Optional[float] = None,
|
||||
) -> "ToolPolicy":
|
||||
return cls(
|
||||
read_only=read_only,
|
||||
@@ -66,6 +68,7 @@ class ToolPolicy:
|
||||
policy_status="declared",
|
||||
scope_dimensions=list(scope_dimensions or []),
|
||||
cancellation_safe=bool(cancellation_safe),
|
||||
timeout_seconds=timeout_seconds,
|
||||
)
|
||||
|
||||
def to_public_dict(self) -> Dict[str, Any]:
|
||||
@@ -85,8 +88,9 @@ class ToolDefinition:
|
||||
description: str
|
||||
parameters: List[ToolParameter]
|
||||
handler: Callable
|
||||
category: str = "data" # data | analysis | search | action
|
||||
category: str = "data" # data | analysis | search | action | market
|
||||
policy: ToolPolicy = field(default_factory=ToolPolicy.unknown)
|
||||
timeout_seconds: Optional[float] = None # Optional per-tool execution timeout
|
||||
|
||||
# ----- Multi-provider schema converters -----
|
||||
|
||||
@@ -172,8 +176,9 @@ class ToolRegistry:
|
||||
registry.execute("get_realtime_quote", stock_code="600519")
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
def __init__(self, category_timeout_map: Optional[Dict[str, float]] = None):
|
||||
self._tools: Dict[str, ToolDefinition] = {}
|
||||
self._category_timeouts: Dict[str, float] = dict(category_timeout_map or {})
|
||||
|
||||
# ----- Registration -----
|
||||
|
||||
@@ -215,6 +220,24 @@ class ToolRegistry:
|
||||
def __contains__(self, name: str) -> bool:
|
||||
return name in self._tools
|
||||
|
||||
def category_default_timeout(self, category: str) -> Optional[float]:
|
||||
"""Return the configured default timeout (seconds) for a tool category.
|
||||
|
||||
Returns ``None`` when no default is configured, letting the caller fall
|
||||
back to the global ``tool_call_timeout_seconds`` budget.
|
||||
"""
|
||||
return self._category_timeouts.get(category)
|
||||
|
||||
@property
|
||||
def category_timeout_map(self) -> Dict[str, float]:
|
||||
"""Return a copy of the per-category default timeout map.
|
||||
|
||||
Used by :meth:`ToolRegistry` consumers that build a filtered registry
|
||||
(e.g. ``BaseAgent._filtered_registry``) so the category ceilings survive
|
||||
the subset copy (review OR-COM-7f3d3f5b).
|
||||
"""
|
||||
return dict(self._category_timeouts)
|
||||
|
||||
# ----- Schema generation -----
|
||||
|
||||
def to_openai_tools(self) -> List[dict]:
|
||||
@@ -315,6 +338,7 @@ def tool(
|
||||
parameters: Optional[List[ToolParameter]] = None,
|
||||
registry: Optional[ToolRegistry] = None,
|
||||
policy: Optional[ToolPolicy] = None,
|
||||
timeout_seconds: Optional[float] = None,
|
||||
):
|
||||
"""Decorator to register a function as an agent tool.
|
||||
|
||||
@@ -333,13 +357,29 @@ def tool(
|
||||
if params is None:
|
||||
params = _infer_parameters(func)
|
||||
|
||||
# Single source of truth for per-tool timeout is
|
||||
# ``ToolDefinition.timeout_seconds`` (the field ``runner`` actually
|
||||
# reads). If the caller supplied a ``timeout_seconds`` via the policy
|
||||
# but omitted the explicit ``@tool(timeout_seconds=...)`` argument, fold
|
||||
# the policy value in here so the two never diverge.
|
||||
effective_timeout = timeout_seconds
|
||||
resolved_policy = policy or ToolPolicy.unknown()
|
||||
if effective_timeout is None and getattr(resolved_policy, "timeout_seconds", None) is not None:
|
||||
effective_timeout = resolved_policy.timeout_seconds
|
||||
logger.debug(
|
||||
"Tool '%s': using ToolPolicy.timeout_seconds=%s as the effective per-tool timeout",
|
||||
name,
|
||||
effective_timeout,
|
||||
)
|
||||
|
||||
tool_def = ToolDefinition(
|
||||
name=name,
|
||||
description=description,
|
||||
parameters=params,
|
||||
handler=func,
|
||||
category=category,
|
||||
policy=policy or ToolPolicy.unknown(),
|
||||
policy=resolved_policy,
|
||||
timeout_seconds=effective_timeout,
|
||||
)
|
||||
|
||||
target_registry = registry or get_default_registry()
|
||||
|
||||
@@ -1014,6 +1014,13 @@ class Config:
|
||||
agent_decision_agent_timeout_s: float = 0
|
||||
agent_portfolio_agent_timeout_s: float = 0
|
||||
agent_skill_agent_timeout_s: float = 0
|
||||
# Per-category default timeouts for agent tool calls (seconds).
|
||||
# 0 / unset means "no category default" -> falls back to the global
|
||||
# tool_call_timeout_seconds budget.
|
||||
agent_data_tool_timeout_s: float = 0.0
|
||||
agent_search_tool_timeout_s: float = 0.0
|
||||
agent_analysis_tool_timeout_s: float = 0.0
|
||||
agent_action_tool_timeout_s: float = 0.0
|
||||
agent_skill_concurrency: int = 3
|
||||
agent_risk_override: bool = True # Allow risk agent to veto buy signals
|
||||
agent_deep_research_budget: int = 30000 # Max token budget for deep research
|
||||
@@ -1946,6 +1953,22 @@ class Config:
|
||||
os.getenv('AGENT_SKILL_AGENT_TIMEOUT_S'), 0,
|
||||
field_name='AGENT_SKILL_AGENT_TIMEOUT_S', minimum=0,
|
||||
),
|
||||
agent_data_tool_timeout_s=parse_env_float(
|
||||
os.getenv('AGENT_DATA_TOOL_TIMEOUT_S'), 0.0,
|
||||
field_name='AGENT_DATA_TOOL_TIMEOUT_S', minimum=0.0,
|
||||
),
|
||||
agent_search_tool_timeout_s=parse_env_float(
|
||||
os.getenv('AGENT_SEARCH_TOOL_TIMEOUT_S'), 0.0,
|
||||
field_name='AGENT_SEARCH_TOOL_TIMEOUT_S', minimum=0.0,
|
||||
),
|
||||
agent_analysis_tool_timeout_s=parse_env_float(
|
||||
os.getenv('AGENT_ANALYSIS_TOOL_TIMEOUT_S'), 0.0,
|
||||
field_name='AGENT_ANALYSIS_TOOL_TIMEOUT_S', minimum=0.0,
|
||||
),
|
||||
agent_action_tool_timeout_s=parse_env_float(
|
||||
os.getenv('AGENT_ACTION_TOOL_TIMEOUT_S'), 0.0,
|
||||
field_name='AGENT_ACTION_TOOL_TIMEOUT_S', minimum=0.0,
|
||||
),
|
||||
agent_skill_concurrency=parse_env_int(
|
||||
os.getenv('AGENT_SKILL_CONCURRENCY'),
|
||||
3,
|
||||
|
||||
@@ -313,9 +313,15 @@ class SystemConfigService:
|
||||
@staticmethod
|
||||
def _reload_runtime_singletons() -> None:
|
||||
"""Reset runtime singleton services after config reload."""
|
||||
from src.agent.factory import reset_tool_registry
|
||||
from src.agent.tools.data_tools import reset_fetcher_manager
|
||||
from src.search_service import reset_search_service
|
||||
|
||||
# Drop the module-level ToolRegistry so the next
|
||||
# ``build_agent_executor`` / ``build_agent_chat_executor`` call
|
||||
# rebuilds ``_TOOL_REGISTRY`` against the fresh Config and picks up
|
||||
# new ``AGENT_*_TOOL_TIMEOUT_S`` overrides (Issue #1890).
|
||||
reset_tool_registry()
|
||||
reset_fetcher_manager()
|
||||
reset_search_service()
|
||||
|
||||
|
||||
1083
tests/agent/test_tool_timeout.py
Normal file
1083
tests/agent/test_tool_timeout.py
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user