mirror of
https://github.com/ZhuLinsen/daily_stock_analysis
synced 2026-09-20 02:43:35 +08:00
* feat: add minimal agent trajectory metrics with golden samples * feat: add runnable agent trajectory eval entry with JSON and text reports * fix: treat unsuccessful AgentResult as a run failure in trajectory eval * fix: reject multi-arch executor and validate golden tools against the real registry
This commit is contained in:
subaoyan16
@@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||
> For user-friendly release highlights, see the [GitHub Releases](https://github.com/ZhuLinsen/daily_stock_analysis/releases) page.
|
||||
|
||||
## [Unreleased]
|
||||
- [新功能] 新增最小 Agent 轨迹评估入口 `evals/agent_trajectory/`(Refs #1956):纯函数指标层只消费真实 `tool_calls_log + AgentResult`,冻结最小指标契约(工具命中、冗余/缓存、失败/重试、总步数/max_steps),`run_eval.py` 经 `build_agent_executor` 真实执行并输出文本摘要 + 结构化 JSON 报告;评估为 reporter 非 gate,零 `src/` 改动
|
||||
|
||||
- [修复] 将 litellm 依赖窗口上界收敛到 `<1.99.0`：1.99.0 起把 `prompt_cache_key` 透传给 OpenAI provider，破坏 provider 缓存测试对不透传行为的既有断言（CI backend-tests 3/3 与 backend-gate 失败）；保留历史最低版本与 `!=1.82.7`/`!=1.82.8` 事故排除，同时同步更新各 LLM 兼容文档中写死的依赖约束表述，避免文档与 requirements.txt 漂移
|
||||
|
||||
- [新功能] 新增 `SEARXNG_TIMEOUT_SECONDS` 配置自建 SearXNG 单次搜索超时（默认 10 秒），已接线全部 SearchService 构造入口（含题材搜索子进程重建）与默认 GitHub Actions 工作流
|
||||
|
||||
82
docs/agent-trajectory-eval.md
Normal file
82
docs/agent-trajectory-eval.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Agent Trajectory 评估(最小版,Refs #1956)
|
||||
|
||||
## 定位
|
||||
|
||||
`evals/agent_trajectory/` 提供一套**离线可运行的最小轨迹评估管线**:用真实 `tool_calls_log + AgentResult` 跑一个 golden 样例,输出结构化 JSON 报告与简短可读文本摘要。评估结果是 **reporter 而非 gate** —— 指标违规只反映在报告里,不会让进程失败,也不进入 CI 门禁。
|
||||
|
||||
- 指标层(`metrics.py`)是纯函数:只消费轨迹日志与 golden 样例,不 import `src/`,不触网、不调 LLM,可离线单测。
|
||||
- 入口(`run_eval.py`)通过 `build_agent_executor()` 构建真实执行器,与 `src/core/pipeline.py` 使用同一个执行捕获钩子,消费真实产物。
|
||||
- 入口只支持**单 agent 运行**(`AGENT_ARCH=single`,默认):`AGENT_ARCH=multi` 时 factory 返回 orchestrator,其轨迹是各 stage 局部步号的拼接、`total_steps` 为 stage 数,与单 runner 指标契约不兼容——入口在构建前明确报错并退出(退出码 1)。
|
||||
- 入口用真实工具注册表校验 golden:`expected_tools` 拼错或过期会被判为无效样例(退出码 1),而不是静默按低命中继续评分。
|
||||
- 本次冻结**最小指标契约**,股票 guard、Codex `arguments_summary` 等扩展语义明确留给后续 PR(见文末「不在范围」)。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
# 跑单个样例(真实 LLM,依赖本地已配置的模型)
|
||||
python evals/agent_trajectory/run_eval.py --sample 600519_technical
|
||||
|
||||
# 跑全部样例,并输出结构化 JSON(按样例 id 键控)
|
||||
python evals/agent_trajectory/run_eval.py --all --json-out eval_report.json
|
||||
```
|
||||
|
||||
参数:
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `--sample ID` / `--all` | 二选一必填:跑单个 golden 样例或全部 |
|
||||
| `--golden-path PATH` | 自定义 golden JSON 路径(默认模块旁 `golden_samples.json`) |
|
||||
| `--json-out PATH` | 写结构化 JSON 报告(`--all` 时为键控对象) |
|
||||
|
||||
退出码:`0` 运行成功(含违规);`1` golden 加载(含 `expected_tools` 不在真实工具注册表)/ 样例选择 / 工具注册表加载失败 / 执行器构建(含 `AGENT_ARCH=multi` 拒绝)/ 运行失败(含执行器返回 `success=false`,如 provider 未配置、LLM 错误、超时、max_steps 耗尽、dashboard 解析失败);`2` 用法错误。
|
||||
|
||||
## 冻结的最小指标契约
|
||||
|
||||
对每条 `tool_calls_log`(runner 契约:每项含 `step / tool / arguments / success / duration / result_length / cached`,可选 `timeout` / `guarded`),只统计:
|
||||
|
||||
| 指标 | 定义 |
|
||||
| --- | --- |
|
||||
| `expected_hit_rate` / `missing_expected` | 期望工具按**工具名**命中(本版不做股票维度判定);`expected_total` 为去重后的期望数 |
|
||||
| `optional_tools_used` | 期望集合之外实际调用的工具;`allow_optional_tools=false` 时记为违规 |
|
||||
| `redundant_calls` | 同一 (tool, args-key) 对在首次出现之后的每一次出现(不论成败)。args-key = `json.dumps(arguments, sort_keys=True, default=str)`,与运行时缓存键同思路 |
|
||||
| `retries` | 紧跟**失败**之后重试同一 (tool, args-key) 对;`retries ⊆ redundant_calls`。成功会清除该对的失败态:`fail → success → success` 只计 1 次 retry(后一次 success 仅计冗余) |
|
||||
| `failed_calls` | `success=false` 的条目数 |
|
||||
| `cached_calls` | `cached=true` 的条目数(runner 语义:复用不可重试的失败结果) |
|
||||
| `distinct_steps` / `max_steps_touched` | 日志 step 与 `AgentResult.total_steps` 取较大者(最后纯回答轮不产生工具调用,日志会低估);`max_steps_touched` 为 `max(step) >= allowed_max_steps` 的启发式 |
|
||||
|
||||
## Golden 样例 schema
|
||||
|
||||
`golden_samples.json` 是一个数组,字段:
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `id` | string | 唯一样例 id(必填) |
|
||||
| `task_description` | string | 交给执行器的任务文本(必填) |
|
||||
| `stock_code` | string | 原样传入 runner context(`{"stock_code": ...}`);空串 = 无 context;不参与打分 |
|
||||
| `expected_tools` | string[] | 期望工具名(必填,非空、无重复) |
|
||||
| `allowed_max_steps` | int | 步数预算启发式(默认 10,>= 1) |
|
||||
| `allow_optional_tools` | bool | 是否容忍期望外工具(默认 true) |
|
||||
|
||||
校验:加载路径(`load_golden_samples`)直接拒绝非法样例;直接构造路径(`compute_trajectory_metrics`)以 validator 完全相同的措辞逐条上报违规,两条路径的契约按构造保持一致。`known_tool_names` 可注入真实工具注册表做成员校验(metrics 层自身不 import `src/`);`run_eval.py` 入口会自动注入真实工具注册表。
|
||||
|
||||
## JSON 报告 schema
|
||||
|
||||
单个样例:
|
||||
|
||||
```json
|
||||
{
|
||||
"sample_id": "600519_technical",
|
||||
"task_description": "…",
|
||||
"stock_code": "600519",
|
||||
"metrics": { "expected_hit_rate": …, "expected_total": …, "…": "… 共 11 个字段" },
|
||||
"violations": ["…"]
|
||||
}
|
||||
```
|
||||
|
||||
`--all --json-out` 时外层为 `{sample_id: <上述对象>}` 键控对象。
|
||||
|
||||
## 不在范围(后续 PR)
|
||||
|
||||
- 股票维度命中判定与 guard 拦截语义(`guarded` / 越界调用违规)
|
||||
- Codex App Server 的 `arguments_summary` 方言识别
|
||||
- 任何 `.env` / 运行时配置与 CI 门禁(本 PR 零 `src/` 改动,不影响现有分析流程)
|
||||
6
evals/agent_trajectory/__init__.py
Normal file
6
evals/agent_trajectory/__init__.py
Normal file
@@ -0,0 +1,6 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Agent trajectory evaluation package (Issue #1956).
|
||||
|
||||
Pure-function metrics layer that scores a real ``tool_calls_log`` against
|
||||
golden samples; the runnable entry point lives in ``run_eval.py``.
|
||||
"""
|
||||
18
evals/agent_trajectory/golden_samples.json
Normal file
18
evals/agent_trajectory/golden_samples.json
Normal file
@@ -0,0 +1,18 @@
|
||||
[
|
||||
{
|
||||
"id": "600519_technical",
|
||||
"task_description": "分析贵州茅台（600519）近期技术面走势，输出趋势判断",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote", "get_daily_history", "analyze_trend"],
|
||||
"allowed_max_steps": 8,
|
||||
"allow_optional_tools": true
|
||||
},
|
||||
{
|
||||
"id": "000001_core_data_strict",
|
||||
"task_description": "获取平安银行（000001）核心行情数据，不做扩展检索",
|
||||
"stock_code": "000001",
|
||||
"expected_tools": ["get_realtime_quote", "get_daily_history"],
|
||||
"allowed_max_steps": 6,
|
||||
"allow_optional_tools": false
|
||||
}
|
||||
]
|
||||
421
evals/agent_trajectory/metrics.py
Normal file
421
evals/agent_trajectory/metrics.py
Normal file
@@ -0,0 +1,421 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Pure-function trajectory metrics for agent evaluation (Issue #1956).
|
||||
|
||||
This module computes quality metrics for an agent execution trajectory from its
|
||||
``tool_calls_log`` (see ``src/agent/runner.py`` for the producer contract):
|
||||
|
||||
* each entry carries ``step / tool / arguments / success / duration /
|
||||
result_length / cached`` and optionally ``timeout`` or ``guarded`` fields;
|
||||
* entries with missing optional fields are tolerated with defaults.
|
||||
|
||||
The scoring functions in this module are pure: ``compute_trajectory_metrics``,
|
||||
``format_text_report`` and ``validate_golden_sample`` consume plain data (a log
|
||||
list and a ``GoldenSample``) and never touch the filesystem, network, or LLM.
|
||||
The one exception is the loader ``load_golden_samples``, which reads the golden
|
||||
JSON file from disk. This keeps the metrics layer deterministic, unit-testable
|
||||
without an API key, and free of ``src/`` imports — it can score trajectories
|
||||
from any source. The runnable entry point that produces real trajectories and
|
||||
feeds them into this layer lives in ``run_eval.py``.
|
||||
|
||||
Both entry paths enforce the same golden-sample structure contract:
|
||||
``validate_golden_sample`` (the loader path) rejects malformed samples
|
||||
outright, while ``compute_trajectory_metrics`` (the direct-construction path)
|
||||
excludes the invalid parts from scoring, reports its own scoring-level
|
||||
violations with the validator's wording, and finally appends *every* issue
|
||||
``validate_golden_sample`` reports, verbatim and deduplicated — a caller who
|
||||
builds a ``GoldenSample`` by hand can never get a silently relaxed result,
|
||||
and a new validator check applies to direct scoring automatically (see the
|
||||
compute docstring for the exact mapping).
|
||||
|
||||
Idempotency key contract
|
||||
------------------------
|
||||
Two log entries are considered "the same call" when their ``tool`` names are
|
||||
equal and their serialized ``arguments`` are equal. Arguments may contain
|
||||
unhashable values (dict / list), so the key is built with
|
||||
``json.dumps(arguments, sort_keys=True, default=str)`` — a *stable string*,
|
||||
not a hash. Do not replace this with ``tuple(arguments)`` or ``repr()``:
|
||||
insertion order or collection type would then change call identity and corrupt
|
||||
redundancy / retry counts.
|
||||
|
||||
Metric semantics
|
||||
----------------
|
||||
* ``redundant_calls``: every occurrence of a (tool, args-key) pair beyond its
|
||||
first — regardless of success.
|
||||
* ``retries``: occurrences that follow a *failed* occurrence of the same pair
|
||||
(i.e. "tried again after a failure"). ``retries`` is a subset of
|
||||
``redundant_calls``; repeats after success count as redundant but not retry,
|
||||
and a success clears the pair's failure state — so ``fail -> success ->
|
||||
success`` counts exactly one retry, not two.
|
||||
* ``failed_calls``: entries with ``success=False``.
|
||||
* ``cached_calls``: entries with ``cached=True`` (runner semantics: reuse of a
|
||||
non-retriable failure result).
|
||||
* ``max_steps_touched``: the log does not carry ``max_steps`` itself, so this
|
||||
is the conservative heuristic ``max(step) >= golden.allowed_max_steps`` —
|
||||
a proxy for "the run reached the step budget", not proof of the loop
|
||||
exhausting it. When ``total_steps`` is supplied (see
|
||||
:func:`compute_trajectory_metrics`), the larger of ``total_steps`` and the
|
||||
log-derived step is compared instead — the final answer round consumes a
|
||||
step but produces no tool call, so the log alone understates consumption.
|
||||
|
||||
Scope
|
||||
-----
|
||||
This is the frozen minimal contract requested for the close-and-rebuild PR
|
||||
(Refs #1956): tool hit, redundancy / caching, failure / retry and total
|
||||
steps / max_steps only. Stock guard and Codex ``arguments_summary``
|
||||
semantics are out of scope here and belong in follow-up PRs (see
|
||||
``docs/agent-trajectory-eval.md``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, fields
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class GoldenSample:
|
||||
"""Expected trajectory for one evaluation task.
|
||||
|
||||
``expected_tools`` are the tool names the agent should call; tools outside
|
||||
this set are tolerated only when ``allow_optional_tools`` is true.
|
||||
``stock_code`` is carried verbatim into the runner context by
|
||||
``run_eval.py`` (empty string = no stock context) and does not take part
|
||||
in scoring.
|
||||
"""
|
||||
|
||||
id: str
|
||||
task_description: str
|
||||
expected_tools: List[str]
|
||||
stock_code: str = ""
|
||||
allowed_max_steps: int = 10
|
||||
allow_optional_tools: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class TrajectoryMetrics:
|
||||
"""All metrics computed for one trajectory against one golden sample."""
|
||||
|
||||
expected_hit_rate: float
|
||||
expected_total: int
|
||||
missing_expected: List[str]
|
||||
optional_tools_used: List[str]
|
||||
redundant_calls: int
|
||||
cached_calls: int
|
||||
failed_calls: int
|
||||
retries: int
|
||||
distinct_steps: int
|
||||
max_steps_touched: bool
|
||||
violations: List[str]
|
||||
|
||||
|
||||
def _args_key(arguments: Any) -> str:
|
||||
"""Return a stable idempotency key for tool-call arguments (see module docstring).
|
||||
|
||||
``None`` (an entry without an ``arguments`` payload) is serialized as an
|
||||
empty object, so arguments-less calls of one tool share a single identity.
|
||||
Non-dict payloads are serialized as-is.
|
||||
"""
|
||||
if arguments is None:
|
||||
arguments = {}
|
||||
return json.dumps(arguments, ensure_ascii=False, sort_keys=True, default=str)
|
||||
|
||||
|
||||
def _entry_arguments(entry: Dict[str, Any]) -> Any:
|
||||
"""Extract the idempotent argument payload from a log entry.
|
||||
|
||||
Runner entries carry ``arguments`` (a dict); entries without the field
|
||||
key as ``None``, which ``_args_key`` treats as an empty object.
|
||||
"""
|
||||
return entry.get("arguments")
|
||||
|
||||
|
||||
def _coerce_step(value: Any) -> int:
|
||||
"""Coerce a log entry's ``step`` to a non-negative int (missing/odd -> 0)."""
|
||||
try:
|
||||
step = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
return step if step > 0 else 0
|
||||
|
||||
|
||||
def compute_trajectory_metrics(
|
||||
log: List[Dict[str, Any]],
|
||||
golden: GoldenSample,
|
||||
total_steps: Optional[int] = None,
|
||||
) -> TrajectoryMetrics:
|
||||
"""Compute all trajectory metrics from a ``tool_calls_log`` and a golden sample.
|
||||
|
||||
``total_steps`` optionally carries the number of loop rounds the run
|
||||
actually consumed (``AgentResult.total_steps``), which includes the
|
||||
final plain-answer round that produces no tool call. When it is larger
|
||||
than the log-derived step count it is used for ``distinct_steps`` and the
|
||||
``max_steps_touched`` heuristic; otherwise the log alone decides, and the
|
||||
default ``None`` keeps the log-only behaviour.
|
||||
|
||||
Direct construction of a hand-edited ``GoldenSample`` is held to the same
|
||||
structure contract as :func:`validate_golden_sample` on the loader path:
|
||||
malformed parts are excluded from scoring instead of silently reshaping
|
||||
the result, scoring-level violations are reported with the validator's
|
||||
wording, and every issue the validator reports is appended verbatim
|
||||
(deduplicated) before returning. Malformed ``expected_tools`` elements —
|
||||
non-strings, empty strings and whitespace-only strings, the validator's
|
||||
exact predicate — are dropped with an explicit violation, and a
|
||||
non-positive ``allowed_max_steps`` is reported with the validator's
|
||||
wording (and the budget assertion stays disabled).
|
||||
"""
|
||||
used_tools: List[str] = []
|
||||
key_counts: Dict[tuple, int] = {}
|
||||
key_failed_seen: Dict[tuple, bool] = {}
|
||||
key_retries: Dict[tuple, int] = {}
|
||||
# Extract the expected tool list before scanning entries. Malformed
|
||||
# elements are not silently dropped: they are reported as a violation
|
||||
# below (mirroring validate_golden_sample, which rejects them at load
|
||||
# time) and only the valid names take part in scoring.
|
||||
if isinstance(golden.expected_tools, list):
|
||||
# Same element predicate as validate_golden_sample(): whitespace-only
|
||||
# strings count as malformed too, not just falsy ones.
|
||||
expected_tools_malformed = [t for t in golden.expected_tools if not isinstance(t, str) or not t.strip()]
|
||||
expected = [t for t in golden.expected_tools if isinstance(t, str) and t.strip()]
|
||||
else:
|
||||
# Defend against hand-edited samples passing a bare string:
|
||||
# validation rejects it at load time, but scoring must not misparse
|
||||
# it into per-character tool names either.
|
||||
expected_tools_malformed = []
|
||||
expected = []
|
||||
# A hand-edited sample may repeat a tool name; normalize to first
|
||||
# occurrences before scoring so the hit rate cannot be inflated
|
||||
# (["quote", "quote"] with one quote call must read 1/2, not 2/3).
|
||||
expected_dupes = len(set(expected)) != len(expected)
|
||||
if expected_dupes:
|
||||
expected = list(dict.fromkeys(expected))
|
||||
failed_calls = 0
|
||||
cached_calls = 0
|
||||
redundant_calls = 0
|
||||
distinct_steps = 0
|
||||
max_step = 0
|
||||
seen_steps: set = set()
|
||||
|
||||
for entry in log:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
tool = entry.get("tool") or ""
|
||||
success = bool(entry.get("success", True))
|
||||
if tool and tool not in used_tools:
|
||||
used_tools.append(tool)
|
||||
step = _coerce_step(entry.get("step"))
|
||||
if step and step not in seen_steps:
|
||||
seen_steps.add(step)
|
||||
distinct_steps += 1
|
||||
max_step = max(max_step, step)
|
||||
if not success:
|
||||
failed_calls += 1
|
||||
if entry.get("cached"):
|
||||
cached_calls += 1
|
||||
|
||||
key = (tool, _args_key(_entry_arguments(entry)))
|
||||
if key_counts.get(key, 0):
|
||||
redundant_calls += 1
|
||||
key_counts[key] = key_counts.get(key, 0) + 1
|
||||
# An occurrence is a retry only when the same call already failed
|
||||
# before it (see module docstring for the precise contract).
|
||||
if key_failed_seen.get(key):
|
||||
key_retries[key] = key_retries.get(key, 0) + 1
|
||||
# A success clears the failure state: repeats after a recovery count
|
||||
# as redundant only, not as further retries.
|
||||
key_failed_seen[key] = not success
|
||||
|
||||
retries = sum(key_retries.values())
|
||||
violations: List[str] = []
|
||||
|
||||
if expected_dupes:
|
||||
violations.append("expected_tools must not contain duplicate names")
|
||||
if expected_tools_malformed:
|
||||
violations.append("expected_tools must contain only non-empty strings")
|
||||
missing_expected = [t for t in expected if t not in used_tools]
|
||||
expected_hit_rate = (len(expected) - len(missing_expected)) / len(expected) if expected else 0.0
|
||||
optional_tools_used = [t for t in used_tools if t not in expected]
|
||||
|
||||
if not isinstance(golden.expected_tools, list):
|
||||
violations.append("expected_tools must be a list of tool names")
|
||||
elif not golden.expected_tools:
|
||||
violations.append("expected_tools must be a non-empty list")
|
||||
|
||||
# Malformed golden samples must not crash scoring nor flip semantics:
|
||||
# a truthy string like "false" must not silently turn a strict sample
|
||||
# permissive, and a non-integer step limit must not crash the comparison.
|
||||
optional_allowed = golden.allow_optional_tools
|
||||
if not isinstance(optional_allowed, bool):
|
||||
violations.append("allow_optional_tools must be a boolean")
|
||||
optional_allowed = False
|
||||
if optional_tools_used and not optional_allowed:
|
||||
violations.append(f"optional tools used but not allowed: {', '.join(optional_tools_used)}")
|
||||
|
||||
# The final answer round consumes a step but produces no tool call, so
|
||||
# when the caller supplies the run's real total it may exceed the log.
|
||||
total = 0
|
||||
if total_steps is not None:
|
||||
try:
|
||||
total = int(total_steps)
|
||||
except (TypeError, ValueError):
|
||||
total = 0
|
||||
total = total if total > 0 else 0
|
||||
distinct_steps = max(distinct_steps, total)
|
||||
max_step = max(max_step, total)
|
||||
|
||||
limit = golden.allowed_max_steps
|
||||
if isinstance(limit, bool) or not isinstance(limit, int):
|
||||
violations.append("allowed_max_steps must be an integer")
|
||||
limit = 0
|
||||
elif limit < 1:
|
||||
# Validator wording for the same field: a non-positive limit must
|
||||
# not silently disable the budget assertion on the direct-score path.
|
||||
violations.append("allowed_max_steps must be >= 1")
|
||||
limit = 0
|
||||
max_steps_touched = bool(max_step and limit > 0 and max_step >= limit)
|
||||
if max_steps_touched:
|
||||
violations.append(f"trajectory reached allowed_max_steps ({golden.allowed_max_steps})")
|
||||
|
||||
# The direct-score path surfaces the validator's complete structure
|
||||
# contract, not only the checks scoring itself depends on: every issue
|
||||
# validate_golden_sample() reports is appended verbatim (deduplicated
|
||||
# against the inline violations above), so a hand-built malformed golden
|
||||
# can never score as valid — and a new validator check applies to direct
|
||||
# scoring automatically. The two entry paths stay aligned by
|
||||
# construction instead of by convention.
|
||||
for issue in validate_golden_sample(golden):
|
||||
if issue not in violations:
|
||||
violations.append(issue)
|
||||
|
||||
return TrajectoryMetrics(
|
||||
expected_hit_rate=expected_hit_rate,
|
||||
expected_total=len(expected),
|
||||
missing_expected=missing_expected,
|
||||
optional_tools_used=optional_tools_used,
|
||||
redundant_calls=redundant_calls,
|
||||
cached_calls=cached_calls,
|
||||
failed_calls=failed_calls,
|
||||
retries=retries,
|
||||
distinct_steps=distinct_steps,
|
||||
max_steps_touched=max_steps_touched,
|
||||
violations=violations,
|
||||
)
|
||||
|
||||
|
||||
def format_text_report(m: TrajectoryMetrics) -> str:
|
||||
"""Render metrics as a deterministic, human-readable text report."""
|
||||
hit_count = max(0, m.expected_total - len(m.missing_expected))
|
||||
hit_percent = f"{m.expected_hit_rate * 100:.1f}%"
|
||||
missing = ", ".join(m.missing_expected) if m.missing_expected else "无"
|
||||
optional = ", ".join(m.optional_tools_used) if m.optional_tools_used else "无"
|
||||
violations = "; ".join(m.violations) if m.violations else "无"
|
||||
max_steps_label = "是" if m.max_steps_touched else "否"
|
||||
return (
|
||||
"============================================\n"
|
||||
"Agent Trajectory 评估报告\n"
|
||||
"============================================\n"
|
||||
f"- 期望工具命中: {hit_count}/{m.expected_total} ({hit_percent})\n"
|
||||
f"- 缺失期望工具: {missing}\n"
|
||||
f"- 期望外工具: {optional}\n"
|
||||
f"- 冗余调用: {m.redundant_calls} | 缓存调用: {m.cached_calls} | "
|
||||
f"失败调用: {m.failed_calls} | 重试: {m.retries}\n"
|
||||
f"- 消耗步数: {m.distinct_steps} (触碰 max_steps: {max_steps_label})\n"
|
||||
f"- 违规项: {violations}\n"
|
||||
)
|
||||
|
||||
|
||||
def load_golden_samples(
|
||||
path: Optional[str] = None,
|
||||
known_tool_names: Optional[Iterable[str]] = None,
|
||||
) -> List[GoldenSample]:
|
||||
"""Load golden samples from ``path`` (default: ``golden_samples.json`` next to this module).
|
||||
|
||||
Raises ``FileNotFoundError`` when the file is missing and ``ValueError`` on
|
||||
malformed JSON or structural issues (see :func:`validate_golden_sample`).
|
||||
Unknown extra JSON keys are ignored so the file can carry forward-looking
|
||||
metadata without breaking the loader.
|
||||
"""
|
||||
if path is None:
|
||||
path = str(Path(__file__).with_name("golden_samples.json"))
|
||||
data = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
if not isinstance(data, list):
|
||||
raise ValueError(f"golden samples file must contain a JSON list, got {type(data).__name__}")
|
||||
|
||||
# Materialize once before the loop: a one-shot generator must survive the
|
||||
# validation of every sample, not just the first.
|
||||
known = set(known_tool_names) if known_tool_names is not None else None
|
||||
golden_fields = {f.name for f in fields(GoldenSample)}
|
||||
samples: List[GoldenSample] = []
|
||||
seen_ids: set = set()
|
||||
for index, item in enumerate(data):
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError(f"sample #{index} must be a JSON object, got {type(item).__name__}")
|
||||
try:
|
||||
sample = GoldenSample(**{k: v for k, v in item.items() if k in golden_fields})
|
||||
except TypeError as exc:
|
||||
raise ValueError(f"sample #{index} has invalid fields: {exc}") from exc
|
||||
# Structural validation runs before duplicate detection so that a
|
||||
# mistyped (possibly unhashable) id is rejected as a ValueError here
|
||||
# instead of crashing the membership check below.
|
||||
issues = validate_golden_sample(sample, known)
|
||||
if issues:
|
||||
raise ValueError(f"sample '{sample.id}': " + "; ".join(issues))
|
||||
if sample.id in seen_ids:
|
||||
raise ValueError(f"duplicate sample id: {sample.id}")
|
||||
seen_ids.add(sample.id)
|
||||
samples.append(sample)
|
||||
return samples
|
||||
|
||||
|
||||
def validate_golden_sample(
|
||||
sample: GoldenSample,
|
||||
known_tool_names: Optional[Iterable[str]] = None,
|
||||
) -> List[str]:
|
||||
"""Return a list of structural issues for ``sample``; empty list means valid.
|
||||
|
||||
When ``known_tool_names`` is provided, ``expected_tools`` must be a subset
|
||||
of it; the caller supplies the authoritative registry names (this module
|
||||
deliberately does not import ``src/``). Any ``Iterable[str]`` is accepted
|
||||
— including one-shot generators — and materialized once internally, so
|
||||
membership checks never consume the caller's iterable.
|
||||
|
||||
Field *types* are part of the structural contract — hand-edited golden
|
||||
JSON must fail with a clear message instead of crashing or silently
|
||||
passing: text fields must be strings, ``expected_tools`` must be a list of
|
||||
non-empty, duplicate-free names, ``allowed_max_steps`` an integer >= 1 and
|
||||
``allow_optional_tools`` a boolean. ``stock_code`` must be a string; the
|
||||
empty string is the documented default ("no stock context", see
|
||||
``run_eval.py``) and passes, while whitespace-only values fail.
|
||||
"""
|
||||
issues: List[str] = []
|
||||
known = set(known_tool_names) if known_tool_names is not None else None
|
||||
if not isinstance(sample.id, str) or not sample.id.strip():
|
||||
issues.append("id must be a non-empty string")
|
||||
if not isinstance(sample.task_description, str) or not sample.task_description.strip():
|
||||
issues.append("task_description must be a non-empty string")
|
||||
if not isinstance(sample.stock_code, str):
|
||||
issues.append("stock_code must be a string")
|
||||
elif sample.stock_code != sample.stock_code.strip():
|
||||
issues.append("stock_code must not contain leading or trailing whitespace")
|
||||
if not isinstance(sample.expected_tools, list):
|
||||
issues.append("expected_tools must be a list of tool names")
|
||||
elif not sample.expected_tools:
|
||||
issues.append("expected_tools must be a non-empty list")
|
||||
elif any(not isinstance(t, str) or not t.strip() for t in sample.expected_tools):
|
||||
issues.append("expected_tools must contain only non-empty strings")
|
||||
elif len(set(sample.expected_tools)) != len(sample.expected_tools):
|
||||
issues.append("expected_tools must not contain duplicate names")
|
||||
elif known is not None:
|
||||
# Only reachable when expected_tools is a non-empty list of non-empty
|
||||
# strings, so malformed values can never crash the membership check.
|
||||
unknown = [t for t in sample.expected_tools if t not in known]
|
||||
if unknown:
|
||||
issues.append(f"unknown expected_tools: {', '.join(unknown)}")
|
||||
if isinstance(sample.allowed_max_steps, bool) or not isinstance(sample.allowed_max_steps, int):
|
||||
issues.append("allowed_max_steps must be an integer")
|
||||
elif sample.allowed_max_steps < 1:
|
||||
issues.append("allowed_max_steps must be >= 1")
|
||||
if not isinstance(sample.allow_optional_tools, bool):
|
||||
issues.append("allow_optional_tools must be a boolean")
|
||||
return issues
|
||||
211
evals/agent_trajectory/run_eval.py
Normal file
211
evals/agent_trajectory/run_eval.py
Normal file
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Run one agent-trajectory eval sample against the real agent executor (Issue #1956).
|
||||
|
||||
Consumes a real ``tool_calls_log + AgentResult`` produced by
|
||||
``src.agent.factory.build_agent_executor`` (the same capture hook the
|
||||
analysis pipeline uses in ``src/core/pipeline.py``), scores it with the
|
||||
pure metrics layer and emits a short human-readable text summary plus an
|
||||
optional structured JSON report.
|
||||
|
||||
The eval is a *reporter*, not a gate: metric violations lower the report,
|
||||
they never fail the process. Exit codes: 0 = ran (violations included),
|
||||
1 = load (golden / tool registry) / build / run failure, 2 = usage error.
|
||||
|
||||
The entry supports the single-agent runner only. When ``AGENT_ARCH=multi``
|
||||
the factory returns the orchestrator whose trajectories use per-stage local
|
||||
step numbers, which breaks the single-runner metric contract, so the entry
|
||||
rejects that arch up front with exit code 1. Golden samples are validated
|
||||
against the real tool registry before running, so misspelled or stale
|
||||
``expected_tools`` fail as invalid samples instead of scoring as low hit
|
||||
rate.
|
||||
|
||||
Usage:
|
||||
python evals/agent_trajectory/run_eval.py --sample 600519_technical
|
||||
python evals/agent_trajectory/run_eval.py --all --json-out eval_report.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
# Add the project root to sys.path so the script also works as
|
||||
# `python evals/agent_trajectory/run_eval.py` from anywhere.
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from evals.agent_trajectory.metrics import (
|
||||
GoldenSample,
|
||||
TrajectoryMetrics,
|
||||
compute_trajectory_metrics,
|
||||
format_text_report,
|
||||
load_golden_samples,
|
||||
)
|
||||
|
||||
|
||||
def _build_report(sample: GoldenSample, metrics: TrajectoryMetrics) -> Dict[str, Any]:
|
||||
"""Structured JSON report for one sample (schema in docs/agent-trajectory-eval.md)."""
|
||||
return {
|
||||
"sample_id": sample.id,
|
||||
"task_description": sample.task_description,
|
||||
"stock_code": sample.stock_code,
|
||||
"metrics": asdict(metrics),
|
||||
"violations": metrics.violations,
|
||||
}
|
||||
|
||||
|
||||
def _write_json(path: Optional[Path], payload: Any) -> None:
|
||||
"""Write ``payload`` as indented UTF-8 JSON (trailing newline)."""
|
||||
if path is None:
|
||||
return
|
||||
Path(path).write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
_KNOWN_TOOL_NAMES: Optional[set] = None
|
||||
|
||||
|
||||
def _check_agent_arch() -> None:
|
||||
"""Reject multi-agent arch up front (mirrors the factory's own decision).
|
||||
|
||||
``src.agent.factory.build_agent_executor`` returns the orchestrator when
|
||||
``config.agent_arch == "multi"``; its trajectories concatenate per-stage
|
||||
logs with local step numbering and set ``total_steps`` to the stage count,
|
||||
which the single-runner metric contract cannot interpret. Fail fast
|
||||
instead of scoring a distorted trajectory.
|
||||
"""
|
||||
from src.config import get_config
|
||||
|
||||
arch = getattr(get_config(), "agent_arch", "single")
|
||||
if arch == "multi":
|
||||
raise RuntimeError(
|
||||
"AGENT_ARCH=multi is not supported by this minimal eval: "
|
||||
"orchestrator trajectories use per-stage local step numbers, "
|
||||
"which break the single-runner metric contract"
|
||||
)
|
||||
|
||||
|
||||
def _known_tool_names():
|
||||
"""Authoritative tool names from the real registry modules (lazy, cached).
|
||||
|
||||
The metrics layer deliberately does not import ``src/``; the entry point
|
||||
supplies these names so ``load_golden_samples`` rejects misspelled or
|
||||
stale ``expected_tools`` instead of scoring them as low hit rate.
|
||||
"""
|
||||
global _KNOWN_TOOL_NAMES
|
||||
if _KNOWN_TOOL_NAMES is None:
|
||||
from src.agent.tools.analysis_tools import ALL_ANALYSIS_TOOLS
|
||||
from src.agent.tools.backtest_tools import ALL_BACKTEST_TOOLS
|
||||
from src.agent.tools.data_tools import ALL_DATA_TOOLS
|
||||
from src.agent.tools.market_tools import ALL_MARKET_TOOLS
|
||||
from src.agent.tools.search_tools import ALL_SEARCH_TOOLS
|
||||
|
||||
all_tools = ALL_DATA_TOOLS + ALL_ANALYSIS_TOOLS + ALL_SEARCH_TOOLS + ALL_MARKET_TOOLS + ALL_BACKTEST_TOOLS
|
||||
_KNOWN_TOOL_NAMES = {tool_def.name for tool_def in all_tools}
|
||||
return _KNOWN_TOOL_NAMES
|
||||
|
||||
|
||||
def _build_executor():
|
||||
"""Build the real agent executor (lazy import so tests can monkeypatch this)."""
|
||||
_check_agent_arch()
|
||||
from src.agent.factory import build_agent_executor
|
||||
|
||||
return build_agent_executor()
|
||||
|
||||
|
||||
def run_sample(executor, sample: GoldenSample, *, json_out: Optional[Path] = None) -> TrajectoryMetrics:
|
||||
"""Run one golden sample against a duck-typed agent executor and score it.
|
||||
|
||||
``executor`` only needs ``run(task, context=None) -> result`` where
|
||||
``result`` carries ``tool_calls_log`` (and optionally ``total_steps``) —
|
||||
the same shape as ``src.agent.executor.AgentResult``. The production
|
||||
executor is built lazily by :func:`_build_executor`; tests may pass a
|
||||
stub. A result carrying an explicit ``success=False`` is a run failure
|
||||
(the executor reported a provider / timeout / budget error) and raises
|
||||
``RuntimeError`` before any scoring; duck-typed results without a
|
||||
``success`` attribute are treated as successful. The text summary is
|
||||
always printed to stdout; ``json_out`` additionally writes the
|
||||
structured report for this sample.
|
||||
"""
|
||||
context: Optional[Dict[str, Any]] = None
|
||||
if sample.stock_code:
|
||||
context = {"stock_code": sample.stock_code}
|
||||
result = executor.run(sample.task_description, context=context)
|
||||
if getattr(result, "success", None) is False:
|
||||
error = getattr(result, "error", None)
|
||||
raise RuntimeError(f"agent run failed (success=false): {error or 'no error detail'}")
|
||||
log = getattr(result, "tool_calls_log", None) or []
|
||||
total_steps = getattr(result, "total_steps", None)
|
||||
metrics = compute_trajectory_metrics(log, sample, total_steps=total_steps)
|
||||
print(format_text_report(metrics))
|
||||
_write_json(json_out, _build_report(sample, metrics))
|
||||
return metrics
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
"""Run one golden sample (--sample ID) or every sample (--all)."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run one agent-trajectory eval sample against the real agent executor (Issue #1956).",
|
||||
)
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument("--sample", metavar="ID", help="run the golden sample with this id")
|
||||
group.add_argument("--all", action="store_true", help="run every golden sample")
|
||||
parser.add_argument(
|
||||
"--golden-path",
|
||||
default=None,
|
||||
help="path to golden_samples.json (default: the checked-in file next to this module)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--json-out",
|
||||
default=None,
|
||||
help="write a structured JSON report to this path (--all writes a keyed object)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
known = _known_tool_names()
|
||||
except Exception as exc:
|
||||
print(f"error: failed to load tool registry for golden validation: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
try:
|
||||
samples = load_golden_samples(path=args.golden_path, known_tool_names=known)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"error: failed to load golden samples: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.sample:
|
||||
selected = next((s for s in samples if s.id == args.sample), None)
|
||||
if selected is None:
|
||||
available = ", ".join(s.id for s in samples) if samples else "(none)"
|
||||
print(f"error: unknown sample id '{args.sample}'; available: {available}", file=sys.stderr)
|
||||
return 1
|
||||
samples = [selected]
|
||||
|
||||
try:
|
||||
executor = _build_executor()
|
||||
except Exception as exc: # pragma: no cover - exercised via monkeypatched failure
|
||||
print(f"error: failed to build agent executor: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
reports: Dict[str, Dict[str, Any]] = {}
|
||||
for sample in samples:
|
||||
print(f"[eval] sample: {sample.id} | task: {sample.task_description}")
|
||||
try:
|
||||
metrics = run_sample(executor, sample)
|
||||
except Exception as exc:
|
||||
print(f"error: sample '{sample.id}' failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
reports[sample.id] = _build_report(sample, metrics)
|
||||
|
||||
if args.json_out:
|
||||
_write_json(Path(args.json_out), reports if args.all else reports[args.sample])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
41
tests/fixtures/agent_trajectory/negative_600519_technical.json
vendored
Normal file
41
tests/fixtures/agent_trajectory/negative_600519_technical.json
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"total_steps": 8,
|
||||
"tool_calls_log": [
|
||||
{
|
||||
"step": 1,
|
||||
"tool": "get_realtime_quote",
|
||||
"arguments": {"stock_code": "600519"},
|
||||
"success": true,
|
||||
"duration": 0.5,
|
||||
"result_length": 100,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 2,
|
||||
"tool": "get_realtime_quote",
|
||||
"arguments": {"stock_code": "600519"},
|
||||
"success": true,
|
||||
"duration": 0.6,
|
||||
"result_length": 110,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 3,
|
||||
"tool": "get_daily_history",
|
||||
"arguments": {"stock_code": "600519", "days": 30},
|
||||
"success": false,
|
||||
"duration": 1.5,
|
||||
"result_length": 0,
|
||||
"cached": true
|
||||
},
|
||||
{
|
||||
"step": 4,
|
||||
"tool": "search_stock_news",
|
||||
"arguments": {"query": "贵州茅台"},
|
||||
"success": true,
|
||||
"duration": 2.0,
|
||||
"result_length": 300,
|
||||
"cached": false
|
||||
}
|
||||
]
|
||||
}
|
||||
41
tests/fixtures/agent_trajectory/positive_600519_technical.json
vendored
Normal file
41
tests/fixtures/agent_trajectory/positive_600519_technical.json
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"total_steps": 5,
|
||||
"tool_calls_log": [
|
||||
{
|
||||
"step": 1,
|
||||
"tool": "get_realtime_quote",
|
||||
"arguments": {"stock_code": "600519"},
|
||||
"success": true,
|
||||
"duration": 0.5,
|
||||
"result_length": 100,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 2,
|
||||
"tool": "get_daily_history",
|
||||
"arguments": {"stock_code": "600519", "days": 30},
|
||||
"success": true,
|
||||
"duration": 0.8,
|
||||
"result_length": 200,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 3,
|
||||
"tool": "analyze_trend",
|
||||
"arguments": {"stock_code": "600519"},
|
||||
"success": true,
|
||||
"duration": 1.2,
|
||||
"result_length": 150,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 4,
|
||||
"tool": "search_stock_news",
|
||||
"arguments": {"query": "贵州茅台"},
|
||||
"success": true,
|
||||
"duration": 2.0,
|
||||
"result_length": 300,
|
||||
"cached": false
|
||||
}
|
||||
]
|
||||
}
|
||||
41
tests/fixtures/agent_trajectory/retry_000001_core_data_strict.json
vendored
Normal file
41
tests/fixtures/agent_trajectory/retry_000001_core_data_strict.json
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"total_steps": 5,
|
||||
"tool_calls_log": [
|
||||
{
|
||||
"step": 1,
|
||||
"tool": "get_realtime_quote",
|
||||
"arguments": {"stock_code": "000001"},
|
||||
"success": true,
|
||||
"duration": 0.5,
|
||||
"result_length": 100,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 2,
|
||||
"tool": "get_daily_history",
|
||||
"arguments": {"stock_code": "000001", "days": 60},
|
||||
"success": false,
|
||||
"duration": 1.2,
|
||||
"result_length": 0,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 3,
|
||||
"tool": "get_daily_history",
|
||||
"arguments": {"stock_code": "000001", "days": 60},
|
||||
"success": true,
|
||||
"duration": 1.3,
|
||||
"result_length": 250,
|
||||
"cached": false
|
||||
},
|
||||
{
|
||||
"step": 4,
|
||||
"tool": "get_daily_history",
|
||||
"arguments": {"stock_code": "000001", "days": 60},
|
||||
"success": true,
|
||||
"duration": 0.4,
|
||||
"result_length": 250,
|
||||
"cached": false
|
||||
}
|
||||
]
|
||||
}
|
||||
716
tests/test_agent_trajectory_metrics.py
Normal file
716
tests/test_agent_trajectory_metrics.py
Normal file
@@ -0,0 +1,716 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Unit tests for the agent trajectory evaluation metrics (Issue #1956).
|
||||
|
||||
All trajectories are synthetic ``tool_calls_log`` lists — no LLM, no network.
|
||||
The golden-samples-file tests verify that the checked-in ``golden_samples.json``
|
||||
stays structurally valid and that every ``expected_tools`` entry exists in the
|
||||
repository's real tool registry (imported lazily so the metrics-only sections
|
||||
run even without ``src/`` importable).
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from evals.agent_trajectory.metrics import (
|
||||
GoldenSample,
|
||||
TrajectoryMetrics,
|
||||
_args_key,
|
||||
compute_trajectory_metrics,
|
||||
format_text_report,
|
||||
load_golden_samples,
|
||||
validate_golden_sample,
|
||||
)
|
||||
|
||||
|
||||
def _entry(tool="get_realtime_quote", arguments=None, step=1, success=True, **extra):
|
||||
"""Build a runner-shaped ``tool_calls_log`` entry with sensible defaults."""
|
||||
entry = {
|
||||
"step": step,
|
||||
"tool": tool,
|
||||
"arguments": arguments if arguments is not None else {"stock_code": "600519"},
|
||||
"success": success,
|
||||
"duration": 0.5,
|
||||
"result_length": 100,
|
||||
"cached": False,
|
||||
}
|
||||
entry.update(extra)
|
||||
return entry
|
||||
|
||||
|
||||
def _golden(**overrides):
|
||||
values = dict(
|
||||
id="600519_technical",
|
||||
task_description="分析贵州茅台近期技术面走势",
|
||||
stock_code="600519",
|
||||
expected_tools=["get_realtime_quote", "get_daily_history", "analyze_trend"],
|
||||
allowed_max_steps=8,
|
||||
allow_optional_tools=True,
|
||||
)
|
||||
values.update(overrides)
|
||||
return GoldenSample(**values)
|
||||
|
||||
|
||||
def _metrics(**overrides):
|
||||
values = dict(
|
||||
expected_hit_rate=2 / 3,
|
||||
expected_total=3,
|
||||
missing_expected=["analyze_trend"],
|
||||
optional_tools_used=[],
|
||||
redundant_calls=0,
|
||||
cached_calls=0,
|
||||
failed_calls=0,
|
||||
retries=0,
|
||||
distinct_steps=3,
|
||||
max_steps_touched=False,
|
||||
violations=[],
|
||||
)
|
||||
values.update(overrides)
|
||||
return TrajectoryMetrics(**values)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Args-key stability
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestArgsKey:
|
||||
def test_key_stable_across_key_order_and_nested_unhashable(self):
|
||||
a = {"b": [1, 2], "a": {"x": {"y": 1}}}
|
||||
b = {"a": {"x": {"y": 1}}, "b": [1, 2]}
|
||||
assert _args_key(a) == _args_key(b)
|
||||
|
||||
def test_key_differs_for_different_arguments(self):
|
||||
assert _args_key({"stock_code": "600519"}) != _args_key({"stock_code": "000001"})
|
||||
|
||||
def test_none_arguments_use_empty_object(self):
|
||||
assert _args_key(None) == json.dumps({})
|
||||
|
||||
def test_non_dict_arguments_serialized(self):
|
||||
assert _args_key(["600519"]) == '["600519"]'
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Hit rate / expected & optional tools
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestComputeMetricsHitRate:
|
||||
@staticmethod
|
||||
def _two_of_three_log():
|
||||
return [
|
||||
_entry(tool="get_realtime_quote", step=1),
|
||||
_entry(tool="get_daily_history", step=2),
|
||||
_entry(tool="search_stock_news", step=3),
|
||||
]
|
||||
|
||||
def test_hit_rate_missing_and_optional(self):
|
||||
m = compute_trajectory_metrics(self._two_of_three_log(), _golden())
|
||||
assert m.expected_hit_rate == pytest.approx(2 / 3)
|
||||
assert m.missing_expected == ["analyze_trend"]
|
||||
assert m.optional_tools_used == ["search_stock_news"]
|
||||
assert m.violations == []
|
||||
|
||||
def test_optional_tools_not_allowed_produces_violation(self):
|
||||
m = compute_trajectory_metrics(self._two_of_three_log(), _golden(allow_optional_tools=False))
|
||||
assert m.violations == ["optional tools used but not allowed: search_stock_news"]
|
||||
assert m.expected_hit_rate == pytest.approx(2 / 3)
|
||||
|
||||
def test_duplicate_tool_calls_still_full_hit(self):
|
||||
log = [
|
||||
_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"}, step=1),
|
||||
_entry(tool="get_realtime_quote", arguments={"stock_code": "000001"}, step=2),
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote"]))
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert m.missing_expected == []
|
||||
|
||||
def test_empty_log_yields_zero_metrics(self):
|
||||
m = compute_trajectory_metrics([], _golden())
|
||||
assert m.expected_hit_rate == 0.0
|
||||
assert m.expected_total == 3
|
||||
assert m.missing_expected == ["get_realtime_quote", "get_daily_history", "analyze_trend"]
|
||||
assert m.redundant_calls == 0 and m.cached_calls == 0 and m.failed_calls == 0
|
||||
assert m.retries == 0 and m.distinct_steps == 0
|
||||
assert m.max_steps_touched is False
|
||||
assert m.violations == []
|
||||
|
||||
def test_non_dict_entries_ignored(self):
|
||||
log = ["garbage", None, _entry(tool="get_realtime_quote", step=1)]
|
||||
m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote"]))
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert m.distinct_steps == 1
|
||||
|
||||
def test_empty_expected_tools_yields_zero_hit_and_violation(self):
|
||||
m = compute_trajectory_metrics([_entry()], _golden(expected_tools=[]))
|
||||
assert m.expected_hit_rate == 0.0
|
||||
assert "expected_tools must be a non-empty list" in m.violations
|
||||
|
||||
def test_string_expected_tools_scored_as_empty_not_as_characters(self):
|
||||
m = compute_trajectory_metrics([_entry()], _golden(expected_tools="get_realtime_quote"))
|
||||
assert m.expected_hit_rate == 0.0
|
||||
assert m.expected_total == 0
|
||||
assert "expected_tools must be a list of tool names" in m.violations
|
||||
|
||||
def test_non_bool_allow_optional_tools_scores_strictly(self):
|
||||
log = [
|
||||
_entry(tool="get_realtime_quote", step=1),
|
||||
_entry(tool="search_stock_news", step=2),
|
||||
]
|
||||
m = compute_trajectory_metrics(
|
||||
log,
|
||||
_golden(expected_tools=["get_realtime_quote"], allow_optional_tools="false"),
|
||||
)
|
||||
assert "allow_optional_tools must be a boolean" in m.violations
|
||||
assert "optional tools used but not allowed: search_stock_news" in m.violations
|
||||
|
||||
def test_duplicate_expected_tools_scored_as_unique_names(self):
|
||||
# Regression: ["quote", "quote", "history"] with a single quote call
|
||||
# must read 1/2, not 2/3.
|
||||
golden = _golden(
|
||||
expected_tools=[
|
||||
"get_realtime_quote",
|
||||
"get_realtime_quote",
|
||||
"get_daily_history",
|
||||
],
|
||||
)
|
||||
m = compute_trajectory_metrics([_entry(tool="get_realtime_quote")], golden)
|
||||
assert m.expected_total == 2
|
||||
assert m.expected_hit_rate == pytest.approx(0.5)
|
||||
assert m.missing_expected == ["get_daily_history"]
|
||||
assert "expected_tools must not contain duplicate names" in m.violations
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2b. Direct-construction path: same structure contract as the loader
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestDirectConstructionContract:
|
||||
def test_malformed_expected_tools_elements_are_reported(self):
|
||||
# Review counter-example: ['get_realtime_quote', ''] must not
|
||||
# silently collapse into a one-tool golden — the malformed element
|
||||
# is reported, only valid names take part in scoring. Whitespace-
|
||||
# only elements follow the validator's predicate (t.strip()).
|
||||
log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})]
|
||||
for malformed in (["get_realtime_quote", ""], ["get_realtime_quote", 1]):
|
||||
m = compute_trajectory_metrics(log, _golden(expected_tools=malformed))
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert "expected_tools must contain only non-empty strings" in m.violations
|
||||
|
||||
def test_whitespace_only_expected_tool_does_not_pollute_scoring(self):
|
||||
# Review counter-example: ' ' must not enter the hit-rate
|
||||
# denominator nor show up as a missing tool — it is malformed.
|
||||
log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})]
|
||||
m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote", " "]))
|
||||
assert m.expected_total == 1
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert m.missing_expected == []
|
||||
assert "expected_tools must contain only non-empty strings" in m.violations
|
||||
|
||||
def test_non_positive_allowed_max_steps_reported_in_direct_score(self):
|
||||
# Review counter-example: allowed_max_steps=0 / -3 must report the
|
||||
# validator's wording instead of silently disabling the budget
|
||||
# assertion, no matter how many steps the trajectory takes.
|
||||
log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"}, step=s) for s in range(1, 100)]
|
||||
for limit in (0, -3):
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=limit))
|
||||
assert "allowed_max_steps must be >= 1" in m.violations
|
||||
assert m.max_steps_touched is False
|
||||
|
||||
def test_direct_score_mirrors_validator_structure_contract(self):
|
||||
# The owner-requested parity: for each malformed golden shape the
|
||||
# validator rejects, the direct-score path must surface the same
|
||||
# issue wording in its violations.
|
||||
log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})]
|
||||
cases = [
|
||||
(dict(expected_tools=["get_realtime_quote", " "]), "expected_tools must contain only non-empty strings"),
|
||||
(dict(expected_tools=["get_realtime_quote", ""]), "expected_tools must contain only non-empty strings"),
|
||||
(dict(allowed_max_steps=0), "allowed_max_steps must be >= 1"),
|
||||
(dict(allowed_max_steps=-3), "allowed_max_steps must be >= 1"),
|
||||
(dict(allow_optional_tools="false"), "allow_optional_tools must be a boolean"),
|
||||
(dict(id=[]), "id must be a non-empty string"),
|
||||
(dict(task_description=None), "task_description must be a non-empty string"),
|
||||
(dict(stock_code=None), "stock_code must be a string"),
|
||||
(dict(stock_code=" "), "stock_code must not contain leading or trailing whitespace"),
|
||||
]
|
||||
for overrides, issue in cases:
|
||||
golden = _golden(**overrides)
|
||||
validator_issues = validate_golden_sample(golden)
|
||||
assert any(issue in i for i in validator_issues), (issue, overrides, validator_issues)
|
||||
m = compute_trajectory_metrics(log, golden)
|
||||
assert any(issue in v for v in m.violations), (issue, overrides, m.violations)
|
||||
|
||||
def test_every_validator_issue_surfaces_in_direct_score(self):
|
||||
# Structural lock: the direct-score path surfaces the validator's
|
||||
# COMPLETE issue list verbatim (this is how id / task_description /
|
||||
# stock_code and any future validator check apply to direct scoring
|
||||
# automatically). A thoroughly malformed sample must yield every
|
||||
# validator issue in compute violations, with no duplicates from the
|
||||
# inline scoring checks.
|
||||
log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})]
|
||||
golden = GoldenSample(
|
||||
id=[],
|
||||
task_description=None,
|
||||
stock_code=" ",
|
||||
expected_tools=["", " "],
|
||||
allowed_max_steps=0,
|
||||
allow_optional_tools="false",
|
||||
)
|
||||
validator_issues = validate_golden_sample(golden)
|
||||
assert validator_issues, "the sample is thoroughly malformed"
|
||||
m = compute_trajectory_metrics(log, golden)
|
||||
for issue in validator_issues:
|
||||
assert any(issue in v for v in m.violations), (issue, m.violations)
|
||||
assert m.violations.count(issue) == 1, (issue, m.violations)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Retries, caching, failure counting
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestRetryAndCaching:
|
||||
def test_fail_then_retry_success_counts_one_retry(self):
|
||||
log = [
|
||||
_entry(step=1, success=False),
|
||||
_entry(step=2, success=True),
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.retries == 1
|
||||
assert m.redundant_calls == 1
|
||||
assert m.failed_calls == 1
|
||||
|
||||
def test_same_tool_different_args_not_redundant(self):
|
||||
log = [
|
||||
_entry(arguments={"stock_code": "600519"}, step=1),
|
||||
_entry(arguments={"stock_code": "000001"}, step=2),
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.redundant_calls == 0
|
||||
assert m.retries == 0
|
||||
|
||||
def test_repeat_after_success_is_redundant_but_not_retry(self):
|
||||
log = [_entry(step=1, success=True), _entry(step=2, success=True)]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.redundant_calls == 1
|
||||
assert m.retries == 0
|
||||
assert m.failed_calls == 0
|
||||
|
||||
def test_fail_fail_success_counts_two_retries(self):
|
||||
log = [
|
||||
_entry(step=1, success=False),
|
||||
_entry(step=2, success=False),
|
||||
_entry(step=3, success=True),
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.retries == 2
|
||||
assert m.redundant_calls == 2
|
||||
assert m.failed_calls == 2
|
||||
|
||||
def test_recovery_clears_failure_state(self):
|
||||
# fail -> success -> success: only the recovery attempt is a retry;
|
||||
# the repeat after success counts as redundant only.
|
||||
log = [
|
||||
_entry(step=1, success=False),
|
||||
_entry(step=2, success=True),
|
||||
_entry(step=3, success=True),
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.retries == 1
|
||||
assert m.redundant_calls == 2
|
||||
assert m.failed_calls == 1
|
||||
|
||||
def test_cached_entry_counted(self):
|
||||
m = compute_trajectory_metrics([_entry(cached=True, success=False)], _golden())
|
||||
assert m.cached_calls == 1
|
||||
assert m.failed_calls == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. max_steps touching
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestMaxStepsTouched:
|
||||
def test_steps_reaching_allowed_max_touched(self):
|
||||
log = [_entry(step=i) for i in range(1, 6)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5))
|
||||
assert m.max_steps_touched is True
|
||||
assert "trajectory reached allowed_max_steps (5)" in m.violations
|
||||
|
||||
def test_steps_below_limit_not_touched(self):
|
||||
log = [_entry(step=i) for i in range(1, 5)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5))
|
||||
assert m.max_steps_touched is False
|
||||
assert m.violations == []
|
||||
|
||||
def test_empty_log_not_touched(self):
|
||||
m = compute_trajectory_metrics([], _golden(allowed_max_steps=5))
|
||||
assert m.max_steps_touched is False
|
||||
|
||||
def test_non_integer_limit_surfaces_violation_without_crash(self):
|
||||
log = [_entry(step=i) for i in range(1, 6)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps="5"))
|
||||
assert m.max_steps_touched is False
|
||||
assert "allowed_max_steps must be an integer" in m.violations
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4b. total_steps input (final answer round)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestTotalStepsInput:
|
||||
def test_total_steps_extends_step_metrics_beyond_log(self):
|
||||
log = [_entry(step=1), _entry(step=2)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5), total_steps=4)
|
||||
assert m.distinct_steps == 4
|
||||
assert m.max_steps_touched is False
|
||||
|
||||
def test_total_steps_can_touch_the_limit(self):
|
||||
log = [_entry(step=1), _entry(step=2)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=3), total_steps=3)
|
||||
assert m.max_steps_touched is True
|
||||
assert m.distinct_steps == 3
|
||||
|
||||
def test_log_wins_when_it_reaches_further(self):
|
||||
log = [_entry(step=i) for i in range(1, 5)]
|
||||
m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5), total_steps=2)
|
||||
assert m.distinct_steps == 4
|
||||
assert m.max_steps_touched is False
|
||||
|
||||
def test_none_keeps_log_only_behaviour(self):
|
||||
m = compute_trajectory_metrics([_entry(step=1)], _golden())
|
||||
assert m.distinct_steps == 1
|
||||
|
||||
def test_non_numeric_total_steps_ignored(self):
|
||||
m = compute_trajectory_metrics([_entry(step=1)], _golden(), total_steps="not-a-number")
|
||||
assert m.distinct_steps == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. Tolerant entry shapes (missing fields)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestTolerantEntries:
|
||||
def test_missing_optional_fields_defaulted(self):
|
||||
m = compute_trajectory_metrics([{"step": 1, "tool": "get_realtime_quote"}], _golden())
|
||||
assert m.failed_calls == 0
|
||||
assert m.cached_calls == 0
|
||||
assert m.expected_hit_rate == pytest.approx(1 / 3)
|
||||
|
||||
def test_entries_without_arguments_share_one_identity(self):
|
||||
# Arguments-less entries key as an empty object: a repeated call of
|
||||
# the same tool is redundant even without an arguments payload.
|
||||
log = [
|
||||
{"step": 1, "tool": "get_realtime_quote", "success": True},
|
||||
{"step": 2, "tool": "get_realtime_quote", "success": True},
|
||||
]
|
||||
m = compute_trajectory_metrics(log, _golden())
|
||||
assert m.redundant_calls == 1
|
||||
assert m.retries == 0
|
||||
|
||||
def test_invalid_step_coerced_to_zero(self):
|
||||
m = compute_trajectory_metrics([_entry(step="not-a-number")], _golden())
|
||||
assert m.distinct_steps == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. Text report rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestFormatTextReport:
|
||||
def test_contains_hit_fraction_and_percent(self):
|
||||
text = format_text_report(_metrics())
|
||||
assert "2/3" in text
|
||||
assert "66.7%" in text
|
||||
|
||||
def test_empties_render_placeholder(self):
|
||||
m = _metrics(
|
||||
expected_hit_rate=0.0,
|
||||
expected_total=3,
|
||||
missing_expected=["get_realtime_quote", "get_daily_history", "analyze_trend"],
|
||||
)
|
||||
text = format_text_report(m)
|
||||
assert "缺失期望工具: get_realtime_quote, get_daily_history, analyze_trend" in text
|
||||
assert "期望外工具: 无" in text
|
||||
assert "违规项: 无" in text
|
||||
assert "触碰 max_steps: 否" in text
|
||||
|
||||
def test_violations_rendered(self):
|
||||
text = format_text_report(_metrics(violations=["trajectory reached allowed_max_steps (5)"]))
|
||||
assert "违规项: trajectory reached allowed_max_steps (5)" in text
|
||||
|
||||
def test_deterministic(self):
|
||||
m = _metrics(redundant_calls=2, retries=1, max_steps_touched=True)
|
||||
assert format_text_report(m) == format_text_report(m)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. Golden samples file (schema + registry membership)
|
||||
# ---------------------------------------------------------------------------
|
||||
def _repo_tool_names():
|
||||
"""Authoritative tool names from the five tool modules (lazy import)."""
|
||||
from src.agent.tools.analysis_tools import ALL_ANALYSIS_TOOLS
|
||||
from src.agent.tools.backtest_tools import ALL_BACKTEST_TOOLS
|
||||
from src.agent.tools.data_tools import ALL_DATA_TOOLS
|
||||
from src.agent.tools.market_tools import ALL_MARKET_TOOLS
|
||||
from src.agent.tools.search_tools import ALL_SEARCH_TOOLS
|
||||
|
||||
all_tools = ALL_DATA_TOOLS + ALL_ANALYSIS_TOOLS + ALL_SEARCH_TOOLS + ALL_MARKET_TOOLS + ALL_BACKTEST_TOOLS
|
||||
return {tool_def.name for tool_def in all_tools}
|
||||
|
||||
|
||||
class TestGoldenSamplesFile:
|
||||
def test_samples_load_clean_with_registry_names(self):
|
||||
samples = load_golden_samples(known_tool_names=_repo_tool_names())
|
||||
assert len(samples) == 2
|
||||
assert [s.id for s in samples] == ["600519_technical", "000001_core_data_strict"]
|
||||
|
||||
def test_each_sample_passes_structural_validation(self):
|
||||
for sample in load_golden_samples():
|
||||
assert validate_golden_sample(sample, _repo_tool_names()) == []
|
||||
|
||||
def test_expected_tools_exist_in_repo_registry(self):
|
||||
known = _repo_tool_names()
|
||||
for sample in load_golden_samples():
|
||||
unknown = [t for t in sample.expected_tools if t not in known]
|
||||
assert unknown == [], f"sample '{sample.id}' expects unknown tools: {unknown}"
|
||||
|
||||
def test_contains_strict_sample_without_optional_tools(self):
|
||||
samples = load_golden_samples()
|
||||
assert any(not s.allow_optional_tools for s in samples)
|
||||
|
||||
def test_samples_have_required_text_fields(self):
|
||||
for sample in load_golden_samples():
|
||||
assert sample.id.strip()
|
||||
assert sample.task_description.strip()
|
||||
assert sample.stock_code.strip()
|
||||
assert sample.allowed_max_steps >= 1
|
||||
|
||||
def test_empty_stock_code_is_the_valid_no_context_default(self):
|
||||
sample = GoldenSample(id="x", task_description="t", stock_code="", expected_tools=["get_realtime_quote"])
|
||||
assert validate_golden_sample(sample) == []
|
||||
|
||||
def test_whitespace_only_stock_code_fails_validation(self):
|
||||
sample = GoldenSample(id="x", task_description="t", stock_code=" ", expected_tools=["get_realtime_quote"])
|
||||
issues = validate_golden_sample(sample)
|
||||
assert any("stock_code must not contain leading or trailing whitespace" in i for i in issues)
|
||||
|
||||
def test_non_string_stock_code_fails_validation(self):
|
||||
sample = GoldenSample(id="x", task_description="t", stock_code=None, expected_tools=["get_realtime_quote"])
|
||||
issues = validate_golden_sample(sample)
|
||||
assert any("stock_code must be a string" in i for i in issues)
|
||||
|
||||
def test_string_expected_tools_fails_validation(self):
|
||||
sample = GoldenSample(
|
||||
id="x",
|
||||
task_description="t",
|
||||
stock_code="600519",
|
||||
expected_tools="get_realtime_quote",
|
||||
)
|
||||
issues = validate_golden_sample(sample)
|
||||
assert any("must be a list" in i for i in issues)
|
||||
|
||||
def test_non_bool_allow_optional_tools_fails_validation(self):
|
||||
sample = GoldenSample(
|
||||
id="x",
|
||||
task_description="t",
|
||||
stock_code="600519",
|
||||
expected_tools=["get_realtime_quote"],
|
||||
allow_optional_tools="false",
|
||||
)
|
||||
issues = validate_golden_sample(sample)
|
||||
assert any("allow_optional_tools must be a boolean" in i for i in issues)
|
||||
|
||||
def test_mistyped_fields_fail_validation_not_crash(self):
|
||||
# Same defect class as the string expected_tools bug: hand-edited JSON
|
||||
# with mistyped fields must be rejected cleanly, never crash or pass.
|
||||
bad = GoldenSample(
|
||||
id=5,
|
||||
task_description=None,
|
||||
stock_code="600519",
|
||||
expected_tools=["get_realtime_quote"],
|
||||
allowed_max_steps="5",
|
||||
allow_optional_tools=1,
|
||||
)
|
||||
issues = validate_golden_sample(bad)
|
||||
assert any("id must be a non-empty string" in i for i in issues)
|
||||
assert any("task_description must be a non-empty string" in i for i in issues)
|
||||
assert any("allowed_max_steps must be an integer" in i for i in issues)
|
||||
assert any("allow_optional_tools must be a boolean" in i for i in issues)
|
||||
|
||||
def test_non_iterable_expected_tools_with_known_names_does_not_crash(self):
|
||||
sample = GoldenSample(
|
||||
id="x",
|
||||
task_description="t",
|
||||
stock_code="600519",
|
||||
expected_tools=1,
|
||||
)
|
||||
issues = validate_golden_sample(sample, {"get_realtime_quote"})
|
||||
assert any("expected_tools must be a list" in i for i in issues)
|
||||
|
||||
def test_registry_membership_with_one_shot_generator(self):
|
||||
# The helper accepts any Iterable[str]; a one-shot generator must be
|
||||
# materialized internally so membership checks never consume it.
|
||||
sample = GoldenSample(
|
||||
id="x",
|
||||
task_description="t",
|
||||
stock_code="600519",
|
||||
expected_tools=["b", "a"],
|
||||
)
|
||||
known = (name for name in ["a", "b"])
|
||||
assert validate_golden_sample(sample, known) == []
|
||||
|
||||
def test_duplicate_expected_tools_fail_validation(self):
|
||||
sample = GoldenSample(
|
||||
id="x",
|
||||
task_description="t",
|
||||
stock_code="600519",
|
||||
expected_tools=["get_realtime_quote", "get_realtime_quote"],
|
||||
)
|
||||
issues = validate_golden_sample(sample)
|
||||
assert any("must not contain duplicate names" in i for i in issues)
|
||||
|
||||
def test_loader_materializes_registry_once_for_multiple_samples(self):
|
||||
# A one-shot generator must survive loading the whole checked-in file:
|
||||
# the first sample must not exhaust it for the remaining samples.
|
||||
samples = load_golden_samples(known_tool_names=(name for name in _repo_tool_names()))
|
||||
assert len(samples) == 2
|
||||
|
||||
|
||||
class TestLoadGoldenSamplesErrors:
|
||||
@staticmethod
|
||||
def _write_sample(tmp_path, payload, name="golden.json"):
|
||||
target = tmp_path / name
|
||||
target.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
|
||||
return str(target)
|
||||
|
||||
def test_missing_file_raises(self):
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_golden_samples(path="no/such/golden_samples.json")
|
||||
|
||||
def test_non_list_root_raises(self, tmp_path):
|
||||
path = self._write_sample(tmp_path, {"id": "x"})
|
||||
with pytest.raises(ValueError, match="JSON list"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_malformed_json_raises(self, tmp_path):
|
||||
target = tmp_path / "golden.json"
|
||||
target.write_text("{not json", encoding="utf-8")
|
||||
with pytest.raises(ValueError):
|
||||
load_golden_samples(path=str(target))
|
||||
|
||||
def test_duplicate_ids_raise(self, tmp_path):
|
||||
sample = {
|
||||
"id": "dup",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample, sample])
|
||||
with pytest.raises(ValueError, match="duplicate sample id: dup"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_unknown_expected_tool_flagged_when_names_given(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["not_a_real_tool"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="unknown expected_tools: not_a_real_tool"):
|
||||
load_golden_samples(path=path, known_tool_names=_repo_tool_names())
|
||||
|
||||
def test_unknown_expected_tool_allowed_without_names(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["not_a_real_tool"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
assert [s.id for s in load_golden_samples(path=path)] == ["x"]
|
||||
|
||||
def test_extra_json_keys_ignored(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
"notes": "forward-looking metadata",
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
assert load_golden_samples(path=path)[0].id == "x"
|
||||
|
||||
def test_invalid_max_steps_flagged(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
"allowed_max_steps": 0,
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="allowed_max_steps must be >= 1"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_whitespace_only_stock_code_raises(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": " ",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="stock_code must not contain leading or trailing whitespace"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_string_expected_tools_raises(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": "get_realtime_quote",
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="expected_tools must be a list"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_non_list_expected_tools_with_known_names_raises_valueerror(self, tmp_path):
|
||||
# Regression for OR-COR-4e0e3cf1: the registry membership check must
|
||||
# not iterate a rejected non-list value and leak a TypeError.
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": 1,
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="expected_tools must be a list"):
|
||||
load_golden_samples(path=path, known_tool_names={"get_realtime_quote"})
|
||||
|
||||
def test_unhashable_id_raises_valueerror(self, tmp_path):
|
||||
# Structural validation must run before duplicate detection: an
|
||||
# unhashable id would otherwise crash the seen_ids membership check
|
||||
# with a TypeError instead of the documented ValueError.
|
||||
sample = {
|
||||
"id": ["x"],
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="id must be a non-empty string"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_non_bool_allow_optional_tools_raises(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
"allow_optional_tools": "false",
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="allow_optional_tools must be a boolean"):
|
||||
load_golden_samples(path=path)
|
||||
|
||||
def test_duplicate_expected_tools_raise(self, tmp_path):
|
||||
sample = {
|
||||
"id": "x",
|
||||
"task_description": "t",
|
||||
"stock_code": "600519",
|
||||
"expected_tools": ["get_realtime_quote", "get_realtime_quote"],
|
||||
}
|
||||
path = self._write_sample(tmp_path, [sample])
|
||||
with pytest.raises(ValueError, match="expected_tools must not contain duplicate names"):
|
||||
load_golden_samples(path=path)
|
||||
436
tests/test_agent_trajectory_run_eval.py
Normal file
436
tests/test_agent_trajectory_run_eval.py
Normal file
@@ -0,0 +1,436 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Unit tests for the runnable agent trajectory eval entry (Issue #1956).
|
||||
|
||||
All tests are offline: ``run_eval._build_executor`` is monkeypatched with a
|
||||
duck-typed stub, and the three checked-in fixtures provide real-shaped
|
||||
``tool_calls_log`` payloads covering the positive path, the negative path
|
||||
(missing expected tool + cached failure + max_steps) and the retry path.
|
||||
The two runtime-seam guards (multi-arch rejection and golden validation
|
||||
against the real tool registry) are exercised via monkeypatched config /
|
||||
registry access plus one lazy real-registry check.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from evals.agent_trajectory import run_eval
|
||||
from evals.agent_trajectory.metrics import GoldenSample, load_golden_samples
|
||||
|
||||
FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures" / "agent_trajectory"
|
||||
|
||||
|
||||
def _stub_executor(log, total_steps=None):
|
||||
"""Duck-typed executor recording its run calls (no src/ import)."""
|
||||
|
||||
class _Stub:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def run(self, task, context=None):
|
||||
self.calls.append((task, context))
|
||||
return SimpleNamespace(tool_calls_log=list(log), total_steps=total_steps)
|
||||
|
||||
return _Stub()
|
||||
|
||||
|
||||
def _fixture(name):
|
||||
"""Load a checked-in fixture payload: (tool_calls_log, total_steps)."""
|
||||
payload = json.loads((FIXTURES_DIR / f"{name}.json").read_text(encoding="utf-8"))
|
||||
return payload["tool_calls_log"], payload["total_steps"]
|
||||
|
||||
|
||||
def _result_executor(result):
|
||||
"""Duck-typed executor whose run always returns a fixed result object."""
|
||||
return SimpleNamespace(run=lambda task, context=None: result)
|
||||
|
||||
|
||||
def _goldens():
|
||||
return {sample.id: sample for sample in load_golden_samples()}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. run_sample: positive path
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestRunSamplePositive:
|
||||
def test_full_hit_and_clean_metrics(self):
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
executor = _stub_executor(log, total_steps)
|
||||
m = run_eval.run_sample(executor, _goldens()["600519_technical"])
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert m.missing_expected == []
|
||||
assert m.optional_tools_used == ["search_stock_news"]
|
||||
assert m.distinct_steps == 5
|
||||
assert m.violations == []
|
||||
|
||||
def test_json_report_contract(self, tmp_path):
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
out = tmp_path / "report.json"
|
||||
m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"], json_out=out)
|
||||
report = json.loads(out.read_text(encoding="utf-8"))
|
||||
assert report["sample_id"] == "600519_technical"
|
||||
assert report["task_description"]
|
||||
assert report["stock_code"] == "600519"
|
||||
assert report["violations"] == m.violations
|
||||
metrics = report["metrics"]
|
||||
expected_fields = {
|
||||
"expected_hit_rate",
|
||||
"expected_total",
|
||||
"missing_expected",
|
||||
"optional_tools_used",
|
||||
"redundant_calls",
|
||||
"cached_calls",
|
||||
"failed_calls",
|
||||
"retries",
|
||||
"distinct_steps",
|
||||
"max_steps_touched",
|
||||
"violations",
|
||||
}
|
||||
assert set(metrics) == expected_fields
|
||||
assert metrics["expected_hit_rate"] == 1.0
|
||||
|
||||
def test_text_summary_printed(self, capsys):
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"])
|
||||
out = capsys.readouterr().out
|
||||
assert "Agent Trajectory 评估报告" in out
|
||||
assert "3/3" in out
|
||||
assert "违规项: 无" in out
|
||||
|
||||
def test_context_carries_stock_code(self):
|
||||
executor = _stub_executor([], total_steps=1)
|
||||
run_eval.run_sample(executor, _goldens()["600519_technical"])
|
||||
assert len(executor.calls) == 1
|
||||
task, context = executor.calls[0]
|
||||
assert context == {"stock_code": "600519"}
|
||||
assert "600519" in task
|
||||
|
||||
def test_empty_stock_code_passes_no_context(self):
|
||||
executor = _stub_executor([], total_steps=1)
|
||||
sample = GoldenSample(
|
||||
id="no_context",
|
||||
task_description="t",
|
||||
stock_code="",
|
||||
expected_tools=["get_realtime_quote"],
|
||||
)
|
||||
run_eval.run_sample(executor, sample)
|
||||
_, context = executor.calls[0]
|
||||
assert context is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. run_sample: negative path (violations are findings, not exceptions)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestRunSampleNegative:
|
||||
def test_missing_tool_and_max_steps_violations(self):
|
||||
log, total_steps = _fixture("negative_600519_technical")
|
||||
m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"])
|
||||
assert m.expected_hit_rate == pytest.approx(2 / 3)
|
||||
assert m.missing_expected == ["analyze_trend"]
|
||||
assert m.redundant_calls == 1
|
||||
assert m.cached_calls == 1
|
||||
assert m.failed_calls == 1
|
||||
assert m.max_steps_touched is True
|
||||
assert "trajectory reached allowed_max_steps (8)" in m.violations
|
||||
|
||||
def test_violations_do_not_raise(self):
|
||||
log, total_steps = _fixture("negative_600519_technical")
|
||||
m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"])
|
||||
assert m.violations # the eval is a reporter, not a gate
|
||||
|
||||
def test_executor_without_total_steps_still_scores(self):
|
||||
# A result without total_steps keeps the log-only step behaviour.
|
||||
log, _ = _fixture("negative_600519_technical")
|
||||
m = run_eval.run_sample(_stub_executor(log, None), _goldens()["600519_technical"])
|
||||
assert m.distinct_steps == 4
|
||||
assert m.max_steps_touched is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. run_sample: retry path (fail -> success -> success contract)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestRunSampleRetry:
|
||||
def test_fail_success_success_counts_one_retry(self):
|
||||
log, total_steps = _fixture("retry_000001_core_data_strict")
|
||||
m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["000001_core_data_strict"])
|
||||
assert m.expected_hit_rate == 1.0
|
||||
assert m.retries == 1
|
||||
assert m.redundant_calls == 2
|
||||
assert m.failed_calls == 1
|
||||
assert m.cached_calls == 0
|
||||
assert m.optional_tools_used == []
|
||||
assert m.violations == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3.5 run failure: results carrying an explicit success=False
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestRunFailure:
|
||||
def test_unsuccessful_result_raises_before_scoring(self, capsys):
|
||||
# Owner repro: a real AgentResult with success=False must not be scored.
|
||||
executor = _result_executor(SimpleNamespace(success=False, error="boom", tool_calls_log=[], total_steps=0))
|
||||
with pytest.raises(RuntimeError, match="boom"):
|
||||
run_eval.run_sample(executor, _goldens()["600519_technical"])
|
||||
assert "Agent Trajectory 评估报告" not in capsys.readouterr().out
|
||||
|
||||
def test_unsuccessful_result_without_error_raises(self):
|
||||
executor = _result_executor(SimpleNamespace(success=False, tool_calls_log=[], total_steps=0))
|
||||
with pytest.raises(RuntimeError, match="success=false"):
|
||||
run_eval.run_sample(executor, _goldens()["600519_technical"])
|
||||
|
||||
def test_explicit_success_true_scores_normally(self):
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
result = SimpleNamespace(success=True, tool_calls_log=list(log), total_steps=total_steps)
|
||||
m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"])
|
||||
assert m.expected_hit_rate == 1.0
|
||||
|
||||
def test_missing_success_field_is_tolerated(self):
|
||||
# Duck-typed stubs without a success attribute keep the old behaviour.
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
result = SimpleNamespace(tool_calls_log=list(log), total_steps=total_steps)
|
||||
m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"])
|
||||
assert m.expected_hit_rate == 1.0
|
||||
|
||||
def test_success_none_is_tolerated(self):
|
||||
log, total_steps = _fixture("positive_600519_technical")
|
||||
result = SimpleNamespace(success=None, tool_calls_log=list(log), total_steps=total_steps)
|
||||
m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"])
|
||||
assert m.expected_hit_rate == 1.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. runtime-seam guards: multi-arch rejection + golden registry validation
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestArchGuard:
|
||||
def test_multi_arch_raises(self, monkeypatch):
|
||||
import src.config
|
||||
|
||||
monkeypatch.setattr(src.config, "get_config", lambda: SimpleNamespace(agent_arch="multi"))
|
||||
with pytest.raises(RuntimeError, match="multi"):
|
||||
run_eval._check_agent_arch()
|
||||
|
||||
def test_single_arch_passes(self, monkeypatch):
|
||||
import src.config
|
||||
|
||||
monkeypatch.setattr(src.config, "get_config", lambda: SimpleNamespace(agent_arch="single"))
|
||||
run_eval._check_agent_arch() # no raise
|
||||
|
||||
|
||||
class TestGoldenRegistryValidation:
|
||||
def test_unknown_expected_tools_exit_one(self, tmp_path, monkeypatch, capsys):
|
||||
# A misspelled expected tool must fail as an invalid sample, not score low.
|
||||
custom = [
|
||||
{
|
||||
"id": "typo_sample",
|
||||
"task_description": "t",
|
||||
"stock_code": "",
|
||||
"expected_tools": ["get_daily_histroy"],
|
||||
"allowed_max_steps": 10,
|
||||
"allow_optional_tools": True,
|
||||
}
|
||||
]
|
||||
golden_path = tmp_path / "golden.json"
|
||||
golden_path.write_text(json.dumps(custom, ensure_ascii=False), encoding="utf-8")
|
||||
monkeypatch.setattr(run_eval, "_known_tool_names", lambda: {"get_realtime_quote"})
|
||||
assert run_eval.main(["--sample", "typo_sample", "--golden-path", str(golden_path)]) == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "unknown expected_tools" in err
|
||||
assert "get_daily_histroy" in err
|
||||
|
||||
def test_checked_in_goldens_pass_real_registry(self):
|
||||
known = run_eval._known_tool_names()
|
||||
assert len(known) > 5 # sanity: the real registry is non-trivial
|
||||
load_golden_samples(known_tool_names=known) # must not raise
|
||||
|
||||
def test_registry_load_failure_exit_one(self, monkeypatch, capsys):
|
||||
def _boom():
|
||||
raise ImportError("no tool modules")
|
||||
|
||||
monkeypatch.setattr(run_eval, "_known_tool_names", _boom)
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 1
|
||||
assert "failed to load tool registry" in capsys.readouterr().err
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. CLI (main): exit codes, selection, JSON output
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestMainCli:
|
||||
@pytest.fixture(autouse=True)
|
||||
def _stub_builder(self, monkeypatch):
|
||||
executor = _stub_executor([], total_steps=1)
|
||||
monkeypatch.setattr(run_eval, "_build_executor", lambda: executor)
|
||||
return executor
|
||||
|
||||
def test_exit_zero_successful_sample(self, capsys):
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "[eval] sample: 600519_technical" in out
|
||||
assert "Agent Trajectory 评估报告" in out
|
||||
|
||||
def test_unknown_sample_id_exit_one(self, capsys):
|
||||
assert run_eval.main(["--sample", "no-such-id"]) == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "unknown sample id 'no-such-id'" in err
|
||||
assert "600519_technical" in err and "000001_core_data_strict" in err
|
||||
|
||||
def test_missing_arguments_exit_two(self):
|
||||
with pytest.raises(SystemExit) as excinfo:
|
||||
run_eval.main([])
|
||||
assert excinfo.value.code == 2
|
||||
|
||||
def test_both_flags_exit_two(self):
|
||||
with pytest.raises(SystemExit) as excinfo:
|
||||
run_eval.main(["--sample", "600519_technical", "--all"])
|
||||
assert excinfo.value.code == 2
|
||||
|
||||
def test_build_failure_exit_one(self, monkeypatch, capsys):
|
||||
def _boom():
|
||||
raise RuntimeError("no api key configured")
|
||||
|
||||
monkeypatch.setattr(run_eval, "_build_executor", _boom)
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 1
|
||||
assert "failed to build agent executor" in capsys.readouterr().err
|
||||
|
||||
def test_multi_arch_build_rejection_exit_one(self, monkeypatch, capsys):
|
||||
def _boom():
|
||||
raise RuntimeError("AGENT_ARCH=multi is not supported by this minimal eval")
|
||||
|
||||
monkeypatch.setattr(run_eval, "_build_executor", _boom)
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 1
|
||||
assert "multi" in capsys.readouterr().err
|
||||
|
||||
def test_run_failure_exit_one(self, monkeypatch, capsys):
|
||||
class _Exploding:
|
||||
def run(self, task, context=None):
|
||||
raise RuntimeError("llm timeout")
|
||||
|
||||
monkeypatch.setattr(run_eval, "_build_executor", lambda: _Exploding())
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 1
|
||||
assert "sample '600519_technical' failed" in capsys.readouterr().err
|
||||
|
||||
def test_unsuccessful_result_exit_one(self, monkeypatch, capsys):
|
||||
# Owner repro: a result object with success=False is a run failure.
|
||||
result = SimpleNamespace(success=False, error="boom", tool_calls_log=[], total_steps=0)
|
||||
monkeypatch.setattr(run_eval, "_build_executor", lambda: _result_executor(result))
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "sample '600519_technical' failed" in err
|
||||
assert "boom" in err
|
||||
|
||||
def test_all_returns_one_when_a_sample_result_is_unsuccessful(self, monkeypatch, capsys):
|
||||
result = SimpleNamespace(success=False, tool_calls_log=[], total_steps=0)
|
||||
monkeypatch.setattr(run_eval, "_build_executor", lambda: _result_executor(result))
|
||||
assert run_eval.main(["--all"]) == 1
|
||||
assert "failed" in capsys.readouterr().err
|
||||
|
||||
def test_violations_still_exit_zero(self, monkeypatch, capsys):
|
||||
log, total_steps = _fixture("negative_600519_technical")
|
||||
monkeypatch.setattr(run_eval, "_build_executor", lambda: _stub_executor(log, total_steps))
|
||||
assert run_eval.main(["--sample", "600519_technical"]) == 0
|
||||
assert "trajectory reached allowed_max_steps (8)" in capsys.readouterr().out
|
||||
|
||||
def test_all_writes_keyed_json(self, tmp_path, capsys):
|
||||
out = tmp_path / "all.json"
|
||||
assert run_eval.main(["--all", "--json-out", str(out)]) == 0
|
||||
report = json.loads(out.read_text(encoding="utf-8"))
|
||||
assert set(report) == {"600519_technical", "000001_core_data_strict"}
|
||||
for sample_report in report.values():
|
||||
assert set(sample_report) == {"sample_id", "task_description", "stock_code", "metrics", "violations"}
|
||||
|
||||
def test_single_sample_json_out_is_not_keyed(self, tmp_path):
|
||||
out = tmp_path / "one.json"
|
||||
assert run_eval.main(["--sample", "600519_technical", "--json-out", str(out)]) == 0
|
||||
report = json.loads(out.read_text(encoding="utf-8"))
|
||||
assert report["sample_id"] == "600519_technical"
|
||||
|
||||
def test_utf8_chinese_roundtrip(self, tmp_path):
|
||||
out = tmp_path / "utf8.json"
|
||||
assert run_eval.main(["--sample", "600519_technical", "--json-out", str(out)]) == 0
|
||||
text = out.read_text(encoding="utf-8")
|
||||
assert "贵州茅台" in text
|
||||
json.loads(text) # valid JSON with raw UTF-8 content
|
||||
|
||||
def test_golden_path_override(self, tmp_path, capsys):
|
||||
custom = [
|
||||
{
|
||||
"id": "custom_sample",
|
||||
"task_description": "自定义任务",
|
||||
"stock_code": "",
|
||||
"expected_tools": ["get_realtime_quote"],
|
||||
"allowed_max_steps": 3,
|
||||
"allow_optional_tools": True,
|
||||
}
|
||||
]
|
||||
golden_path = tmp_path / "golden.json"
|
||||
golden_path.write_text(json.dumps(custom, ensure_ascii=False), encoding="utf-8")
|
||||
assert run_eval.main(["--sample", "custom_sample", "--golden-path", str(golden_path)]) == 0
|
||||
assert "[eval] sample: custom_sample" in capsys.readouterr().out
|
||||
|
||||
def test_load_failure_exit_one(self, capsys):
|
||||
assert run_eval.main(["--sample", "x", "--golden-path", "no/such/file.json"]) == 1
|
||||
assert "failed to load golden samples" in capsys.readouterr().err
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. Checked-in fixtures end to end (the owner-requested real-entry coverage)
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.parametrize(
|
||||
"fixture_name,golden_id,expected",
|
||||
[
|
||||
(
|
||||
"positive_600519_technical",
|
||||
"600519_technical",
|
||||
dict(
|
||||
expected_hit_rate=1.0,
|
||||
missing_expected=[],
|
||||
redundant_calls=0,
|
||||
cached_calls=0,
|
||||
failed_calls=0,
|
||||
retries=0,
|
||||
distinct_steps=5,
|
||||
max_steps_touched=False,
|
||||
violations=[],
|
||||
),
|
||||
),
|
||||
(
|
||||
"negative_600519_technical",
|
||||
"600519_technical",
|
||||
dict(
|
||||
expected_hit_rate=pytest.approx(2 / 3),
|
||||
missing_expected=["analyze_trend"],
|
||||
redundant_calls=1,
|
||||
cached_calls=1,
|
||||
failed_calls=1,
|
||||
retries=0,
|
||||
distinct_steps=8,
|
||||
max_steps_touched=True,
|
||||
),
|
||||
),
|
||||
(
|
||||
"retry_000001_core_data_strict",
|
||||
"000001_core_data_strict",
|
||||
dict(
|
||||
expected_hit_rate=1.0,
|
||||
missing_expected=[],
|
||||
redundant_calls=2,
|
||||
cached_calls=0,
|
||||
failed_calls=1,
|
||||
retries=1,
|
||||
distinct_steps=5,
|
||||
max_steps_touched=False,
|
||||
violations=[],
|
||||
),
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_fixtures_end_to_end(fixture_name, golden_id, expected):
|
||||
log, total_steps = _fixture(fixture_name)
|
||||
m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()[golden_id])
|
||||
for field, value in expected.items():
|
||||
assert getattr(m, field) == value, (fixture_name, field)
|
||||
if "violations" in expected:
|
||||
assert m.violations == expected["violations"]
|
||||
else:
|
||||
assert m.violations and "trajectory reached allowed_max_steps (8)" in m.violations
|
||||
Reference in New Issue
Block a user