diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f5ae6cb50..4b1183d19 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/). > For user-friendly release highlights, see the [GitHub Releases](https://github.com/ZhuLinsen/daily_stock_analysis/releases) page. ## [Unreleased] +- [新功能] 新增最小 Agent 轨迹评估入口 `evals/agent_trajectory/`(Refs #1956):纯函数指标层只消费真实 `tool_calls_log + AgentResult`,冻结最小指标契约(工具命中、冗余/缓存、失败/重试、总步数/max_steps),`run_eval.py` 经 `build_agent_executor` 真实执行并输出文本摘要 + 结构化 JSON 报告;评估为 reporter 非 gate,零 `src/` 改动 + - [修复] 将 litellm 依赖窗口上界收敛到 `<1.99.0`:1.99.0 起把 `prompt_cache_key` 透传给 OpenAI provider,破坏 provider 缓存测试对不透传行为的既有断言(CI backend-tests 3/3 与 backend-gate 失败);保留历史最低版本与 `!=1.82.7`/`!=1.82.8` 事故排除,同时同步更新各 LLM 兼容文档中写死的依赖约束表述,避免文档与 requirements.txt 漂移 - [新功能] 新增 `SEARXNG_TIMEOUT_SECONDS` 配置自建 SearXNG 单次搜索超时(默认 10 秒),已接线全部 SearchService 构造入口(含题材搜索子进程重建)与默认 GitHub Actions 工作流 diff --git a/docs/agent-trajectory-eval.md b/docs/agent-trajectory-eval.md new file mode 100644 index 000000000..8a322a834 --- /dev/null +++ b/docs/agent-trajectory-eval.md @@ -0,0 +1,82 @@ +# Agent Trajectory 评估(最小版,Refs #1956) + +## 定位 + +`evals/agent_trajectory/` 提供一套**离线可运行的最小轨迹评估管线**:用真实 `tool_calls_log + AgentResult` 跑一个 golden 样例,输出结构化 JSON 报告与简短可读文本摘要。评估结果是 **reporter 而非 gate** —— 指标违规只反映在报告里,不会让进程失败,也不进入 CI 门禁。 + +- 指标层(`metrics.py`)是纯函数:只消费轨迹日志与 golden 样例,不 import `src/`,不触网、不调 LLM,可离线单测。 +- 入口(`run_eval.py`)通过 `build_agent_executor()` 构建真实执行器,与 `src/core/pipeline.py` 使用同一个执行捕获钩子,消费真实产物。 +- 入口只支持**单 agent 运行**(`AGENT_ARCH=single`,默认):`AGENT_ARCH=multi` 时 factory 返回 orchestrator,其轨迹是各 stage 局部步号的拼接、`total_steps` 为 stage 数,与单 runner 指标契约不兼容——入口在构建前明确报错并退出(退出码 1)。 +- 入口用真实工具注册表校验 golden:`expected_tools` 拼错或过期会被判为无效样例(退出码 1),而不是静默按低命中继续评分。 +- 本次冻结**最小指标契约**,股票 guard、Codex `arguments_summary` 等扩展语义明确留给后续 PR(见文末「不在范围」)。 + +## 快速开始 + +```bash +# 跑单个样例(真实 LLM,依赖本地已配置的模型) +python evals/agent_trajectory/run_eval.py --sample 600519_technical + +# 跑全部样例,并输出结构化 JSON(按样例 id 键控) +python evals/agent_trajectory/run_eval.py --all --json-out eval_report.json +``` + +参数: + +| 参数 | 说明 | +| --- | --- | +| `--sample ID` / `--all` | 二选一必填:跑单个 golden 样例或全部 | +| `--golden-path PATH` | 自定义 golden JSON 路径(默认模块旁 `golden_samples.json`) | +| `--json-out PATH` | 写结构化 JSON 报告(`--all` 时为键控对象) | + +退出码:`0` 运行成功(含违规);`1` golden 加载(含 `expected_tools` 不在真实工具注册表)/ 样例选择 / 工具注册表加载失败 / 执行器构建(含 `AGENT_ARCH=multi` 拒绝)/ 运行失败(含执行器返回 `success=false`,如 provider 未配置、LLM 错误、超时、max_steps 耗尽、dashboard 解析失败);`2` 用法错误。 + +## 冻结的最小指标契约 + +对每条 `tool_calls_log`(runner 契约:每项含 `step / tool / arguments / success / duration / result_length / cached`,可选 `timeout` / `guarded`),只统计: + +| 指标 | 定义 | +| --- | --- | +| `expected_hit_rate` / `missing_expected` | 期望工具按**工具名**命中(本版不做股票维度判定);`expected_total` 为去重后的期望数 | +| `optional_tools_used` | 期望集合之外实际调用的工具;`allow_optional_tools=false` 时记为违规 | +| `redundant_calls` | 同一 (tool, args-key) 对在首次出现之后的每一次出现(不论成败)。args-key = `json.dumps(arguments, sort_keys=True, default=str)`,与运行时缓存键同思路 | +| `retries` | 紧跟**失败**之后重试同一 (tool, args-key) 对;`retries ⊆ redundant_calls`。成功会清除该对的失败态:`fail → success → success` 只计 1 次 retry(后一次 success 仅计冗余) | +| `failed_calls` | `success=false` 的条目数 | +| `cached_calls` | `cached=true` 的条目数(runner 语义:复用不可重试的失败结果) | +| `distinct_steps` / `max_steps_touched` | 日志 step 与 `AgentResult.total_steps` 取较大者(最后纯回答轮不产生工具调用,日志会低估);`max_steps_touched` 为 `max(step) >= allowed_max_steps` 的启发式 | + +## Golden 样例 schema + +`golden_samples.json` 是一个数组,字段: + +| 字段 | 类型 | 说明 | +| --- | --- | --- | +| `id` | string | 唯一样例 id(必填) | +| `task_description` | string | 交给执行器的任务文本(必填) | +| `stock_code` | string | 原样传入 runner context(`{"stock_code": ...}`);空串 = 无 context;不参与打分 | +| `expected_tools` | string[] | 期望工具名(必填,非空、无重复) | +| `allowed_max_steps` | int | 步数预算启发式(默认 10,>= 1) | +| `allow_optional_tools` | bool | 是否容忍期望外工具(默认 true) | + +校验:加载路径(`load_golden_samples`)直接拒绝非法样例;直接构造路径(`compute_trajectory_metrics`)以 validator 完全相同的措辞逐条上报违规,两条路径的契约按构造保持一致。`known_tool_names` 可注入真实工具注册表做成员校验(metrics 层自身不 import `src/`);`run_eval.py` 入口会自动注入真实工具注册表。 + +## JSON 报告 schema + +单个样例: + +```json +{ + "sample_id": "600519_technical", + "task_description": "…", + "stock_code": "600519", + "metrics": { "expected_hit_rate": …, "expected_total": …, "…": "… 共 11 个字段" }, + "violations": ["…"] +} +``` + +`--all --json-out` 时外层为 `{sample_id: <上述对象>}` 键控对象。 + +## 不在范围(后续 PR) + +- 股票维度命中判定与 guard 拦截语义(`guarded` / 越界调用违规) +- Codex App Server 的 `arguments_summary` 方言识别 +- 任何 `.env` / 运行时配置与 CI 门禁(本 PR 零 `src/` 改动,不影响现有分析流程) diff --git a/evals/agent_trajectory/__init__.py b/evals/agent_trajectory/__init__.py new file mode 100644 index 000000000..c362965c2 --- /dev/null +++ b/evals/agent_trajectory/__init__.py @@ -0,0 +1,6 @@ +# -*- coding: utf-8 -*- +"""Agent trajectory evaluation package (Issue #1956). + +Pure-function metrics layer that scores a real ``tool_calls_log`` against +golden samples; the runnable entry point lives in ``run_eval.py``. +""" diff --git a/evals/agent_trajectory/golden_samples.json b/evals/agent_trajectory/golden_samples.json new file mode 100644 index 000000000..a6e2270ff --- /dev/null +++ b/evals/agent_trajectory/golden_samples.json @@ -0,0 +1,18 @@ +[ + { + "id": "600519_technical", + "task_description": "分析贵州茅台(600519)近期技术面走势,输出趋势判断", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote", "get_daily_history", "analyze_trend"], + "allowed_max_steps": 8, + "allow_optional_tools": true + }, + { + "id": "000001_core_data_strict", + "task_description": "获取平安银行(000001)核心行情数据,不做扩展检索", + "stock_code": "000001", + "expected_tools": ["get_realtime_quote", "get_daily_history"], + "allowed_max_steps": 6, + "allow_optional_tools": false + } +] diff --git a/evals/agent_trajectory/metrics.py b/evals/agent_trajectory/metrics.py new file mode 100644 index 000000000..b3365ca17 --- /dev/null +++ b/evals/agent_trajectory/metrics.py @@ -0,0 +1,421 @@ +# -*- coding: utf-8 -*- +"""Pure-function trajectory metrics for agent evaluation (Issue #1956). + +This module computes quality metrics for an agent execution trajectory from its +``tool_calls_log`` (see ``src/agent/runner.py`` for the producer contract): + +* each entry carries ``step / tool / arguments / success / duration / + result_length / cached`` and optionally ``timeout`` or ``guarded`` fields; +* entries with missing optional fields are tolerated with defaults. + +The scoring functions in this module are pure: ``compute_trajectory_metrics``, +``format_text_report`` and ``validate_golden_sample`` consume plain data (a log +list and a ``GoldenSample``) and never touch the filesystem, network, or LLM. +The one exception is the loader ``load_golden_samples``, which reads the golden +JSON file from disk. This keeps the metrics layer deterministic, unit-testable +without an API key, and free of ``src/`` imports — it can score trajectories +from any source. The runnable entry point that produces real trajectories and +feeds them into this layer lives in ``run_eval.py``. + +Both entry paths enforce the same golden-sample structure contract: +``validate_golden_sample`` (the loader path) rejects malformed samples +outright, while ``compute_trajectory_metrics`` (the direct-construction path) +excludes the invalid parts from scoring, reports its own scoring-level +violations with the validator's wording, and finally appends *every* issue +``validate_golden_sample`` reports, verbatim and deduplicated — a caller who +builds a ``GoldenSample`` by hand can never get a silently relaxed result, +and a new validator check applies to direct scoring automatically (see the +compute docstring for the exact mapping). + +Idempotency key contract +------------------------ +Two log entries are considered "the same call" when their ``tool`` names are +equal and their serialized ``arguments`` are equal. Arguments may contain +unhashable values (dict / list), so the key is built with +``json.dumps(arguments, sort_keys=True, default=str)`` — a *stable string*, +not a hash. Do not replace this with ``tuple(arguments)`` or ``repr()``: +insertion order or collection type would then change call identity and corrupt +redundancy / retry counts. + +Metric semantics +---------------- +* ``redundant_calls``: every occurrence of a (tool, args-key) pair beyond its + first — regardless of success. +* ``retries``: occurrences that follow a *failed* occurrence of the same pair + (i.e. "tried again after a failure"). ``retries`` is a subset of + ``redundant_calls``; repeats after success count as redundant but not retry, + and a success clears the pair's failure state — so ``fail -> success -> + success`` counts exactly one retry, not two. +* ``failed_calls``: entries with ``success=False``. +* ``cached_calls``: entries with ``cached=True`` (runner semantics: reuse of a + non-retriable failure result). +* ``max_steps_touched``: the log does not carry ``max_steps`` itself, so this + is the conservative heuristic ``max(step) >= golden.allowed_max_steps`` — + a proxy for "the run reached the step budget", not proof of the loop + exhausting it. When ``total_steps`` is supplied (see + :func:`compute_trajectory_metrics`), the larger of ``total_steps`` and the + log-derived step is compared instead — the final answer round consumes a + step but produces no tool call, so the log alone understates consumption. + +Scope +----- +This is the frozen minimal contract requested for the close-and-rebuild PR +(Refs #1956): tool hit, redundancy / caching, failure / retry and total +steps / max_steps only. Stock guard and Codex ``arguments_summary`` +semantics are out of scope here and belong in follow-up PRs (see +``docs/agent-trajectory-eval.md``). +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, fields +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + + +@dataclass +class GoldenSample: + """Expected trajectory for one evaluation task. + + ``expected_tools`` are the tool names the agent should call; tools outside + this set are tolerated only when ``allow_optional_tools`` is true. + ``stock_code`` is carried verbatim into the runner context by + ``run_eval.py`` (empty string = no stock context) and does not take part + in scoring. + """ + + id: str + task_description: str + expected_tools: List[str] + stock_code: str = "" + allowed_max_steps: int = 10 + allow_optional_tools: bool = True + + +@dataclass +class TrajectoryMetrics: + """All metrics computed for one trajectory against one golden sample.""" + + expected_hit_rate: float + expected_total: int + missing_expected: List[str] + optional_tools_used: List[str] + redundant_calls: int + cached_calls: int + failed_calls: int + retries: int + distinct_steps: int + max_steps_touched: bool + violations: List[str] + + +def _args_key(arguments: Any) -> str: + """Return a stable idempotency key for tool-call arguments (see module docstring). + + ``None`` (an entry without an ``arguments`` payload) is serialized as an + empty object, so arguments-less calls of one tool share a single identity. + Non-dict payloads are serialized as-is. + """ + if arguments is None: + arguments = {} + return json.dumps(arguments, ensure_ascii=False, sort_keys=True, default=str) + + +def _entry_arguments(entry: Dict[str, Any]) -> Any: + """Extract the idempotent argument payload from a log entry. + + Runner entries carry ``arguments`` (a dict); entries without the field + key as ``None``, which ``_args_key`` treats as an empty object. + """ + return entry.get("arguments") + + +def _coerce_step(value: Any) -> int: + """Coerce a log entry's ``step`` to a non-negative int (missing/odd -> 0).""" + try: + step = int(value) + except (TypeError, ValueError): + return 0 + return step if step > 0 else 0 + + +def compute_trajectory_metrics( + log: List[Dict[str, Any]], + golden: GoldenSample, + total_steps: Optional[int] = None, +) -> TrajectoryMetrics: + """Compute all trajectory metrics from a ``tool_calls_log`` and a golden sample. + + ``total_steps`` optionally carries the number of loop rounds the run + actually consumed (``AgentResult.total_steps``), which includes the + final plain-answer round that produces no tool call. When it is larger + than the log-derived step count it is used for ``distinct_steps`` and the + ``max_steps_touched`` heuristic; otherwise the log alone decides, and the + default ``None`` keeps the log-only behaviour. + + Direct construction of a hand-edited ``GoldenSample`` is held to the same + structure contract as :func:`validate_golden_sample` on the loader path: + malformed parts are excluded from scoring instead of silently reshaping + the result, scoring-level violations are reported with the validator's + wording, and every issue the validator reports is appended verbatim + (deduplicated) before returning. Malformed ``expected_tools`` elements — + non-strings, empty strings and whitespace-only strings, the validator's + exact predicate — are dropped with an explicit violation, and a + non-positive ``allowed_max_steps`` is reported with the validator's + wording (and the budget assertion stays disabled). + """ + used_tools: List[str] = [] + key_counts: Dict[tuple, int] = {} + key_failed_seen: Dict[tuple, bool] = {} + key_retries: Dict[tuple, int] = {} + # Extract the expected tool list before scanning entries. Malformed + # elements are not silently dropped: they are reported as a violation + # below (mirroring validate_golden_sample, which rejects them at load + # time) and only the valid names take part in scoring. + if isinstance(golden.expected_tools, list): + # Same element predicate as validate_golden_sample(): whitespace-only + # strings count as malformed too, not just falsy ones. + expected_tools_malformed = [t for t in golden.expected_tools if not isinstance(t, str) or not t.strip()] + expected = [t for t in golden.expected_tools if isinstance(t, str) and t.strip()] + else: + # Defend against hand-edited samples passing a bare string: + # validation rejects it at load time, but scoring must not misparse + # it into per-character tool names either. + expected_tools_malformed = [] + expected = [] + # A hand-edited sample may repeat a tool name; normalize to first + # occurrences before scoring so the hit rate cannot be inflated + # (["quote", "quote"] with one quote call must read 1/2, not 2/3). + expected_dupes = len(set(expected)) != len(expected) + if expected_dupes: + expected = list(dict.fromkeys(expected)) + failed_calls = 0 + cached_calls = 0 + redundant_calls = 0 + distinct_steps = 0 + max_step = 0 + seen_steps: set = set() + + for entry in log: + if not isinstance(entry, dict): + continue + tool = entry.get("tool") or "" + success = bool(entry.get("success", True)) + if tool and tool not in used_tools: + used_tools.append(tool) + step = _coerce_step(entry.get("step")) + if step and step not in seen_steps: + seen_steps.add(step) + distinct_steps += 1 + max_step = max(max_step, step) + if not success: + failed_calls += 1 + if entry.get("cached"): + cached_calls += 1 + + key = (tool, _args_key(_entry_arguments(entry))) + if key_counts.get(key, 0): + redundant_calls += 1 + key_counts[key] = key_counts.get(key, 0) + 1 + # An occurrence is a retry only when the same call already failed + # before it (see module docstring for the precise contract). + if key_failed_seen.get(key): + key_retries[key] = key_retries.get(key, 0) + 1 + # A success clears the failure state: repeats after a recovery count + # as redundant only, not as further retries. + key_failed_seen[key] = not success + + retries = sum(key_retries.values()) + violations: List[str] = [] + + if expected_dupes: + violations.append("expected_tools must not contain duplicate names") + if expected_tools_malformed: + violations.append("expected_tools must contain only non-empty strings") + missing_expected = [t for t in expected if t not in used_tools] + expected_hit_rate = (len(expected) - len(missing_expected)) / len(expected) if expected else 0.0 + optional_tools_used = [t for t in used_tools if t not in expected] + + if not isinstance(golden.expected_tools, list): + violations.append("expected_tools must be a list of tool names") + elif not golden.expected_tools: + violations.append("expected_tools must be a non-empty list") + + # Malformed golden samples must not crash scoring nor flip semantics: + # a truthy string like "false" must not silently turn a strict sample + # permissive, and a non-integer step limit must not crash the comparison. + optional_allowed = golden.allow_optional_tools + if not isinstance(optional_allowed, bool): + violations.append("allow_optional_tools must be a boolean") + optional_allowed = False + if optional_tools_used and not optional_allowed: + violations.append(f"optional tools used but not allowed: {', '.join(optional_tools_used)}") + + # The final answer round consumes a step but produces no tool call, so + # when the caller supplies the run's real total it may exceed the log. + total = 0 + if total_steps is not None: + try: + total = int(total_steps) + except (TypeError, ValueError): + total = 0 + total = total if total > 0 else 0 + distinct_steps = max(distinct_steps, total) + max_step = max(max_step, total) + + limit = golden.allowed_max_steps + if isinstance(limit, bool) or not isinstance(limit, int): + violations.append("allowed_max_steps must be an integer") + limit = 0 + elif limit < 1: + # Validator wording for the same field: a non-positive limit must + # not silently disable the budget assertion on the direct-score path. + violations.append("allowed_max_steps must be >= 1") + limit = 0 + max_steps_touched = bool(max_step and limit > 0 and max_step >= limit) + if max_steps_touched: + violations.append(f"trajectory reached allowed_max_steps ({golden.allowed_max_steps})") + + # The direct-score path surfaces the validator's complete structure + # contract, not only the checks scoring itself depends on: every issue + # validate_golden_sample() reports is appended verbatim (deduplicated + # against the inline violations above), so a hand-built malformed golden + # can never score as valid — and a new validator check applies to direct + # scoring automatically. The two entry paths stay aligned by + # construction instead of by convention. + for issue in validate_golden_sample(golden): + if issue not in violations: + violations.append(issue) + + return TrajectoryMetrics( + expected_hit_rate=expected_hit_rate, + expected_total=len(expected), + missing_expected=missing_expected, + optional_tools_used=optional_tools_used, + redundant_calls=redundant_calls, + cached_calls=cached_calls, + failed_calls=failed_calls, + retries=retries, + distinct_steps=distinct_steps, + max_steps_touched=max_steps_touched, + violations=violations, + ) + + +def format_text_report(m: TrajectoryMetrics) -> str: + """Render metrics as a deterministic, human-readable text report.""" + hit_count = max(0, m.expected_total - len(m.missing_expected)) + hit_percent = f"{m.expected_hit_rate * 100:.1f}%" + missing = ", ".join(m.missing_expected) if m.missing_expected else "无" + optional = ", ".join(m.optional_tools_used) if m.optional_tools_used else "无" + violations = "; ".join(m.violations) if m.violations else "无" + max_steps_label = "是" if m.max_steps_touched else "否" + return ( + "============================================\n" + "Agent Trajectory 评估报告\n" + "============================================\n" + f"- 期望工具命中: {hit_count}/{m.expected_total} ({hit_percent})\n" + f"- 缺失期望工具: {missing}\n" + f"- 期望外工具: {optional}\n" + f"- 冗余调用: {m.redundant_calls} | 缓存调用: {m.cached_calls} | " + f"失败调用: {m.failed_calls} | 重试: {m.retries}\n" + f"- 消耗步数: {m.distinct_steps} (触碰 max_steps: {max_steps_label})\n" + f"- 违规项: {violations}\n" + ) + + +def load_golden_samples( + path: Optional[str] = None, + known_tool_names: Optional[Iterable[str]] = None, +) -> List[GoldenSample]: + """Load golden samples from ``path`` (default: ``golden_samples.json`` next to this module). + + Raises ``FileNotFoundError`` when the file is missing and ``ValueError`` on + malformed JSON or structural issues (see :func:`validate_golden_sample`). + Unknown extra JSON keys are ignored so the file can carry forward-looking + metadata without breaking the loader. + """ + if path is None: + path = str(Path(__file__).with_name("golden_samples.json")) + data = json.loads(Path(path).read_text(encoding="utf-8")) + if not isinstance(data, list): + raise ValueError(f"golden samples file must contain a JSON list, got {type(data).__name__}") + + # Materialize once before the loop: a one-shot generator must survive the + # validation of every sample, not just the first. + known = set(known_tool_names) if known_tool_names is not None else None + golden_fields = {f.name for f in fields(GoldenSample)} + samples: List[GoldenSample] = [] + seen_ids: set = set() + for index, item in enumerate(data): + if not isinstance(item, dict): + raise ValueError(f"sample #{index} must be a JSON object, got {type(item).__name__}") + try: + sample = GoldenSample(**{k: v for k, v in item.items() if k in golden_fields}) + except TypeError as exc: + raise ValueError(f"sample #{index} has invalid fields: {exc}") from exc + # Structural validation runs before duplicate detection so that a + # mistyped (possibly unhashable) id is rejected as a ValueError here + # instead of crashing the membership check below. + issues = validate_golden_sample(sample, known) + if issues: + raise ValueError(f"sample '{sample.id}': " + "; ".join(issues)) + if sample.id in seen_ids: + raise ValueError(f"duplicate sample id: {sample.id}") + seen_ids.add(sample.id) + samples.append(sample) + return samples + + +def validate_golden_sample( + sample: GoldenSample, + known_tool_names: Optional[Iterable[str]] = None, +) -> List[str]: + """Return a list of structural issues for ``sample``; empty list means valid. + + When ``known_tool_names`` is provided, ``expected_tools`` must be a subset + of it; the caller supplies the authoritative registry names (this module + deliberately does not import ``src/``). Any ``Iterable[str]`` is accepted + — including one-shot generators — and materialized once internally, so + membership checks never consume the caller's iterable. + + Field *types* are part of the structural contract — hand-edited golden + JSON must fail with a clear message instead of crashing or silently + passing: text fields must be strings, ``expected_tools`` must be a list of + non-empty, duplicate-free names, ``allowed_max_steps`` an integer >= 1 and + ``allow_optional_tools`` a boolean. ``stock_code`` must be a string; the + empty string is the documented default ("no stock context", see + ``run_eval.py``) and passes, while whitespace-only values fail. + """ + issues: List[str] = [] + known = set(known_tool_names) if known_tool_names is not None else None + if not isinstance(sample.id, str) or not sample.id.strip(): + issues.append("id must be a non-empty string") + if not isinstance(sample.task_description, str) or not sample.task_description.strip(): + issues.append("task_description must be a non-empty string") + if not isinstance(sample.stock_code, str): + issues.append("stock_code must be a string") + elif sample.stock_code != sample.stock_code.strip(): + issues.append("stock_code must not contain leading or trailing whitespace") + if not isinstance(sample.expected_tools, list): + issues.append("expected_tools must be a list of tool names") + elif not sample.expected_tools: + issues.append("expected_tools must be a non-empty list") + elif any(not isinstance(t, str) or not t.strip() for t in sample.expected_tools): + issues.append("expected_tools must contain only non-empty strings") + elif len(set(sample.expected_tools)) != len(sample.expected_tools): + issues.append("expected_tools must not contain duplicate names") + elif known is not None: + # Only reachable when expected_tools is a non-empty list of non-empty + # strings, so malformed values can never crash the membership check. + unknown = [t for t in sample.expected_tools if t not in known] + if unknown: + issues.append(f"unknown expected_tools: {', '.join(unknown)}") + if isinstance(sample.allowed_max_steps, bool) or not isinstance(sample.allowed_max_steps, int): + issues.append("allowed_max_steps must be an integer") + elif sample.allowed_max_steps < 1: + issues.append("allowed_max_steps must be >= 1") + if not isinstance(sample.allow_optional_tools, bool): + issues.append("allow_optional_tools must be a boolean") + return issues diff --git a/evals/agent_trajectory/run_eval.py b/evals/agent_trajectory/run_eval.py new file mode 100644 index 000000000..28dd9cfda --- /dev/null +++ b/evals/agent_trajectory/run_eval.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""Run one agent-trajectory eval sample against the real agent executor (Issue #1956). + +Consumes a real ``tool_calls_log + AgentResult`` produced by +``src.agent.factory.build_agent_executor`` (the same capture hook the +analysis pipeline uses in ``src/core/pipeline.py``), scores it with the +pure metrics layer and emits a short human-readable text summary plus an +optional structured JSON report. + +The eval is a *reporter*, not a gate: metric violations lower the report, +they never fail the process. Exit codes: 0 = ran (violations included), +1 = load (golden / tool registry) / build / run failure, 2 = usage error. + +The entry supports the single-agent runner only. When ``AGENT_ARCH=multi`` +the factory returns the orchestrator whose trajectories use per-stage local +step numbers, which breaks the single-runner metric contract, so the entry +rejects that arch up front with exit code 1. Golden samples are validated +against the real tool registry before running, so misspelled or stale +``expected_tools`` fail as invalid samples instead of scoring as low hit +rate. + +Usage: + python evals/agent_trajectory/run_eval.py --sample 600519_technical + python evals/agent_trajectory/run_eval.py --all --json-out eval_report.json +""" + +from __future__ import annotations + +import argparse +import json +import sys +from dataclasses import asdict +from pathlib import Path +from typing import Any, Dict, List, Optional + +# Add the project root to sys.path so the script also works as +# `python evals/agent_trajectory/run_eval.py` from anywhere. +REPO_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO_ROOT)) + +from evals.agent_trajectory.metrics import ( + GoldenSample, + TrajectoryMetrics, + compute_trajectory_metrics, + format_text_report, + load_golden_samples, +) + + +def _build_report(sample: GoldenSample, metrics: TrajectoryMetrics) -> Dict[str, Any]: + """Structured JSON report for one sample (schema in docs/agent-trajectory-eval.md).""" + return { + "sample_id": sample.id, + "task_description": sample.task_description, + "stock_code": sample.stock_code, + "metrics": asdict(metrics), + "violations": metrics.violations, + } + + +def _write_json(path: Optional[Path], payload: Any) -> None: + """Write ``payload`` as indented UTF-8 JSON (trailing newline).""" + if path is None: + return + Path(path).write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +_KNOWN_TOOL_NAMES: Optional[set] = None + + +def _check_agent_arch() -> None: + """Reject multi-agent arch up front (mirrors the factory's own decision). + + ``src.agent.factory.build_agent_executor`` returns the orchestrator when + ``config.agent_arch == "multi"``; its trajectories concatenate per-stage + logs with local step numbering and set ``total_steps`` to the stage count, + which the single-runner metric contract cannot interpret. Fail fast + instead of scoring a distorted trajectory. + """ + from src.config import get_config + + arch = getattr(get_config(), "agent_arch", "single") + if arch == "multi": + raise RuntimeError( + "AGENT_ARCH=multi is not supported by this minimal eval: " + "orchestrator trajectories use per-stage local step numbers, " + "which break the single-runner metric contract" + ) + + +def _known_tool_names(): + """Authoritative tool names from the real registry modules (lazy, cached). + + The metrics layer deliberately does not import ``src/``; the entry point + supplies these names so ``load_golden_samples`` rejects misspelled or + stale ``expected_tools`` instead of scoring them as low hit rate. + """ + global _KNOWN_TOOL_NAMES + if _KNOWN_TOOL_NAMES is None: + from src.agent.tools.analysis_tools import ALL_ANALYSIS_TOOLS + from src.agent.tools.backtest_tools import ALL_BACKTEST_TOOLS + from src.agent.tools.data_tools import ALL_DATA_TOOLS + from src.agent.tools.market_tools import ALL_MARKET_TOOLS + from src.agent.tools.search_tools import ALL_SEARCH_TOOLS + + all_tools = ALL_DATA_TOOLS + ALL_ANALYSIS_TOOLS + ALL_SEARCH_TOOLS + ALL_MARKET_TOOLS + ALL_BACKTEST_TOOLS + _KNOWN_TOOL_NAMES = {tool_def.name for tool_def in all_tools} + return _KNOWN_TOOL_NAMES + + +def _build_executor(): + """Build the real agent executor (lazy import so tests can monkeypatch this).""" + _check_agent_arch() + from src.agent.factory import build_agent_executor + + return build_agent_executor() + + +def run_sample(executor, sample: GoldenSample, *, json_out: Optional[Path] = None) -> TrajectoryMetrics: + """Run one golden sample against a duck-typed agent executor and score it. + + ``executor`` only needs ``run(task, context=None) -> result`` where + ``result`` carries ``tool_calls_log`` (and optionally ``total_steps``) — + the same shape as ``src.agent.executor.AgentResult``. The production + executor is built lazily by :func:`_build_executor`; tests may pass a + stub. A result carrying an explicit ``success=False`` is a run failure + (the executor reported a provider / timeout / budget error) and raises + ``RuntimeError`` before any scoring; duck-typed results without a + ``success`` attribute are treated as successful. The text summary is + always printed to stdout; ``json_out`` additionally writes the + structured report for this sample. + """ + context: Optional[Dict[str, Any]] = None + if sample.stock_code: + context = {"stock_code": sample.stock_code} + result = executor.run(sample.task_description, context=context) + if getattr(result, "success", None) is False: + error = getattr(result, "error", None) + raise RuntimeError(f"agent run failed (success=false): {error or 'no error detail'}") + log = getattr(result, "tool_calls_log", None) or [] + total_steps = getattr(result, "total_steps", None) + metrics = compute_trajectory_metrics(log, sample, total_steps=total_steps) + print(format_text_report(metrics)) + _write_json(json_out, _build_report(sample, metrics)) + return metrics + + +def main(argv=None) -> int: + """Run one golden sample (--sample ID) or every sample (--all).""" + parser = argparse.ArgumentParser( + description="Run one agent-trajectory eval sample against the real agent executor (Issue #1956).", + ) + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument("--sample", metavar="ID", help="run the golden sample with this id") + group.add_argument("--all", action="store_true", help="run every golden sample") + parser.add_argument( + "--golden-path", + default=None, + help="path to golden_samples.json (default: the checked-in file next to this module)", + ) + parser.add_argument( + "--json-out", + default=None, + help="write a structured JSON report to this path (--all writes a keyed object)", + ) + args = parser.parse_args(argv) + + try: + known = _known_tool_names() + except Exception as exc: + print(f"error: failed to load tool registry for golden validation: {exc}", file=sys.stderr) + return 1 + + try: + samples = load_golden_samples(path=args.golden_path, known_tool_names=known) + except (OSError, ValueError) as exc: + print(f"error: failed to load golden samples: {exc}", file=sys.stderr) + return 1 + + if args.sample: + selected = next((s for s in samples if s.id == args.sample), None) + if selected is None: + available = ", ".join(s.id for s in samples) if samples else "(none)" + print(f"error: unknown sample id '{args.sample}'; available: {available}", file=sys.stderr) + return 1 + samples = [selected] + + try: + executor = _build_executor() + except Exception as exc: # pragma: no cover - exercised via monkeypatched failure + print(f"error: failed to build agent executor: {exc}", file=sys.stderr) + return 1 + + reports: Dict[str, Dict[str, Any]] = {} + for sample in samples: + print(f"[eval] sample: {sample.id} | task: {sample.task_description}") + try: + metrics = run_sample(executor, sample) + except Exception as exc: + print(f"error: sample '{sample.id}' failed: {exc}", file=sys.stderr) + return 1 + reports[sample.id] = _build_report(sample, metrics) + + if args.json_out: + _write_json(Path(args.json_out), reports if args.all else reports[args.sample]) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/fixtures/agent_trajectory/negative_600519_technical.json b/tests/fixtures/agent_trajectory/negative_600519_technical.json new file mode 100644 index 000000000..25fff4430 --- /dev/null +++ b/tests/fixtures/agent_trajectory/negative_600519_technical.json @@ -0,0 +1,41 @@ +{ + "total_steps": 8, + "tool_calls_log": [ + { + "step": 1, + "tool": "get_realtime_quote", + "arguments": {"stock_code": "600519"}, + "success": true, + "duration": 0.5, + "result_length": 100, + "cached": false + }, + { + "step": 2, + "tool": "get_realtime_quote", + "arguments": {"stock_code": "600519"}, + "success": true, + "duration": 0.6, + "result_length": 110, + "cached": false + }, + { + "step": 3, + "tool": "get_daily_history", + "arguments": {"stock_code": "600519", "days": 30}, + "success": false, + "duration": 1.5, + "result_length": 0, + "cached": true + }, + { + "step": 4, + "tool": "search_stock_news", + "arguments": {"query": "贵州茅台"}, + "success": true, + "duration": 2.0, + "result_length": 300, + "cached": false + } + ] +} diff --git a/tests/fixtures/agent_trajectory/positive_600519_technical.json b/tests/fixtures/agent_trajectory/positive_600519_technical.json new file mode 100644 index 000000000..852174142 --- /dev/null +++ b/tests/fixtures/agent_trajectory/positive_600519_technical.json @@ -0,0 +1,41 @@ +{ + "total_steps": 5, + "tool_calls_log": [ + { + "step": 1, + "tool": "get_realtime_quote", + "arguments": {"stock_code": "600519"}, + "success": true, + "duration": 0.5, + "result_length": 100, + "cached": false + }, + { + "step": 2, + "tool": "get_daily_history", + "arguments": {"stock_code": "600519", "days": 30}, + "success": true, + "duration": 0.8, + "result_length": 200, + "cached": false + }, + { + "step": 3, + "tool": "analyze_trend", + "arguments": {"stock_code": "600519"}, + "success": true, + "duration": 1.2, + "result_length": 150, + "cached": false + }, + { + "step": 4, + "tool": "search_stock_news", + "arguments": {"query": "贵州茅台"}, + "success": true, + "duration": 2.0, + "result_length": 300, + "cached": false + } + ] +} diff --git a/tests/fixtures/agent_trajectory/retry_000001_core_data_strict.json b/tests/fixtures/agent_trajectory/retry_000001_core_data_strict.json new file mode 100644 index 000000000..897b44f8a --- /dev/null +++ b/tests/fixtures/agent_trajectory/retry_000001_core_data_strict.json @@ -0,0 +1,41 @@ +{ + "total_steps": 5, + "tool_calls_log": [ + { + "step": 1, + "tool": "get_realtime_quote", + "arguments": {"stock_code": "000001"}, + "success": true, + "duration": 0.5, + "result_length": 100, + "cached": false + }, + { + "step": 2, + "tool": "get_daily_history", + "arguments": {"stock_code": "000001", "days": 60}, + "success": false, + "duration": 1.2, + "result_length": 0, + "cached": false + }, + { + "step": 3, + "tool": "get_daily_history", + "arguments": {"stock_code": "000001", "days": 60}, + "success": true, + "duration": 1.3, + "result_length": 250, + "cached": false + }, + { + "step": 4, + "tool": "get_daily_history", + "arguments": {"stock_code": "000001", "days": 60}, + "success": true, + "duration": 0.4, + "result_length": 250, + "cached": false + } + ] +} diff --git a/tests/test_agent_trajectory_metrics.py b/tests/test_agent_trajectory_metrics.py new file mode 100644 index 000000000..9e1a7a31c --- /dev/null +++ b/tests/test_agent_trajectory_metrics.py @@ -0,0 +1,716 @@ +# -*- coding: utf-8 -*- +"""Unit tests for the agent trajectory evaluation metrics (Issue #1956). + +All trajectories are synthetic ``tool_calls_log`` lists — no LLM, no network. +The golden-samples-file tests verify that the checked-in ``golden_samples.json`` +stays structurally valid and that every ``expected_tools`` entry exists in the +repository's real tool registry (imported lazily so the metrics-only sections +run even without ``src/`` importable). +""" + +import json + +import pytest + +from evals.agent_trajectory.metrics import ( + GoldenSample, + TrajectoryMetrics, + _args_key, + compute_trajectory_metrics, + format_text_report, + load_golden_samples, + validate_golden_sample, +) + + +def _entry(tool="get_realtime_quote", arguments=None, step=1, success=True, **extra): + """Build a runner-shaped ``tool_calls_log`` entry with sensible defaults.""" + entry = { + "step": step, + "tool": tool, + "arguments": arguments if arguments is not None else {"stock_code": "600519"}, + "success": success, + "duration": 0.5, + "result_length": 100, + "cached": False, + } + entry.update(extra) + return entry + + +def _golden(**overrides): + values = dict( + id="600519_technical", + task_description="分析贵州茅台近期技术面走势", + stock_code="600519", + expected_tools=["get_realtime_quote", "get_daily_history", "analyze_trend"], + allowed_max_steps=8, + allow_optional_tools=True, + ) + values.update(overrides) + return GoldenSample(**values) + + +def _metrics(**overrides): + values = dict( + expected_hit_rate=2 / 3, + expected_total=3, + missing_expected=["analyze_trend"], + optional_tools_used=[], + redundant_calls=0, + cached_calls=0, + failed_calls=0, + retries=0, + distinct_steps=3, + max_steps_touched=False, + violations=[], + ) + values.update(overrides) + return TrajectoryMetrics(**values) + + +# --------------------------------------------------------------------------- +# 1. Args-key stability +# --------------------------------------------------------------------------- +class TestArgsKey: + def test_key_stable_across_key_order_and_nested_unhashable(self): + a = {"b": [1, 2], "a": {"x": {"y": 1}}} + b = {"a": {"x": {"y": 1}}, "b": [1, 2]} + assert _args_key(a) == _args_key(b) + + def test_key_differs_for_different_arguments(self): + assert _args_key({"stock_code": "600519"}) != _args_key({"stock_code": "000001"}) + + def test_none_arguments_use_empty_object(self): + assert _args_key(None) == json.dumps({}) + + def test_non_dict_arguments_serialized(self): + assert _args_key(["600519"]) == '["600519"]' + + +# --------------------------------------------------------------------------- +# 2. Hit rate / expected & optional tools +# --------------------------------------------------------------------------- +class TestComputeMetricsHitRate: + @staticmethod + def _two_of_three_log(): + return [ + _entry(tool="get_realtime_quote", step=1), + _entry(tool="get_daily_history", step=2), + _entry(tool="search_stock_news", step=3), + ] + + def test_hit_rate_missing_and_optional(self): + m = compute_trajectory_metrics(self._two_of_three_log(), _golden()) + assert m.expected_hit_rate == pytest.approx(2 / 3) + assert m.missing_expected == ["analyze_trend"] + assert m.optional_tools_used == ["search_stock_news"] + assert m.violations == [] + + def test_optional_tools_not_allowed_produces_violation(self): + m = compute_trajectory_metrics(self._two_of_three_log(), _golden(allow_optional_tools=False)) + assert m.violations == ["optional tools used but not allowed: search_stock_news"] + assert m.expected_hit_rate == pytest.approx(2 / 3) + + def test_duplicate_tool_calls_still_full_hit(self): + log = [ + _entry(tool="get_realtime_quote", arguments={"stock_code": "600519"}, step=1), + _entry(tool="get_realtime_quote", arguments={"stock_code": "000001"}, step=2), + ] + m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote"])) + assert m.expected_hit_rate == 1.0 + assert m.missing_expected == [] + + def test_empty_log_yields_zero_metrics(self): + m = compute_trajectory_metrics([], _golden()) + assert m.expected_hit_rate == 0.0 + assert m.expected_total == 3 + assert m.missing_expected == ["get_realtime_quote", "get_daily_history", "analyze_trend"] + assert m.redundant_calls == 0 and m.cached_calls == 0 and m.failed_calls == 0 + assert m.retries == 0 and m.distinct_steps == 0 + assert m.max_steps_touched is False + assert m.violations == [] + + def test_non_dict_entries_ignored(self): + log = ["garbage", None, _entry(tool="get_realtime_quote", step=1)] + m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote"])) + assert m.expected_hit_rate == 1.0 + assert m.distinct_steps == 1 + + def test_empty_expected_tools_yields_zero_hit_and_violation(self): + m = compute_trajectory_metrics([_entry()], _golden(expected_tools=[])) + assert m.expected_hit_rate == 0.0 + assert "expected_tools must be a non-empty list" in m.violations + + def test_string_expected_tools_scored_as_empty_not_as_characters(self): + m = compute_trajectory_metrics([_entry()], _golden(expected_tools="get_realtime_quote")) + assert m.expected_hit_rate == 0.0 + assert m.expected_total == 0 + assert "expected_tools must be a list of tool names" in m.violations + + def test_non_bool_allow_optional_tools_scores_strictly(self): + log = [ + _entry(tool="get_realtime_quote", step=1), + _entry(tool="search_stock_news", step=2), + ] + m = compute_trajectory_metrics( + log, + _golden(expected_tools=["get_realtime_quote"], allow_optional_tools="false"), + ) + assert "allow_optional_tools must be a boolean" in m.violations + assert "optional tools used but not allowed: search_stock_news" in m.violations + + def test_duplicate_expected_tools_scored_as_unique_names(self): + # Regression: ["quote", "quote", "history"] with a single quote call + # must read 1/2, not 2/3. + golden = _golden( + expected_tools=[ + "get_realtime_quote", + "get_realtime_quote", + "get_daily_history", + ], + ) + m = compute_trajectory_metrics([_entry(tool="get_realtime_quote")], golden) + assert m.expected_total == 2 + assert m.expected_hit_rate == pytest.approx(0.5) + assert m.missing_expected == ["get_daily_history"] + assert "expected_tools must not contain duplicate names" in m.violations + + +# --------------------------------------------------------------------------- +# 2b. Direct-construction path: same structure contract as the loader +# --------------------------------------------------------------------------- +class TestDirectConstructionContract: + def test_malformed_expected_tools_elements_are_reported(self): + # Review counter-example: ['get_realtime_quote', ''] must not + # silently collapse into a one-tool golden — the malformed element + # is reported, only valid names take part in scoring. Whitespace- + # only elements follow the validator's predicate (t.strip()). + log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})] + for malformed in (["get_realtime_quote", ""], ["get_realtime_quote", 1]): + m = compute_trajectory_metrics(log, _golden(expected_tools=malformed)) + assert m.expected_hit_rate == 1.0 + assert "expected_tools must contain only non-empty strings" in m.violations + + def test_whitespace_only_expected_tool_does_not_pollute_scoring(self): + # Review counter-example: ' ' must not enter the hit-rate + # denominator nor show up as a missing tool — it is malformed. + log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})] + m = compute_trajectory_metrics(log, _golden(expected_tools=["get_realtime_quote", " "])) + assert m.expected_total == 1 + assert m.expected_hit_rate == 1.0 + assert m.missing_expected == [] + assert "expected_tools must contain only non-empty strings" in m.violations + + def test_non_positive_allowed_max_steps_reported_in_direct_score(self): + # Review counter-example: allowed_max_steps=0 / -3 must report the + # validator's wording instead of silently disabling the budget + # assertion, no matter how many steps the trajectory takes. + log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"}, step=s) for s in range(1, 100)] + for limit in (0, -3): + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=limit)) + assert "allowed_max_steps must be >= 1" in m.violations + assert m.max_steps_touched is False + + def test_direct_score_mirrors_validator_structure_contract(self): + # The owner-requested parity: for each malformed golden shape the + # validator rejects, the direct-score path must surface the same + # issue wording in its violations. + log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})] + cases = [ + (dict(expected_tools=["get_realtime_quote", " "]), "expected_tools must contain only non-empty strings"), + (dict(expected_tools=["get_realtime_quote", ""]), "expected_tools must contain only non-empty strings"), + (dict(allowed_max_steps=0), "allowed_max_steps must be >= 1"), + (dict(allowed_max_steps=-3), "allowed_max_steps must be >= 1"), + (dict(allow_optional_tools="false"), "allow_optional_tools must be a boolean"), + (dict(id=[]), "id must be a non-empty string"), + (dict(task_description=None), "task_description must be a non-empty string"), + (dict(stock_code=None), "stock_code must be a string"), + (dict(stock_code=" "), "stock_code must not contain leading or trailing whitespace"), + ] + for overrides, issue in cases: + golden = _golden(**overrides) + validator_issues = validate_golden_sample(golden) + assert any(issue in i for i in validator_issues), (issue, overrides, validator_issues) + m = compute_trajectory_metrics(log, golden) + assert any(issue in v for v in m.violations), (issue, overrides, m.violations) + + def test_every_validator_issue_surfaces_in_direct_score(self): + # Structural lock: the direct-score path surfaces the validator's + # COMPLETE issue list verbatim (this is how id / task_description / + # stock_code and any future validator check apply to direct scoring + # automatically). A thoroughly malformed sample must yield every + # validator issue in compute violations, with no duplicates from the + # inline scoring checks. + log = [_entry(tool="get_realtime_quote", arguments={"stock_code": "600519"})] + golden = GoldenSample( + id=[], + task_description=None, + stock_code=" ", + expected_tools=["", " "], + allowed_max_steps=0, + allow_optional_tools="false", + ) + validator_issues = validate_golden_sample(golden) + assert validator_issues, "the sample is thoroughly malformed" + m = compute_trajectory_metrics(log, golden) + for issue in validator_issues: + assert any(issue in v for v in m.violations), (issue, m.violations) + assert m.violations.count(issue) == 1, (issue, m.violations) + + +# --------------------------------------------------------------------------- +# 3. Retries, caching, failure counting +# --------------------------------------------------------------------------- +class TestRetryAndCaching: + def test_fail_then_retry_success_counts_one_retry(self): + log = [ + _entry(step=1, success=False), + _entry(step=2, success=True), + ] + m = compute_trajectory_metrics(log, _golden()) + assert m.retries == 1 + assert m.redundant_calls == 1 + assert m.failed_calls == 1 + + def test_same_tool_different_args_not_redundant(self): + log = [ + _entry(arguments={"stock_code": "600519"}, step=1), + _entry(arguments={"stock_code": "000001"}, step=2), + ] + m = compute_trajectory_metrics(log, _golden()) + assert m.redundant_calls == 0 + assert m.retries == 0 + + def test_repeat_after_success_is_redundant_but_not_retry(self): + log = [_entry(step=1, success=True), _entry(step=2, success=True)] + m = compute_trajectory_metrics(log, _golden()) + assert m.redundant_calls == 1 + assert m.retries == 0 + assert m.failed_calls == 0 + + def test_fail_fail_success_counts_two_retries(self): + log = [ + _entry(step=1, success=False), + _entry(step=2, success=False), + _entry(step=3, success=True), + ] + m = compute_trajectory_metrics(log, _golden()) + assert m.retries == 2 + assert m.redundant_calls == 2 + assert m.failed_calls == 2 + + def test_recovery_clears_failure_state(self): + # fail -> success -> success: only the recovery attempt is a retry; + # the repeat after success counts as redundant only. + log = [ + _entry(step=1, success=False), + _entry(step=2, success=True), + _entry(step=3, success=True), + ] + m = compute_trajectory_metrics(log, _golden()) + assert m.retries == 1 + assert m.redundant_calls == 2 + assert m.failed_calls == 1 + + def test_cached_entry_counted(self): + m = compute_trajectory_metrics([_entry(cached=True, success=False)], _golden()) + assert m.cached_calls == 1 + assert m.failed_calls == 1 + + +# --------------------------------------------------------------------------- +# 4. max_steps touching +# --------------------------------------------------------------------------- +class TestMaxStepsTouched: + def test_steps_reaching_allowed_max_touched(self): + log = [_entry(step=i) for i in range(1, 6)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5)) + assert m.max_steps_touched is True + assert "trajectory reached allowed_max_steps (5)" in m.violations + + def test_steps_below_limit_not_touched(self): + log = [_entry(step=i) for i in range(1, 5)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5)) + assert m.max_steps_touched is False + assert m.violations == [] + + def test_empty_log_not_touched(self): + m = compute_trajectory_metrics([], _golden(allowed_max_steps=5)) + assert m.max_steps_touched is False + + def test_non_integer_limit_surfaces_violation_without_crash(self): + log = [_entry(step=i) for i in range(1, 6)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps="5")) + assert m.max_steps_touched is False + assert "allowed_max_steps must be an integer" in m.violations + + +# --------------------------------------------------------------------------- +# 4b. total_steps input (final answer round) +# --------------------------------------------------------------------------- +class TestTotalStepsInput: + def test_total_steps_extends_step_metrics_beyond_log(self): + log = [_entry(step=1), _entry(step=2)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5), total_steps=4) + assert m.distinct_steps == 4 + assert m.max_steps_touched is False + + def test_total_steps_can_touch_the_limit(self): + log = [_entry(step=1), _entry(step=2)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=3), total_steps=3) + assert m.max_steps_touched is True + assert m.distinct_steps == 3 + + def test_log_wins_when_it_reaches_further(self): + log = [_entry(step=i) for i in range(1, 5)] + m = compute_trajectory_metrics(log, _golden(allowed_max_steps=5), total_steps=2) + assert m.distinct_steps == 4 + assert m.max_steps_touched is False + + def test_none_keeps_log_only_behaviour(self): + m = compute_trajectory_metrics([_entry(step=1)], _golden()) + assert m.distinct_steps == 1 + + def test_non_numeric_total_steps_ignored(self): + m = compute_trajectory_metrics([_entry(step=1)], _golden(), total_steps="not-a-number") + assert m.distinct_steps == 1 + + +# --------------------------------------------------------------------------- +# 5. Tolerant entry shapes (missing fields) +# --------------------------------------------------------------------------- +class TestTolerantEntries: + def test_missing_optional_fields_defaulted(self): + m = compute_trajectory_metrics([{"step": 1, "tool": "get_realtime_quote"}], _golden()) + assert m.failed_calls == 0 + assert m.cached_calls == 0 + assert m.expected_hit_rate == pytest.approx(1 / 3) + + def test_entries_without_arguments_share_one_identity(self): + # Arguments-less entries key as an empty object: a repeated call of + # the same tool is redundant even without an arguments payload. + log = [ + {"step": 1, "tool": "get_realtime_quote", "success": True}, + {"step": 2, "tool": "get_realtime_quote", "success": True}, + ] + m = compute_trajectory_metrics(log, _golden()) + assert m.redundant_calls == 1 + assert m.retries == 0 + + def test_invalid_step_coerced_to_zero(self): + m = compute_trajectory_metrics([_entry(step="not-a-number")], _golden()) + assert m.distinct_steps == 0 + + +# --------------------------------------------------------------------------- +# 6. Text report rendering +# --------------------------------------------------------------------------- +class TestFormatTextReport: + def test_contains_hit_fraction_and_percent(self): + text = format_text_report(_metrics()) + assert "2/3" in text + assert "66.7%" in text + + def test_empties_render_placeholder(self): + m = _metrics( + expected_hit_rate=0.0, + expected_total=3, + missing_expected=["get_realtime_quote", "get_daily_history", "analyze_trend"], + ) + text = format_text_report(m) + assert "缺失期望工具: get_realtime_quote, get_daily_history, analyze_trend" in text + assert "期望外工具: 无" in text + assert "违规项: 无" in text + assert "触碰 max_steps: 否" in text + + def test_violations_rendered(self): + text = format_text_report(_metrics(violations=["trajectory reached allowed_max_steps (5)"])) + assert "违规项: trajectory reached allowed_max_steps (5)" in text + + def test_deterministic(self): + m = _metrics(redundant_calls=2, retries=1, max_steps_touched=True) + assert format_text_report(m) == format_text_report(m) + + +# --------------------------------------------------------------------------- +# 7. Golden samples file (schema + registry membership) +# --------------------------------------------------------------------------- +def _repo_tool_names(): + """Authoritative tool names from the five tool modules (lazy import).""" + from src.agent.tools.analysis_tools import ALL_ANALYSIS_TOOLS + from src.agent.tools.backtest_tools import ALL_BACKTEST_TOOLS + from src.agent.tools.data_tools import ALL_DATA_TOOLS + from src.agent.tools.market_tools import ALL_MARKET_TOOLS + from src.agent.tools.search_tools import ALL_SEARCH_TOOLS + + all_tools = ALL_DATA_TOOLS + ALL_ANALYSIS_TOOLS + ALL_SEARCH_TOOLS + ALL_MARKET_TOOLS + ALL_BACKTEST_TOOLS + return {tool_def.name for tool_def in all_tools} + + +class TestGoldenSamplesFile: + def test_samples_load_clean_with_registry_names(self): + samples = load_golden_samples(known_tool_names=_repo_tool_names()) + assert len(samples) == 2 + assert [s.id for s in samples] == ["600519_technical", "000001_core_data_strict"] + + def test_each_sample_passes_structural_validation(self): + for sample in load_golden_samples(): + assert validate_golden_sample(sample, _repo_tool_names()) == [] + + def test_expected_tools_exist_in_repo_registry(self): + known = _repo_tool_names() + for sample in load_golden_samples(): + unknown = [t for t in sample.expected_tools if t not in known] + assert unknown == [], f"sample '{sample.id}' expects unknown tools: {unknown}" + + def test_contains_strict_sample_without_optional_tools(self): + samples = load_golden_samples() + assert any(not s.allow_optional_tools for s in samples) + + def test_samples_have_required_text_fields(self): + for sample in load_golden_samples(): + assert sample.id.strip() + assert sample.task_description.strip() + assert sample.stock_code.strip() + assert sample.allowed_max_steps >= 1 + + def test_empty_stock_code_is_the_valid_no_context_default(self): + sample = GoldenSample(id="x", task_description="t", stock_code="", expected_tools=["get_realtime_quote"]) + assert validate_golden_sample(sample) == [] + + def test_whitespace_only_stock_code_fails_validation(self): + sample = GoldenSample(id="x", task_description="t", stock_code=" ", expected_tools=["get_realtime_quote"]) + issues = validate_golden_sample(sample) + assert any("stock_code must not contain leading or trailing whitespace" in i for i in issues) + + def test_non_string_stock_code_fails_validation(self): + sample = GoldenSample(id="x", task_description="t", stock_code=None, expected_tools=["get_realtime_quote"]) + issues = validate_golden_sample(sample) + assert any("stock_code must be a string" in i for i in issues) + + def test_string_expected_tools_fails_validation(self): + sample = GoldenSample( + id="x", + task_description="t", + stock_code="600519", + expected_tools="get_realtime_quote", + ) + issues = validate_golden_sample(sample) + assert any("must be a list" in i for i in issues) + + def test_non_bool_allow_optional_tools_fails_validation(self): + sample = GoldenSample( + id="x", + task_description="t", + stock_code="600519", + expected_tools=["get_realtime_quote"], + allow_optional_tools="false", + ) + issues = validate_golden_sample(sample) + assert any("allow_optional_tools must be a boolean" in i for i in issues) + + def test_mistyped_fields_fail_validation_not_crash(self): + # Same defect class as the string expected_tools bug: hand-edited JSON + # with mistyped fields must be rejected cleanly, never crash or pass. + bad = GoldenSample( + id=5, + task_description=None, + stock_code="600519", + expected_tools=["get_realtime_quote"], + allowed_max_steps="5", + allow_optional_tools=1, + ) + issues = validate_golden_sample(bad) + assert any("id must be a non-empty string" in i for i in issues) + assert any("task_description must be a non-empty string" in i for i in issues) + assert any("allowed_max_steps must be an integer" in i for i in issues) + assert any("allow_optional_tools must be a boolean" in i for i in issues) + + def test_non_iterable_expected_tools_with_known_names_does_not_crash(self): + sample = GoldenSample( + id="x", + task_description="t", + stock_code="600519", + expected_tools=1, + ) + issues = validate_golden_sample(sample, {"get_realtime_quote"}) + assert any("expected_tools must be a list" in i for i in issues) + + def test_registry_membership_with_one_shot_generator(self): + # The helper accepts any Iterable[str]; a one-shot generator must be + # materialized internally so membership checks never consume it. + sample = GoldenSample( + id="x", + task_description="t", + stock_code="600519", + expected_tools=["b", "a"], + ) + known = (name for name in ["a", "b"]) + assert validate_golden_sample(sample, known) == [] + + def test_duplicate_expected_tools_fail_validation(self): + sample = GoldenSample( + id="x", + task_description="t", + stock_code="600519", + expected_tools=["get_realtime_quote", "get_realtime_quote"], + ) + issues = validate_golden_sample(sample) + assert any("must not contain duplicate names" in i for i in issues) + + def test_loader_materializes_registry_once_for_multiple_samples(self): + # A one-shot generator must survive loading the whole checked-in file: + # the first sample must not exhaust it for the remaining samples. + samples = load_golden_samples(known_tool_names=(name for name in _repo_tool_names())) + assert len(samples) == 2 + + +class TestLoadGoldenSamplesErrors: + @staticmethod + def _write_sample(tmp_path, payload, name="golden.json"): + target = tmp_path / name + target.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8") + return str(target) + + def test_missing_file_raises(self): + with pytest.raises(FileNotFoundError): + load_golden_samples(path="no/such/golden_samples.json") + + def test_non_list_root_raises(self, tmp_path): + path = self._write_sample(tmp_path, {"id": "x"}) + with pytest.raises(ValueError, match="JSON list"): + load_golden_samples(path=path) + + def test_malformed_json_raises(self, tmp_path): + target = tmp_path / "golden.json" + target.write_text("{not json", encoding="utf-8") + with pytest.raises(ValueError): + load_golden_samples(path=str(target)) + + def test_duplicate_ids_raise(self, tmp_path): + sample = { + "id": "dup", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote"], + } + path = self._write_sample(tmp_path, [sample, sample]) + with pytest.raises(ValueError, match="duplicate sample id: dup"): + load_golden_samples(path=path) + + def test_unknown_expected_tool_flagged_when_names_given(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["not_a_real_tool"], + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="unknown expected_tools: not_a_real_tool"): + load_golden_samples(path=path, known_tool_names=_repo_tool_names()) + + def test_unknown_expected_tool_allowed_without_names(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["not_a_real_tool"], + } + path = self._write_sample(tmp_path, [sample]) + assert [s.id for s in load_golden_samples(path=path)] == ["x"] + + def test_extra_json_keys_ignored(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote"], + "notes": "forward-looking metadata", + } + path = self._write_sample(tmp_path, [sample]) + assert load_golden_samples(path=path)[0].id == "x" + + def test_invalid_max_steps_flagged(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote"], + "allowed_max_steps": 0, + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="allowed_max_steps must be >= 1"): + load_golden_samples(path=path) + + def test_whitespace_only_stock_code_raises(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": " ", + "expected_tools": ["get_realtime_quote"], + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="stock_code must not contain leading or trailing whitespace"): + load_golden_samples(path=path) + + def test_string_expected_tools_raises(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": "get_realtime_quote", + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="expected_tools must be a list"): + load_golden_samples(path=path) + + def test_non_list_expected_tools_with_known_names_raises_valueerror(self, tmp_path): + # Regression for OR-COR-4e0e3cf1: the registry membership check must + # not iterate a rejected non-list value and leak a TypeError. + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": 1, + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="expected_tools must be a list"): + load_golden_samples(path=path, known_tool_names={"get_realtime_quote"}) + + def test_unhashable_id_raises_valueerror(self, tmp_path): + # Structural validation must run before duplicate detection: an + # unhashable id would otherwise crash the seen_ids membership check + # with a TypeError instead of the documented ValueError. + sample = { + "id": ["x"], + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote"], + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="id must be a non-empty string"): + load_golden_samples(path=path) + + def test_non_bool_allow_optional_tools_raises(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote"], + "allow_optional_tools": "false", + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="allow_optional_tools must be a boolean"): + load_golden_samples(path=path) + + def test_duplicate_expected_tools_raise(self, tmp_path): + sample = { + "id": "x", + "task_description": "t", + "stock_code": "600519", + "expected_tools": ["get_realtime_quote", "get_realtime_quote"], + } + path = self._write_sample(tmp_path, [sample]) + with pytest.raises(ValueError, match="expected_tools must not contain duplicate names"): + load_golden_samples(path=path) diff --git a/tests/test_agent_trajectory_run_eval.py b/tests/test_agent_trajectory_run_eval.py new file mode 100644 index 000000000..a9fe9beff --- /dev/null +++ b/tests/test_agent_trajectory_run_eval.py @@ -0,0 +1,436 @@ +# -*- coding: utf-8 -*- +"""Unit tests for the runnable agent trajectory eval entry (Issue #1956). + +All tests are offline: ``run_eval._build_executor`` is monkeypatched with a +duck-typed stub, and the three checked-in fixtures provide real-shaped +``tool_calls_log`` payloads covering the positive path, the negative path +(missing expected tool + cached failure + max_steps) and the retry path. +The two runtime-seam guards (multi-arch rejection and golden validation +against the real tool registry) are exercised via monkeypatched config / +registry access plus one lazy real-registry check. +""" + +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from evals.agent_trajectory import run_eval +from evals.agent_trajectory.metrics import GoldenSample, load_golden_samples + +FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures" / "agent_trajectory" + + +def _stub_executor(log, total_steps=None): + """Duck-typed executor recording its run calls (no src/ import).""" + + class _Stub: + def __init__(self): + self.calls = [] + + def run(self, task, context=None): + self.calls.append((task, context)) + return SimpleNamespace(tool_calls_log=list(log), total_steps=total_steps) + + return _Stub() + + +def _fixture(name): + """Load a checked-in fixture payload: (tool_calls_log, total_steps).""" + payload = json.loads((FIXTURES_DIR / f"{name}.json").read_text(encoding="utf-8")) + return payload["tool_calls_log"], payload["total_steps"] + + +def _result_executor(result): + """Duck-typed executor whose run always returns a fixed result object.""" + return SimpleNamespace(run=lambda task, context=None: result) + + +def _goldens(): + return {sample.id: sample for sample in load_golden_samples()} + + +# --------------------------------------------------------------------------- +# 1. run_sample: positive path +# --------------------------------------------------------------------------- +class TestRunSamplePositive: + def test_full_hit_and_clean_metrics(self): + log, total_steps = _fixture("positive_600519_technical") + executor = _stub_executor(log, total_steps) + m = run_eval.run_sample(executor, _goldens()["600519_technical"]) + assert m.expected_hit_rate == 1.0 + assert m.missing_expected == [] + assert m.optional_tools_used == ["search_stock_news"] + assert m.distinct_steps == 5 + assert m.violations == [] + + def test_json_report_contract(self, tmp_path): + log, total_steps = _fixture("positive_600519_technical") + out = tmp_path / "report.json" + m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"], json_out=out) + report = json.loads(out.read_text(encoding="utf-8")) + assert report["sample_id"] == "600519_technical" + assert report["task_description"] + assert report["stock_code"] == "600519" + assert report["violations"] == m.violations + metrics = report["metrics"] + expected_fields = { + "expected_hit_rate", + "expected_total", + "missing_expected", + "optional_tools_used", + "redundant_calls", + "cached_calls", + "failed_calls", + "retries", + "distinct_steps", + "max_steps_touched", + "violations", + } + assert set(metrics) == expected_fields + assert metrics["expected_hit_rate"] == 1.0 + + def test_text_summary_printed(self, capsys): + log, total_steps = _fixture("positive_600519_technical") + run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"]) + out = capsys.readouterr().out + assert "Agent Trajectory 评估报告" in out + assert "3/3" in out + assert "违规项: 无" in out + + def test_context_carries_stock_code(self): + executor = _stub_executor([], total_steps=1) + run_eval.run_sample(executor, _goldens()["600519_technical"]) + assert len(executor.calls) == 1 + task, context = executor.calls[0] + assert context == {"stock_code": "600519"} + assert "600519" in task + + def test_empty_stock_code_passes_no_context(self): + executor = _stub_executor([], total_steps=1) + sample = GoldenSample( + id="no_context", + task_description="t", + stock_code="", + expected_tools=["get_realtime_quote"], + ) + run_eval.run_sample(executor, sample) + _, context = executor.calls[0] + assert context is None + + +# --------------------------------------------------------------------------- +# 2. run_sample: negative path (violations are findings, not exceptions) +# --------------------------------------------------------------------------- +class TestRunSampleNegative: + def test_missing_tool_and_max_steps_violations(self): + log, total_steps = _fixture("negative_600519_technical") + m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"]) + assert m.expected_hit_rate == pytest.approx(2 / 3) + assert m.missing_expected == ["analyze_trend"] + assert m.redundant_calls == 1 + assert m.cached_calls == 1 + assert m.failed_calls == 1 + assert m.max_steps_touched is True + assert "trajectory reached allowed_max_steps (8)" in m.violations + + def test_violations_do_not_raise(self): + log, total_steps = _fixture("negative_600519_technical") + m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["600519_technical"]) + assert m.violations # the eval is a reporter, not a gate + + def test_executor_without_total_steps_still_scores(self): + # A result without total_steps keeps the log-only step behaviour. + log, _ = _fixture("negative_600519_technical") + m = run_eval.run_sample(_stub_executor(log, None), _goldens()["600519_technical"]) + assert m.distinct_steps == 4 + assert m.max_steps_touched is False + + +# --------------------------------------------------------------------------- +# 3. run_sample: retry path (fail -> success -> success contract) +# --------------------------------------------------------------------------- +class TestRunSampleRetry: + def test_fail_success_success_counts_one_retry(self): + log, total_steps = _fixture("retry_000001_core_data_strict") + m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()["000001_core_data_strict"]) + assert m.expected_hit_rate == 1.0 + assert m.retries == 1 + assert m.redundant_calls == 2 + assert m.failed_calls == 1 + assert m.cached_calls == 0 + assert m.optional_tools_used == [] + assert m.violations == [] + + +# --------------------------------------------------------------------------- +# 3.5 run failure: results carrying an explicit success=False +# --------------------------------------------------------------------------- +class TestRunFailure: + def test_unsuccessful_result_raises_before_scoring(self, capsys): + # Owner repro: a real AgentResult with success=False must not be scored. + executor = _result_executor(SimpleNamespace(success=False, error="boom", tool_calls_log=[], total_steps=0)) + with pytest.raises(RuntimeError, match="boom"): + run_eval.run_sample(executor, _goldens()["600519_technical"]) + assert "Agent Trajectory 评估报告" not in capsys.readouterr().out + + def test_unsuccessful_result_without_error_raises(self): + executor = _result_executor(SimpleNamespace(success=False, tool_calls_log=[], total_steps=0)) + with pytest.raises(RuntimeError, match="success=false"): + run_eval.run_sample(executor, _goldens()["600519_technical"]) + + def test_explicit_success_true_scores_normally(self): + log, total_steps = _fixture("positive_600519_technical") + result = SimpleNamespace(success=True, tool_calls_log=list(log), total_steps=total_steps) + m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"]) + assert m.expected_hit_rate == 1.0 + + def test_missing_success_field_is_tolerated(self): + # Duck-typed stubs without a success attribute keep the old behaviour. + log, total_steps = _fixture("positive_600519_technical") + result = SimpleNamespace(tool_calls_log=list(log), total_steps=total_steps) + m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"]) + assert m.expected_hit_rate == 1.0 + + def test_success_none_is_tolerated(self): + log, total_steps = _fixture("positive_600519_technical") + result = SimpleNamespace(success=None, tool_calls_log=list(log), total_steps=total_steps) + m = run_eval.run_sample(_result_executor(result), _goldens()["600519_technical"]) + assert m.expected_hit_rate == 1.0 + + +# --------------------------------------------------------------------------- +# 4. runtime-seam guards: multi-arch rejection + golden registry validation +# --------------------------------------------------------------------------- +class TestArchGuard: + def test_multi_arch_raises(self, monkeypatch): + import src.config + + monkeypatch.setattr(src.config, "get_config", lambda: SimpleNamespace(agent_arch="multi")) + with pytest.raises(RuntimeError, match="multi"): + run_eval._check_agent_arch() + + def test_single_arch_passes(self, monkeypatch): + import src.config + + monkeypatch.setattr(src.config, "get_config", lambda: SimpleNamespace(agent_arch="single")) + run_eval._check_agent_arch() # no raise + + +class TestGoldenRegistryValidation: + def test_unknown_expected_tools_exit_one(self, tmp_path, monkeypatch, capsys): + # A misspelled expected tool must fail as an invalid sample, not score low. + custom = [ + { + "id": "typo_sample", + "task_description": "t", + "stock_code": "", + "expected_tools": ["get_daily_histroy"], + "allowed_max_steps": 10, + "allow_optional_tools": True, + } + ] + golden_path = tmp_path / "golden.json" + golden_path.write_text(json.dumps(custom, ensure_ascii=False), encoding="utf-8") + monkeypatch.setattr(run_eval, "_known_tool_names", lambda: {"get_realtime_quote"}) + assert run_eval.main(["--sample", "typo_sample", "--golden-path", str(golden_path)]) == 1 + err = capsys.readouterr().err + assert "unknown expected_tools" in err + assert "get_daily_histroy" in err + + def test_checked_in_goldens_pass_real_registry(self): + known = run_eval._known_tool_names() + assert len(known) > 5 # sanity: the real registry is non-trivial + load_golden_samples(known_tool_names=known) # must not raise + + def test_registry_load_failure_exit_one(self, monkeypatch, capsys): + def _boom(): + raise ImportError("no tool modules") + + monkeypatch.setattr(run_eval, "_known_tool_names", _boom) + assert run_eval.main(["--sample", "600519_technical"]) == 1 + assert "failed to load tool registry" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# 5. CLI (main): exit codes, selection, JSON output +# --------------------------------------------------------------------------- +class TestMainCli: + @pytest.fixture(autouse=True) + def _stub_builder(self, monkeypatch): + executor = _stub_executor([], total_steps=1) + monkeypatch.setattr(run_eval, "_build_executor", lambda: executor) + return executor + + def test_exit_zero_successful_sample(self, capsys): + assert run_eval.main(["--sample", "600519_technical"]) == 0 + out = capsys.readouterr().out + assert "[eval] sample: 600519_technical" in out + assert "Agent Trajectory 评估报告" in out + + def test_unknown_sample_id_exit_one(self, capsys): + assert run_eval.main(["--sample", "no-such-id"]) == 1 + err = capsys.readouterr().err + assert "unknown sample id 'no-such-id'" in err + assert "600519_technical" in err and "000001_core_data_strict" in err + + def test_missing_arguments_exit_two(self): + with pytest.raises(SystemExit) as excinfo: + run_eval.main([]) + assert excinfo.value.code == 2 + + def test_both_flags_exit_two(self): + with pytest.raises(SystemExit) as excinfo: + run_eval.main(["--sample", "600519_technical", "--all"]) + assert excinfo.value.code == 2 + + def test_build_failure_exit_one(self, monkeypatch, capsys): + def _boom(): + raise RuntimeError("no api key configured") + + monkeypatch.setattr(run_eval, "_build_executor", _boom) + assert run_eval.main(["--sample", "600519_technical"]) == 1 + assert "failed to build agent executor" in capsys.readouterr().err + + def test_multi_arch_build_rejection_exit_one(self, monkeypatch, capsys): + def _boom(): + raise RuntimeError("AGENT_ARCH=multi is not supported by this minimal eval") + + monkeypatch.setattr(run_eval, "_build_executor", _boom) + assert run_eval.main(["--sample", "600519_technical"]) == 1 + assert "multi" in capsys.readouterr().err + + def test_run_failure_exit_one(self, monkeypatch, capsys): + class _Exploding: + def run(self, task, context=None): + raise RuntimeError("llm timeout") + + monkeypatch.setattr(run_eval, "_build_executor", lambda: _Exploding()) + assert run_eval.main(["--sample", "600519_technical"]) == 1 + assert "sample '600519_technical' failed" in capsys.readouterr().err + + def test_unsuccessful_result_exit_one(self, monkeypatch, capsys): + # Owner repro: a result object with success=False is a run failure. + result = SimpleNamespace(success=False, error="boom", tool_calls_log=[], total_steps=0) + monkeypatch.setattr(run_eval, "_build_executor", lambda: _result_executor(result)) + assert run_eval.main(["--sample", "600519_technical"]) == 1 + err = capsys.readouterr().err + assert "sample '600519_technical' failed" in err + assert "boom" in err + + def test_all_returns_one_when_a_sample_result_is_unsuccessful(self, monkeypatch, capsys): + result = SimpleNamespace(success=False, tool_calls_log=[], total_steps=0) + monkeypatch.setattr(run_eval, "_build_executor", lambda: _result_executor(result)) + assert run_eval.main(["--all"]) == 1 + assert "failed" in capsys.readouterr().err + + def test_violations_still_exit_zero(self, monkeypatch, capsys): + log, total_steps = _fixture("negative_600519_technical") + monkeypatch.setattr(run_eval, "_build_executor", lambda: _stub_executor(log, total_steps)) + assert run_eval.main(["--sample", "600519_technical"]) == 0 + assert "trajectory reached allowed_max_steps (8)" in capsys.readouterr().out + + def test_all_writes_keyed_json(self, tmp_path, capsys): + out = tmp_path / "all.json" + assert run_eval.main(["--all", "--json-out", str(out)]) == 0 + report = json.loads(out.read_text(encoding="utf-8")) + assert set(report) == {"600519_technical", "000001_core_data_strict"} + for sample_report in report.values(): + assert set(sample_report) == {"sample_id", "task_description", "stock_code", "metrics", "violations"} + + def test_single_sample_json_out_is_not_keyed(self, tmp_path): + out = tmp_path / "one.json" + assert run_eval.main(["--sample", "600519_technical", "--json-out", str(out)]) == 0 + report = json.loads(out.read_text(encoding="utf-8")) + assert report["sample_id"] == "600519_technical" + + def test_utf8_chinese_roundtrip(self, tmp_path): + out = tmp_path / "utf8.json" + assert run_eval.main(["--sample", "600519_technical", "--json-out", str(out)]) == 0 + text = out.read_text(encoding="utf-8") + assert "贵州茅台" in text + json.loads(text) # valid JSON with raw UTF-8 content + + def test_golden_path_override(self, tmp_path, capsys): + custom = [ + { + "id": "custom_sample", + "task_description": "自定义任务", + "stock_code": "", + "expected_tools": ["get_realtime_quote"], + "allowed_max_steps": 3, + "allow_optional_tools": True, + } + ] + golden_path = tmp_path / "golden.json" + golden_path.write_text(json.dumps(custom, ensure_ascii=False), encoding="utf-8") + assert run_eval.main(["--sample", "custom_sample", "--golden-path", str(golden_path)]) == 0 + assert "[eval] sample: custom_sample" in capsys.readouterr().out + + def test_load_failure_exit_one(self, capsys): + assert run_eval.main(["--sample", "x", "--golden-path", "no/such/file.json"]) == 1 + assert "failed to load golden samples" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# 6. Checked-in fixtures end to end (the owner-requested real-entry coverage) +# --------------------------------------------------------------------------- +@pytest.mark.parametrize( + "fixture_name,golden_id,expected", + [ + ( + "positive_600519_technical", + "600519_technical", + dict( + expected_hit_rate=1.0, + missing_expected=[], + redundant_calls=0, + cached_calls=0, + failed_calls=0, + retries=0, + distinct_steps=5, + max_steps_touched=False, + violations=[], + ), + ), + ( + "negative_600519_technical", + "600519_technical", + dict( + expected_hit_rate=pytest.approx(2 / 3), + missing_expected=["analyze_trend"], + redundant_calls=1, + cached_calls=1, + failed_calls=1, + retries=0, + distinct_steps=8, + max_steps_touched=True, + ), + ), + ( + "retry_000001_core_data_strict", + "000001_core_data_strict", + dict( + expected_hit_rate=1.0, + missing_expected=[], + redundant_calls=2, + cached_calls=0, + failed_calls=1, + retries=1, + distinct_steps=5, + max_steps_touched=False, + violations=[], + ), + ), + ], +) +def test_fixtures_end_to_end(fixture_name, golden_id, expected): + log, total_steps = _fixture(fixture_name) + m = run_eval.run_sample(_stub_executor(log, total_steps), _goldens()[golden_id]) + for field, value in expected.items(): + assert getattr(m, field) == value, (fixture_name, field) + if "violations" in expected: + assert m.violations == expected["violations"] + else: + assert m.violations and "trajectory reached allowed_max_steps (8)" in m.violations