diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 0b374d504..92fb217e4 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -15,6 +15,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - [新功能] 新增按 individual SkillAgent 自身 signal、版本化 engine 与本地已存同源日线窗口计算并持久化 `skill_opinion_outcomes` 的核心服务；本阶段不提供管理员 API、表现统计、样本充足度或权重调整。 +- [修复] 本地 CLI 的 `stdout_preview` / `stderr_preview` 按环境变量、JSON、YAML/日志标量与 URL 的独立契约脱敏短凭证，避免小于 32 字符的 API key、secret 或 token 进入诊断；普通字段仅按敏感名称判定，未加引号的 YAML 敏感标量则 fail-closed 脱敏至行尾（refs #1784）。 +- [修复] `redact_diagnostic_text()` 在 `export SENSITIVE_ENV=$(printenv OTHER_SECRET) session_id=...` 形态下不再因第二遍 `$(...)` 扫描与第一遍敏感赋值替换区重叠而吞掉 `session_id` 等尾随非敏感诊断字段；第二遍扫描现以 first-pass 已替换 span 列表为可信跳过表，并对 prior-head / prior-semicolon 分支的 leading regex 加上 `(?:export[ \t]+)?` 前缀，使 `export FOO=$(...)` 与 `FOO=$(...)` 在所有分支行为对齐（关闭 PR #2118 review blocker OR-COR-7c0a5d41）。 ## [3.28.0] - 2026-07-26 diff --git a/docs/LLM_CONFIG_GUIDE.md b/docs/LLM_CONFIG_GUIDE.md index 391752d01..0484775e9 100644 --- a/docs/LLM_CONFIG_GUIDE.md +++ b/docs/LLM_CONFIG_GUIDE.md @@ -52,6 +52,7 @@ AGENT_GENERATION_BACKEND=auto - 本地 CLI backend 不支持 streaming。请求 stream 时会自动降级为 non-stream，不会因此返回 `capability_unsupported`。 - 本地 CLI usage 通常不可用，系统不会写入 fake 0 token、fake cost 或 fake cache telemetry。 - 本地 CLI 执行上限有硬边界：`GENERATION_BACKEND_TIMEOUT_SECONDS` 最大 `3600`，`GENERATION_BACKEND_MAX_OUTPUT_BYTES` 最大 `33554432`，`GENERATION_BACKEND_MAX_CONCURRENCY` 最大 `16`，`LOCAL_CLI_BACKEND_MAX_CONCURRENCY` 最大 `4`。诊断 stdout/stderr 与最终响应合计超过输出上限时会返回结构化 `output_too_large`；对 `--output-last-message` preset，stdout 中重复打印的最终响应不会重复计入，也不会作为 `stdout_preview` 暴露。 +- 本地 CLI 的 `stdout_preview` / `stderr_preview` 在写入结构化 diagnostics 前会脱敏短凭证赋值，不依赖值长度：大写环境变量赋值沿用 child-env 的 fail-closed 敏感名称判定，JSON 与 YAML / 普通日志中的标量赋值使用更窄的凭证字段 allowlist，URL 继续使用独立的 userinfo / webhook / 敏感参数规则。`token_budget`、`session_id`、`sort_key` 等普通排障字段不会仅因包含 `token`、`session` 或 `key` 子串而被删除；但未加引号的 YAML 敏感标量没有可靠的同行边界，因此从该值起到行尾会 fail-closed 脱敏，同行后续字段也不会保留。 - 本地 CLI 默认并发为 1；有效并发为 `min(LOCAL_CLI_BACKEND_MAX_CONCURRENCY, GENERATION_BACKEND_MAX_CONCURRENCY)`，不继承 `MAX_WORKERS`。 - `AGENT_GENERATION_BACKEND=auto` 不会继承 `GENERATION_BACKEND` 的 local CLI 值；Agent 工具调用继续使用 LiteLLM。Web 设置页仅暴露 `auto|litellm`；手写 `AGENT_GENERATION_BACKEND=codex_cli|claude_code_cli|opencode_cli` 不实现 text-only Agent mode，会返回明确 unsupported tool-calling 诊断。 - Phase 6a 的 DSA Tool Surface 仍是唯一工具 schema、权限元数据、scope guard、结构化错误和审计/脱敏边界；Phase 6 的 Codex AgentBackend 只能通过该 Tool Surface 执行工具。`codex_cli` / `claude_code_cli` / `opencode_cli` 仍是 generation-only，不能作为 Agent tool fallback。 diff --git a/docs/LLM_CONFIG_GUIDE_EN.md b/docs/LLM_CONFIG_GUIDE_EN.md index e7fd3a3d7..eb27c381a 100644 --- a/docs/LLM_CONFIG_GUIDE_EN.md +++ b/docs/LLM_CONFIG_GUIDE_EN.md @@ -45,6 +45,7 @@ AGENT_GENERATION_BACKEND=auto - Local CLI backends do not support streaming. Stream requests degrade to non-stream and do not return `capability_unsupported`. - Local CLI usage is normally unavailable. DSA does not persist fake 0-token, fake cost, or fake cache telemetry. - Local CLI execution has hard caps: `GENERATION_BACKEND_TIMEOUT_SECONDS` max `3600`, `GENERATION_BACKEND_MAX_OUTPUT_BYTES` max `33554432`, `GENERATION_BACKEND_MAX_CONCURRENCY` max `16`, and `LOCAL_CLI_BACKEND_MAX_CONCURRENCY` max `4`. Diagnostic stdout/stderr plus the final response are counted together; for `--output-last-message` presets, the final response duplicated to stdout is not counted twice and is not exposed in `stdout_preview`. +- Local CLI `stdout_preview` / `stderr_preview` redact short credential assignments before diagnostics are emitted, regardless of value length. Uppercase environment assignments reuse the child-env fail-closed sensitive-name contract; scalar assignments in JSON, YAML, and ordinary logs use a narrower credential-field allowlist; URLs keep separate userinfo, webhook, and sensitive-parameter rules. Ordinary troubleshooting fields such as `token_budget`, `session_id`, and `sort_key` are not removed merely because their names contain `token`, `session`, or `key`; however, an unquoted sensitive YAML scalar has no reliable same-line boundary, so redaction fails closed from that value through the end of the line and does not preserve later fields on that line. - Local CLI default concurrency is 1. Effective local CLI concurrency is `min(LOCAL_CLI_BACKEND_MAX_CONCURRENCY, GENERATION_BACKEND_MAX_CONCURRENCY)` and does not inherit `MAX_WORKERS`. - `AGENT_GENERATION_BACKEND=auto` does not inherit local CLI values from `GENERATION_BACKEND`; Agent tool calling remains on LiteLLM. The Web settings page only exposes `auto|litellm`; a hand-written `AGENT_GENERATION_BACKEND=codex_cli|claude_code_cli|opencode_cli` does not enable Agent text-only mode and returns an explicit unsupported tool-calling diagnostic. - The Phase 6a DSA Tool Surface remains the only tool-schema, permission, scope-guard, structured-error, audit, and redaction boundary. The Phase 6 Codex AgentBackend can execute tools only through this surface. `codex_cli` / `claude_code_cli` / `opencode_cli` remain generation-only and are never Agent tool fallbacks. diff --git a/src/llm/local_cli_backend.py b/src/llm/local_cli_backend.py index 2a434e518..76e908c1b 100644 --- a/src/llm/local_cli_backend.py +++ b/src/llm/local_cli_backend.py @@ -12,6 +12,7 @@ from __future__ import annotations from dataclasses import dataclass import hashlib from contextlib import ExitStack, contextmanager +from functools import lru_cache import json import os from pathlib import Path @@ -22,7 +23,7 @@ import subprocess import tempfile import threading import time -from typing import Any, Callable, Dict, Iterator, Mapping, Optional, Sequence +from typing import Any, Callable, Dict, Iterator, Mapping, Optional, Sequence, Tuple from urllib.parse import parse_qsl, urlsplit from src.llm.backend_registry import ( @@ -53,6 +54,19 @@ _FINAL_MESSAGE_OMITTED_PREVIEW = "" _STDOUT_PREVIEW_OMITTED = "" _PROCESS_POLL_INTERVAL_SECONDS = 0.05 _URL_PATTERN = re.compile(r"https?://[^\s,;)\]}]+", re.IGNORECASE) +_ANSI_ESCAPE_PATTERN = re.compile( + r""" + \x1B + (?: + \[[0-?]*[ -/]*[@-~] + | + \][^\x07\x1B]*(?:\x07|\x1B\\) + | + [@-_] + ) + """, + re.VERBOSE, +) _SHELL_META_CHARS = ("|", ">", "<", ";", "`") _SHELL_META_STRINGS = ("&&", "||", "$(") _UNSUPPORTED_ARG_MARKERS = ( @@ -133,6 +147,280 @@ _SENSITIVE_ENV_PATTERNS = ( "VERTEX_", "WEBHOOK", ) +_SENSITIVE_ENV_EXACT_NAMES = frozenset({ + "AIHUBMIX_KEY", + "DINGTALK_APP_KEY", + "LONGBRIDGE_APP_KEY", + "PUSHOVER_USER_KEY", + "WECOM_ENCODING_AES_KEY", +}) +_SENSITIVE_DIAGNOSTIC_FIELDS = frozenset({ + "access_token", + "access_key", + "access_key_id", + "api_key", + "api_keys", + "apikey", + "api_secret", + "app_secret", + "auth_token", + "authorization", + "client_secret", + "cookie", + "credential", + "credentials", + "csrf_token", + "database_url", + "db_url", + "github_token", + "id_token", + "encryption_key", + "password", + "passwd", + "private_key", + "proxy_authorization", + "refresh_token", + "secret", + "secret_access_key", + "secret_key", + "sendkey", + "set_cookie", + "session_secret", + "signing_key", + "token", + "tushare_token", + "verification_token", + "webhook", + "webhook_url", +}) +_SENSITIVE_DIAGNOSTIC_FIELD_SUFFIXES = ( + "_access_key", + "_access_key_id", + "_access_token", + "_api_key", + "_api_keys", + "_api_secret", + "_app_secret", + "_auth_token", + "_client_secret", + "_credential", + "_credentials", + "_database_url", + "_db_url", + "_encryption_key", + "_password", + "_passwd", + "_private_key", + "_secret", + "_secret_access_key", + "_secret_key", + "_sendkey", + "_session_secret", + "_signing_key", + "_token", + "_webhook", + "_webhook_url", +) +_DIGEST_AUTH_PARAM_NAMES = frozenset({ + "algorithm", + "charset", + "cnonce", + "nc", + "nonce", + "opaque", + "qop", + "realm", + "response", + "uri", + "userhash", + "username", +}) +_DIAGNOSTIC_ASSIGNMENT_VALUE_PATTERN = r""" + (?P + (?: + "(?:\\.|[^"\\])*" + | + '(?:''|\\.|[^'\\])*' + | + \\\r?\n[ \t]* + | + \\[^\r\n] + | + [^\s,;}\]"'] + )+ + ) +""" + + +@lru_cache(maxsize=1) +def _diagnostic_field_name_pattern() -> str: + """Build field syntax after the lazy config registry can be imported safely.""" + + sensitive_names = ( + {name.upper() for name in _SENSITIVE_DIAGNOSTIC_FIELDS} + | _SENSITIVE_ENV_EXACT_NAMES + | _registered_sensitive_env_exact_names() + ) + title_patterns = sorted( + (re.escape(title) for title in _registered_sensitive_field_titles()), + key=len, + reverse=True, + ) + spaced_sensitive_names = sorted( + ( + re.escape(name).replace("_", r"[ \t]+") + for name in sensitive_names + if "_" in name + ), + key=len, + reverse=True, + ) + explicit_sensitive_names = "|".join(title_patterns + spaced_sensitive_names) + return ( + rf"(?:(?i:{explicit_sensitive_names})|" + r"[A-Za-z][A-Za-z0-9_-]*)" + ) + + +@lru_cache(maxsize=1) +def _diagnostic_field_assignment_pattern() -> re.Pattern[str]: + return re.compile( + rf""" + (?'?) + (?P{_diagnostic_field_name_pattern()}) + (?P=name_quote) + (?P[ \t]*(?:=|:)[ \t]*) + {_DIAGNOSTIC_ASSIGNMENT_VALUE_PATTERN} + """, + re.VERBOSE, + ) + + +@lru_cache(maxsize=1) +def _diagnostic_json_assignment_pattern() -> re.Pattern[str]: + return re.compile( + rf""" + " + (?P(?:\\.|[^"\\])*) + " + (?P[ \t\r\n]*:[ \t\r\n]*) + {_DIAGNOSTIC_ASSIGNMENT_VALUE_PATTERN} + """, + re.VERBOSE, + ) + + +@lru_cache(maxsize=1) +def _diagnostic_line_field_pattern() -> re.Pattern[str]: + field_name_pattern = _diagnostic_field_name_pattern() + return re.compile( + rf""" + (?'?) + (?P{field_name_pattern}) + (?P=name_quote) + (?P[ \t]*(?:=|:)[ \t]*) + (?P[^\r\n]*?) + (?= + (?:(?:[,;][ \t]*)|[ \t]+)'?{field_name_pattern}'?[ \t]*(?:=|:)[ \t]* + | + \r?\n? + $ + ) + """, + re.VERBOSE, + ) + + +@lru_cache(maxsize=1) +def _diagnostic_double_quoted_yaml_line_pattern() -> re.Pattern[str]: + return re.compile( + r""" + ^ + (?P[ ]*) + (?P-[ \t]+)? + (?P(?:(?:!(?:<[^>\r\n]+>|[^ \t\r\n]*)?|&[^ \t\r\n]+)[ \t]+)*) + " + (?P(?:\\.|[^"\\])*) + " + (?P[ \t]*:[ \t]*) + (?P[^\r\n]*) + (?P\r?\n?) + $ + """, + re.VERBOSE, + ) + + +@lru_cache(maxsize=1) +def _diagnostic_yaml_explicit_key_pattern() -> re.Pattern[str]: + return re.compile( + rf""" + ^ + (?P[ ]*) + (?P-[ \t]+)? + \?[ \t]+ + (?P(?:(?:!(?:<[^>\r\n]+>|[^ \t\r\n]*)?|&[^ \t\r\n]+)[ \t]+)*) + (?P + "(?:\\.|[^"\\])*" + | + '(?:''|[^'])*' + | + [^\r\n#]+? + ) + [ \t]*(?:\#.*)? + \r?\n? + $ + """, + re.VERBOSE, + ) + + +_DIAGNOSTIC_ENV_ASSIGNMENT_PATTERN = re.compile( + rf""" + (?(?:export[ \t]+)?) + (?P[A-Z][A-Z0-9_]*) + (?P[ \t]*\+?=[ \t]*) + {_DIAGNOSTIC_ASSIGNMENT_VALUE_PATTERN} + """, + re.VERBOSE, +) +_DIAGNOSTIC_ENV_ASSIGNMENT_PREFIX_PATTERN = re.compile( + r""" + (?(?:export[ \t]+)?) + (?P[A-Z][A-Z0-9_]*) + (?P[ \t]*\+?=[ \t]*) + """, + re.VERBOSE, +) +_AUTHORIZATION_FIELD_PATTERN = re.compile( + r""" + (? + (?: + (?P["']) + (?:(?:proxy[-_ \t]?)?authorization) + (?P=quote) + | + (?:proxy[-_ \t]?)?authorization + ) + [ \t]*(?:=|:)[ \t]* + ) + (?P[^\r\n]*?) + (?= + [ \t]+["']?(?:proxy[-_ \t]?)?authorization["']?[ \t]*(?:=|:) + | + \r?\n + | + $ + ) + """, + re.IGNORECASE | re.VERBOSE, +) +_YAML_BLOCK_SCALAR_PATTERN = re.compile(r"^[|>][0-9+-]*$") _CLAUDE_CODE_STATIC_INSTRUCTION = ( "Generate the requested DSA analysis output from stdin. " "Return only the final response content. Do not call tools, read files, " @@ -350,17 +638,1081 @@ def _popen_session_kwargs() -> Dict[str, Any]: return {"start_new_session": True} -def redact_diagnostic_text(text: str, *, home: Optional[str] = None, limit: int = _PREVIEW_LIMIT) -> str: - """Redact sensitive diagnostics and return a bounded preview.""" +def _redact_assignment_value(match: re.Match[str]) -> str: + """Replace one parsed assignment value while preserving its surrounding syntax.""" - redacted = text or "" + original = match.group(0) + value = match.group("value") + replacement = "" + if len(value) >= 2 and value[0] in {"'", '"'} and value[-1] == value[0]: + replacement = f"{value[0]}{value[0]}" + value_start = match.start("value") - match.start() + value_end = match.end("value") - match.start() + return f"{original[:value_start]}{replacement}{original[value_end:]}" + + +def _is_field_specific_sensitive_redaction_target(name: str) -> bool: + normalized_name = _normalize_diagnostic_field_name(name) + return ( + _is_sensitive_structured_assignment_name(name) + and normalized_name not in {"authorization", "proxy_authorization"} + ) + + +def _is_multiline_sensitive_redaction_target(name: str) -> bool: + return _is_sensitive_structured_assignment_name(name) + + +def _normalize_diagnostic_field_name(name: str) -> str: + normalized = re.sub(r"[- \t]+", "_", str(name or "")) + normalized = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1_\2", normalized) + normalized = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", normalized) + return normalized.lower() + + +def _decode_diagnostic_double_quoted_field_name(name: str) -> str: + """Decode YAML/JSON double-quoted escapes before applying name matching.""" + + if "\\" not in name: + return name + + simple_escapes = { + "0": "\0", + "a": "\a", + "b": "\b", + "t": "\t", + "\t": "\t", + "n": "\n", + "v": "\v", + "f": "\f", + "r": "\r", + "e": "\x1b", + " ": " ", + '"': '"', + "/": "/", + "\\": "\\", + "N": "\x85", + "_": "\xa0", + "L": "\u2028", + "P": "\u2029", + } + decoded = [] + index = 0 + while index < len(name): + char = name[index] + if char != "\\": + decoded.append(char) + index += 1 + continue + if index + 1 >= len(name): + return name + escape = name[index + 1] + if escape in simple_escapes: + decoded.append(simple_escapes[escape]) + index += 2 + continue + if escape in {"x", "u", "U"}: + widths = {"x": 2, "u": 4, "U": 8} + width = widths[escape] + digits = name[index + 2:index + 2 + width] + if len(digits) != width or not re.fullmatch(r"[0-9A-Fa-f]+", digits): + return name + try: + decoded.append(chr(int(digits, 16))) + except ValueError: + return name + index += 2 + width + continue + if escape in {"\n", "\r"}: + index += 2 + if escape == "\r" and index < len(name) and name[index] == "\n": + index += 1 + while index < len(name) and name[index] in {" ", "\t"}: + index += 1 + continue + return name + return "".join(decoded) + + +def _decode_diagnostic_yaml_field_name(name_token: str) -> str: + """Decode a YAML key token to its logical field name.""" + + token = str(name_token or "") + if len(token) >= 2 and token[0] == token[-1] == '"': + return _decode_diagnostic_double_quoted_field_name(token[1:-1]) + if len(token) >= 2 and token[0] == token[-1] == "'": + return token[1:-1].replace("''", "'") + return token.strip() + + +def _is_sensitive_diagnostic_field_name(name: str) -> bool: + normalized = _normalize_diagnostic_field_name(name) + return ( + normalized in _SENSITIVE_DIAGNOSTIC_FIELDS + or any(normalized.endswith(suffix) for suffix in _SENSITIVE_DIAGNOSTIC_FIELD_SUFFIXES) + ) + + +@lru_cache(maxsize=1) +def _sensitive_exact_diagnostic_field_names() -> frozenset[str]: + return _SENSITIVE_ENV_EXACT_NAMES | _registered_sensitive_env_exact_names() + + +@lru_cache(maxsize=1) +def _registered_sensitive_field_titles() -> frozenset[str]: + try: + from src.core.config_registry import _FIELD_DEFINITIONS + except Exception: + return frozenset() + + return frozenset( + str(metadata.get("title")) + for metadata in _FIELD_DEFINITIONS.values() + if isinstance(metadata, Mapping) + and metadata.get("is_sensitive") + and isinstance(metadata.get("title"), str) + and metadata.get("title") + ) + + +def _compact_diagnostic_name(name: str) -> str: + return re.sub(r"[^A-Za-z0-9]+", "", str(name or "")).upper() + + +@lru_cache(maxsize=1) +def _compact_sensitive_exact_diagnostic_field_names() -> frozenset[str]: + return frozenset( + _compact_diagnostic_name(name) + for name in _sensitive_exact_diagnostic_field_names() + ) + + +@lru_cache(maxsize=1) +def _sensitive_registered_diagnostic_field_titles() -> frozenset[str]: + return frozenset(title.upper() for title in _registered_sensitive_field_titles()) + + +@lru_cache(maxsize=1) +def _compact_sensitive_registered_diagnostic_field_titles() -> frozenset[str]: + return frozenset( + _compact_diagnostic_name(title) for title in _registered_sensitive_field_titles() + ) + + +def _is_sensitive_structured_assignment_name(name: str) -> bool: + exact_name = _normalize_diagnostic_field_name(name).upper() + upper_name = str(name or "").upper() + compact_name = _compact_diagnostic_name(name) + return ( + _is_sensitive_diagnostic_field_name(name) + or exact_name in _sensitive_exact_diagnostic_field_names() + or upper_name in _sensitive_registered_diagnostic_field_titles() + or compact_name in _compact_sensitive_exact_diagnostic_field_names() + or compact_name in _compact_sensitive_registered_diagnostic_field_titles() + ) + + +def _is_registered_sensitive_field_title(name: str) -> bool: + return str(name or "").upper() in _sensitive_registered_diagnostic_field_titles() + + +def _leading_space_count(text: str) -> int: + return len(text) - len(text.lstrip(" ")) + + +def _yaml_value_without_node_properties(value: str) -> str: + """Return a YAML value with leading tags and anchors removed.""" + + remaining = value.strip() + while remaining.startswith(("!", "&")): + if remaining.startswith("&"): + property_match = re.match(r"&[^ \t]+(?:[ \t]+|$)", remaining) + else: + property_match = re.match(r"!(?:<[^>]+>|[^ \t]*)?(?:[ \t]+|$)", remaining) + if property_match is None: + break + remaining = remaining[property_match.end():].lstrip() + return remaining + + +def _is_yaml_block_value(value: str) -> bool: + """Return whether a YAML value introduces content on following lines.""" + + stripped = _yaml_value_without_node_properties(value) + if not stripped or stripped.startswith("#"): + return True + + tokens = stripped.split() + if not tokens or tokens[0].startswith("#"): + return True + if not _YAML_BLOCK_SCALAR_PATTERN.match(tokens[0]): + return False + return len(tokens) == 1 or tokens[1].startswith("#") + + +def _yaml_value_allows_indentless_sequence(value: str) -> bool: + """Return whether a following same-indent sequence belongs to this value.""" + + stripped = _yaml_value_without_node_properties(value) + return not stripped or stripped.startswith("#") + + +def _replace_spans(text: str, replacements: Sequence[Tuple[int, int, str]]) -> str: + updated = text + for start, end, replacement in sorted(replacements, reverse=True): + updated = f"{updated[:start]}{replacement}{updated[end:]}" + return updated + + +def _consume_redacted_yaml_block_lines( + lines: Sequence[str], + *, + start_index: int, + base_indent: int, + allows_indentless_sequence: bool, +) -> tuple[list[str], int]: + kept_lines: list[str] = [] + index = start_index + while index < len(lines): + next_line = lines[index] + stripped_next_line = next_line.strip() + next_indent = _leading_space_count(next_line) + if stripped_next_line and next_indent <= base_indent: + is_same_indent_comment = ( + next_indent == base_indent and stripped_next_line.startswith("#") + ) + is_indentless_sequence = ( + allows_indentless_sequence + and next_indent == base_indent + and ( + stripped_next_line == "-" + or stripped_next_line.startswith("- ") + ) + ) + if not is_same_indent_comment and not is_indentless_sequence: + break + if not stripped_next_line: + kept_lines.append(next_line) + index += 1 + return kept_lines, index + + +def _consume_redacted_multiline_quote_lines( + lines: Sequence[str], + *, + start_index: int, + multiline_quote: str, +) -> tuple[list[str], int]: + kept_lines: list[str] = [] + index = start_index + while index < len(lines): + next_line = lines[index] + close_index = _diagnostic_quote_close_index(next_line, multiline_quote, start=0) + if close_index is None: + index += 1 + continue + trailing = next_line[close_index:] + if trailing.strip(): + kept_lines.append(trailing) + index += 1 + break + return kept_lines, index + + +def _is_inside_diagnostic_flow_collection(text: str) -> bool: + closing_by_opening = {"{": "}", "[": "]"} + stack: list[str] = [] + index = 0 + while index < len(text): + char = text[index] + if char in {"'", '"'}: + index = _consume_diagnostic_scalar(text, index) + continue + if char in closing_by_opening: + stack.append(closing_by_opening[char]) + index += 1 + continue + if stack and char == stack[-1]: + stack.pop() + index += 1 + return bool(stack) + + +def _redact_double_quoted_yaml_sensitive_field( + lines: Sequence[str], + index: int, +) -> Optional[tuple[list[str], int]]: + line = lines[index] + match = _diagnostic_double_quoted_yaml_line_pattern().match(line) + if match is None: + return None + + decoded_name = _decode_diagnostic_double_quoted_field_name(match.group("name")) + if not _is_multiline_sensitive_redaction_target(decoded_name): + return None + if _is_inside_diagnostic_flow_collection("".join(lines[:index])): + return None + + value = match.group("value") + normalized_name = _normalize_diagnostic_field_name(decoded_name) + if ( + normalized_name in {"authorization", "proxy_authorization"} + and value.lstrip(" \t").startswith("") + ): + return None + stripped_value = value.strip() + yaml_scalar_value = _yaml_value_without_node_properties(value) + base_indent = _leading_space_count(line) + value_span = (match.start("value"), match.end("value")) + + if _is_yaml_block_value(value): + replacement = "" + if not match.group("separator")[-1:].isspace(): + replacement = f" {replacement}" + kept_lines, next_index = _consume_redacted_yaml_block_lines( + lines, + start_index=index + 1, + base_indent=base_indent, + allows_indentless_sequence=_yaml_value_allows_indentless_sequence(value), + ) + return ( + [_replace_spans(line, [(value_span[0], value_span[1], replacement)]), *kept_lines], + next_index, + ) + + yaml_quote = yaml_scalar_value[:1] + yaml_quote_is_closed = bool( + yaml_quote and _has_closed_diagnostic_quote(yaml_scalar_value, yaml_quote) + ) + if yaml_quote in {"'", '"'} and ( + yaml_scalar_value != stripped_value or not yaml_quote_is_closed + ): + kept_lines = [_replace_spans(line, [(value_span[0], value_span[1], "")])] + next_index = index + 1 + if not yaml_quote_is_closed: + trailing_lines, next_index = _consume_redacted_multiline_quote_lines( + lines, + start_index=next_index, + multiline_quote=yaml_quote, + ) + kept_lines.extend(trailing_lines) + return kept_lines, next_index + + if stripped_value and stripped_value[0] not in {"'", '"', "{", "["}: + redacted_line = _replace_spans(line, [(value_span[0], value_span[1], "")]) + kept_lines = [redacted_line] + continuation_index = index + 1 + while continuation_index < len(lines): + next_line = lines[continuation_index] + if not next_line.strip(): + kept_lines.append(next_line) + continuation_index += 1 + continue + if _leading_space_count(next_line) <= base_indent: + break + continuation_index += 1 + return kept_lines, continuation_index + + return None + + +def _redact_sensitive_collection_assignments(text: str) -> str: + def replace_matches( + source: str, + pattern: re.Pattern[str], + is_sensitive_name: Callable[[str], bool], + ) -> str: + replacements = [] + last_end = -1 + for match in pattern.finditer(source): + name = match.group("name") + value = match.group("value") + if not is_sensitive_name(name) or not value or value[0] not in "{[": + continue + value_start = match.start("value") + if value_start < last_end: + continue + value_end = _consume_diagnostic_collection(source, value_start) + replacements.append((value_start, value_end, "")) + last_end = value_end + return _replace_spans(source, replacements) + + redacted = replace_matches(text, _DIAGNOSTIC_ENV_ASSIGNMENT_PATTERN, _is_sensitive_env_name) + redacted = replace_matches( + redacted, + _diagnostic_json_assignment_pattern(), + lambda name: _is_sensitive_structured_assignment_name( + _decode_diagnostic_double_quoted_field_name(name) + ), + ) + return replace_matches( + redacted, + _diagnostic_field_assignment_pattern(), + _is_field_specific_sensitive_redaction_target, + ) + + +def _redact_multiline_sensitive_fields(text: str) -> str: + """Redact YAML/log scalar fields that span spaces or indented block lines.""" + + lines = text.splitlines(keepends=True) + if not lines: + return text + + redacted_lines = [] + index = 0 + while index < len(lines): + line = lines[index] + quoted_yaml_redaction = _redact_double_quoted_yaml_sensitive_field(lines, index) + if quoted_yaml_redaction is not None: + kept_lines, index = quoted_yaml_redaction + redacted_lines.extend(kept_lines) + continue + + matches = list(_diagnostic_line_field_pattern().finditer(line)) + if not matches: + redacted_lines.append(line) + index += 1 + continue + + replacements = [] + block_match: Optional[re.Match[str]] = None + block_allows_indentless_sequence = False + multiline_quote: Optional[str] = None + for match in matches: + name = match.group("name") + value = match.group("value") + normalized_name = _normalize_diagnostic_field_name(name) + is_redacted_authorization = ( + normalized_name in {"authorization", "proxy_authorization"} + and value.strip() == "" + ) + is_authorization_yaml_block = ( + normalized_name in {"authorization", "proxy_authorization"} + and ":" in match.group("separator") + and _is_yaml_block_value(value) + ) + if ( + not _is_field_specific_sensitive_redaction_target(name) + and not is_redacted_authorization + and not is_authorization_yaml_block + ): + continue + + stripped_value = value.strip() + yaml_scalar_value = _yaml_value_without_node_properties(value) + if ":" in match.group("separator") and _is_yaml_block_value(value): + replacement = "" + if not match.group("separator")[-1:].isspace(): + replacement = f" {replacement}" + replacements.append((match.start("value"), match.end("value"), replacement)) + block_match = block_match or match + block_allows_indentless_sequence = ( + block_allows_indentless_sequence + or _yaml_value_allows_indentless_sequence(value) + ) + continue + + yaml_quote = yaml_scalar_value[:1] + yaml_quote_is_closed = bool( + yaml_quote + and _has_closed_diagnostic_quote(yaml_scalar_value, yaml_quote) + ) + if yaml_quote in {"'", '"'} and ( + yaml_scalar_value != stripped_value or not yaml_quote_is_closed + ): + replacements.append((match.start("value"), match.end("value"), "")) + if not yaml_quote_is_closed: + multiline_quote = multiline_quote or yaml_quote + continue + + if stripped_value and stripped_value[0] not in {"'", '"', "{", "["}: + value_end = match.end("value") + if ( + ":" in match.group("separator") + and stripped_value != "" + and not _is_registered_sensitive_field_title(name) + ): + flow_value_end = _find_diagnostic_flow_scalar_end(line, match.start("value")) + if flow_value_end is not None: + value_end = flow_value_end + else: + # Outside YAML flow collections, an unquoted scalar has + # no reliable same-line boundary. Fail closed instead + # of treating assignment-like text inside the + # credential as a separate diagnostic field. + value_end = len(line.rstrip("\r\n")) + replacements.append((match.start("value"), value_end, "")) + if ":" in match.group("separator") and value_end == len(line.rstrip("\r\n")): + block_match = block_match or match + if value_end == len(line.rstrip("\r\n")): + break + + if not replacements: + redacted_lines.append(line) + index += 1 + continue + + redacted_lines.append(_replace_spans(line, replacements)) + index += 1 + + if block_match is not None: + base_indent = _leading_space_count(line) + kept_lines, index = _consume_redacted_yaml_block_lines( + lines, + start_index=index, + base_indent=base_indent, + allows_indentless_sequence=block_allows_indentless_sequence, + ) + redacted_lines.extend(kept_lines) + continue + + if multiline_quote is None: + continue + + kept_lines, index = _consume_redacted_multiline_quote_lines( + lines, + start_index=index, + multiline_quote=multiline_quote, + ) + redacted_lines.extend(kept_lines) + + return "".join(redacted_lines) + + +def _redact_yaml_explicit_sensitive_fields(text: str) -> str: + """Redact YAML ``? key`` / ``: value`` entries and their continuations.""" + + lines = text.splitlines(keepends=True) + redacted_lines = [] + index = 0 + while index < len(lines): + key_line = lines[index] + key_match = _diagnostic_yaml_explicit_key_pattern().match(key_line) + key_name = ( + _decode_diagnostic_yaml_field_name(key_match.group("name_token")) + if key_match is not None + else "" + ) + if ( + key_match is None + or not _is_multiline_sensitive_redaction_target(key_name) + ): + redacted_lines.append(key_line) + index += 1 + continue + + base_indent = len(key_match.group("indent")) + len(key_match.group("sequence_prefix") or "") + value_index = index + 1 + while value_index < len(lines): + candidate_line = lines[value_index] + candidate_stripped = candidate_line.strip() + if not candidate_stripped: + value_index += 1 + continue + if candidate_line[_leading_space_count(candidate_line):].startswith("#"): + value_index += 1 + continue + break + if value_index >= len(lines): + redacted_lines.append(key_line) + index += 1 + continue + + value_line = lines[value_index] + value_indent = _leading_space_count(value_line) + value_content = value_line[value_indent:].rstrip("\r\n") + if ( + value_indent != base_indent + or not value_content.startswith(":") + or ( + len(value_content) > 1 + and value_content[1] not in {" ", "\t"} + ) + ): + redacted_lines.append(key_line) + index += 1 + continue + + newline = ( + "\r\n" + if value_line.endswith("\r\n") + else "\n" + if value_line.endswith("\n") + else "" + ) + value = value_content[1:].lstrip(" \t") + redacted_lines.append(key_line) + for skipped_line in lines[index + 1:value_index]: + if not skipped_line.strip(): + redacted_lines.append(skipped_line) + redacted_lines.append(f"{' ' * value_indent}: {newline}") + index = value_index + 1 + allows_indentless_sequence = _yaml_value_allows_indentless_sequence(value) + + while index < len(lines): + next_line = lines[index] + stripped_next_line = next_line.strip() + next_indent = _leading_space_count(next_line) + if stripped_next_line and next_indent <= base_indent: + is_same_indent_comment = ( + next_indent == base_indent and stripped_next_line.startswith("#") + ) + is_indentless_sequence = ( + allows_indentless_sequence + and next_indent == base_indent + and ( + stripped_next_line == "-" + or stripped_next_line.startswith("- ") + ) + ) + if not is_same_indent_comment and not is_indentless_sequence: + break + if not stripped_next_line: + redacted_lines.append(next_line) + index += 1 + + return "".join(redacted_lines) + + +def _redact_sensitive_diagnostic_assignments(text: str) -> str: + """Redact parsed env and structured-field assignments under separate contracts.""" + + def redact_env(match: re.Match[str]) -> str: + name = match.group("name") + return _redact_assignment_value(match) if _is_sensitive_env_name(name) else match.group(0) + + def redact_structured_field(match: re.Match[str]) -> str: + return ( + _redact_assignment_value(match) + if _is_field_specific_sensitive_redaction_target(match.group("name")) + else match.group(0) + ) + + def redact_sensitive_env_command_substitutions(source: str) -> str: + replacements = [] + last_end = -1 + # Collect all sensitive-env-assignment spans from the first pass + # so the second pass can skip any ``$(...)`` that sits inside one + # of those already-redacted regions. Without this overlap guard + # the second pass re-adds the same span and ``_replace_spans`` + # silently drops trailing diagnostics such as ``session_id`` + # (regression OR-COR-7c0a5d41). + first_pass_spans: list[tuple[int, int]] = [] + for match in _DIAGNOSTIC_ENV_ASSIGNMENT_PREFIX_PATTERN.finditer(source): + value_start = match.end() + if value_start < last_end or source[value_start:value_start + 2] != "$(": + continue + if not _is_sensitive_env_name(match.group("name")): + continue + value_end = _consume_shell_command_substitution(source, value_start) + if value_end <= value_start: + continue + replacements.append((value_start, value_end, "")) + first_pass_spans.append((value_start, value_end)) + last_end = value_end + # Scan remaining $(...) command substitutions not bound to any env + # assignment, so multi-segment diagnostics like + # A=$(echo X);B=ok; tail $(printenv SECRET_TOKEN) + # redact every sensitive reference regardless of how it is invoked. + tail_start = 0 + while True: + sub = source.find("$(", tail_start) + if sub == -1: + break + if sub > 0 and source[sub - 1] == "$": + tail_start = sub + 1 + continue + # Skip $( that is the direct value of a sensitive env assignment + # already handled above, so we don't double-rewrite it. A leading + # non-sensitive token like A=$(echo SECRET) must still be scanned + # because the inner token triggers the redaction. We use the + # collected spans rather than re-deriving the leading prefix + # so that the ``export SENSITIVE=$(...)`` shape is recognised + # the same way as ``SENSITIVE=$(...)`` (both share the same + # leading match in ``_DIAGNOSTIC_ENV_ASSIGNMENT_PREFIX_PATTERN`` + # which already accepts an optional ``export`` prefix). + skip_due_to_first_pass = any( + start <= sub < end for start, end in first_pass_spans + ) + if skip_due_to_first_pass: + tail_start = sub + 1 + continue + prior_semi = source.rfind(";", 0, sub) + if prior_semi == -1: + prior_nl = source.rfind("\n", 0, sub) + else: + prior_nl = -1 + skip_due_to_prior = False + if prior_semi != -1: + candidate = source[prior_semi + 1:sub].strip(" \t") + if candidate: + prior_match = re.match( + r"(?:export[ \t]+)?(?P[A-Z][A-Z0-9_]*)\s*=\s*$", + candidate, + ) + if prior_match and _is_sensitive_env_name(prior_match.group("name")): + skip_due_to_prior = True + if not skip_due_to_prior and prior_nl != -1: + candidate = source[prior_nl + 1:sub].strip(" \t") + if candidate: + prior_match = re.match( + r"(?:export[ \t]+)?(?P[A-Z][A-Z0-9_]*)\s*=\s*$", + candidate, + ) + if prior_match and _is_sensitive_env_name(prior_match.group("name")): + skip_due_to_prior = True + if not skip_due_to_prior and prior_semi == -1 and prior_nl == -1: + head = source[:sub].lstrip(" \t") + if head: + prior_match = re.match( + r"(?:export[ \t]+)?(?P[A-Z][A-Z0-9_]*)\s*=\s*$", + head, + ) + if prior_match and _is_sensitive_env_name(prior_match.group("name")): + skip_due_to_prior = True + if skip_due_to_prior: + tail_start = sub + 1 + continue + value_end = _consume_shell_command_substitution(source, sub) + if value_end <= sub: + tail_start = sub + 1 + continue + snippet = source[sub + 2:value_end - 1] if value_end > sub + 2 else source[sub + 2:] + # Scan every uppercase token inside the command substitution so + # that printenv SECRET_TOKEN, echo API_KEY=..., ${TOKEN:+x}, and + # similar forms each trigger redaction even when the leading word + # is a generic command name like "echo" or "printenv". + sensitive_hit = any( + _is_sensitive_env_name(token) + for token in re.findall(r"[A-Z][A-Z0-9_]*", snippet) + ) + if sensitive_hit: + replacements.append((sub, value_end, "")) + last_end = value_end + tail_start = value_end + else: + tail_start = sub + 1 + return _replace_spans(source, replacements) + + redacted = _redact_yaml_explicit_sensitive_fields(text) + redacted = redact_sensitive_env_command_substitutions(redacted) + redacted = _DIAGNOSTIC_ENV_ASSIGNMENT_PATTERN.sub(redact_env, redacted) + redacted = _redact_sensitive_collection_assignments(redacted) + redacted = _redact_multiline_sensitive_fields(redacted) + redacted = _diagnostic_json_assignment_pattern().sub( + lambda match: ( + _redact_assignment_value(match) + if _is_sensitive_structured_assignment_name( + _decode_diagnostic_double_quoted_field_name(match.group("name")) + ) + else match.group(0) + ), + redacted, + ) + redacted = _diagnostic_field_assignment_pattern().sub(redact_structured_field, redacted) + return _redact_partially_redacted_flow_scalars(redacted) + + +def _redact_partially_redacted_flow_scalars(text: str) -> str: + """Collapse any flow-style sensitive scalar tail left after token-level redaction.""" + + field_name_pattern = _diagnostic_field_name_pattern() + pattern = re.compile( + r""" + (?P + " + (?P(?:\\.|[^"\\])*) + " + (?P[ \t\r\n]*:[ \t\r\n]*) + | + (?') + (?P""" + + field_name_pattern + + r""") + (?P=field_name_quote) + (?P[ \t]*(?:=|:)[ \t]*) + ) + + (?P[ \t]+[^\r\n,}\]]+?) + (?=[ \t]*[,}\]]|\r?\n?$) + """, + re.VERBOSE, + ) + + def replace(match: re.Match[str]) -> str: + json_name = match.group("json_name") + normalized_name: Optional[str] + if json_name is not None: + normalized_name = _normalize_diagnostic_field_name( + _decode_diagnostic_double_quoted_field_name(json_name) + ) + sensitive = _is_sensitive_structured_assignment_name(normalized_name) + else: + normalized_name = _normalize_diagnostic_field_name(match.group("field_name")) + sensitive = _is_field_specific_sensitive_redaction_target(match.group("field_name")) + if not sensitive: + return match.group(0) + if normalized_name in {"authorization", "proxy_authorization"}: + trailing_field = match.group("tail").lstrip(" \t") + if _diagnostic_field_assignment_pattern().match(trailing_field): + return match.group(0) + return f"{match.group('prefix')}" + + return pattern.sub(replace, text) + + +def _has_closed_diagnostic_quote(value: str, quote: str) -> bool: + return _diagnostic_quote_close_index(value, quote, start=1) is not None + + +def _diagnostic_quote_close_index(value: str, quote: str, *, start: int) -> Optional[int]: + index = start + while index < len(value): + char = value[index] + if quote == "'" and char == "'" and index + 1 < len(value) and value[index + 1] == "'": + index += 2 + continue + if char == "\\" and index + 1 < len(value): + index += 2 + continue + if char == quote: + return index + 1 + index += 1 + return None + + +def _consume_diagnostic_scalar( + value: str, + start: int, + *, + stop_chars: str = " \t,;", +) -> int: + if start >= len(value): + return start + quote = value[start] + if quote in {"'", '"'}: + index = start + 1 + while index < len(value): + char = value[index] + if quote == "'" and char == "'" and index + 1 < len(value) and value[index + 1] == "'": + index += 2 + continue + if char == "\\" and index + 1 < len(value): + index += 2 + continue + if char == quote: + return index + 1 + index += 1 + return len(value) + + index = start + while index < len(value) and value[index] not in stop_chars: + index += 1 + return index + + +def _consume_diagnostic_collection(value: str, start: int) -> int: + if start >= len(value) or value[start] not in "{[": + return start + + closing_by_opening = {"{": "}", "[": "]"} + stack = [closing_by_opening[value[start]]] + index = start + 1 + while index < len(value): + char = value[index] + if char in {"'", '"'}: + index = _consume_diagnostic_scalar(value, index) + continue + if char in closing_by_opening: + stack.append(closing_by_opening[char]) + index += 1 + continue + if stack and char == stack[-1]: + stack.pop() + index += 1 + if not stack: + return index + continue + index += 1 + return len(value) + + +def _consume_shell_command_substitution(value: str, start: int) -> int: + if start + 1 >= len(value) or value[start:start + 2] != "$(": + return start + + depth = 1 + index = start + 2 + while index < len(value): + char = value[index] + if char == "\\" and index + 1 < len(value): + index += 2 + continue + if char in {"'", '"'}: + index = _consume_diagnostic_scalar(value, index) + continue + if value.startswith("$(", index): + depth += 1 + index += 2 + continue + if char == "(": + depth += 1 + index += 1 + continue + if char == ")": + depth -= 1 + index += 1 + if depth == 0: + return index + continue + index += 1 + return len(value) + + +def _find_diagnostic_flow_scalar_end(value: str, start: int) -> Optional[int]: + """Return the end of an unquoted YAML flow scalar when delimiters are reliable.""" + + closing_by_opening = {"{": "}", "[": "]"} + stack: list[str] = [] + index = 0 + while index < start: + char = value[index] + if char in {"'", '"'}: + index = _consume_diagnostic_scalar(value, index) + continue + if char in closing_by_opening: + stack.append(closing_by_opening[char]) + index += 1 + continue + if stack and char == stack[-1]: + stack.pop() + index += 1 + + if not stack: + return None + + index = start + while index < len(value): + char = value[index] + if char in {"'", '"'}: + index = _consume_diagnostic_scalar(value, index) + continue + if char in closing_by_opening: + stack.append(closing_by_opening[char]) + index += 1 + continue + if len(stack) == 1 and char in {",", stack[-1]}: + return index + if stack and char == stack[-1]: + stack.pop() + index += 1 + continue + index += 1 + + return None + + +def _authorization_value_end(value: str) -> int: + scheme_match = re.match(r"[A-Za-z][A-Za-z0-9_-]*", value) + if scheme_match is None: + return len(value) + + index = scheme_match.end() + while index < len(value) and value[index].isspace(): + index += 1 + + auth_end = _consume_authorization_param_list(value, index) + if auth_end > index: + return auth_end + + simple_value_end = _consume_diagnostic_scalar(value, index) + return simple_value_end if simple_value_end > index else len(value) + + +def _consume_authorization_param_list( + value: str, + start: int, + *, + allowed_names: Optional[frozenset[str]] = None, +) -> int: + index = start + auth_end = start + consumed_any = False + first_param = True + while index < len(value): + while index < len(value) and value[index].isspace(): + index += 1 + if not first_param: + if index >= len(value) or value[index] != ",": + break + index += 1 + while index < len(value) and value[index].isspace(): + index += 1 + name_match = re.match(r"[A-Za-z][A-Za-z0-9_-]*", value[index:]) + if name_match is None: + break + name = _normalize_diagnostic_field_name(name_match.group(0)) + if allowed_names is not None and name not in allowed_names: + break + index += name_match.end() + while index < len(value) and value[index].isspace(): + index += 1 + if index >= len(value) or value[index] != "=": + break + index += 1 + saw_whitespace_after_equals = False + while index < len(value) and value[index].isspace(): + saw_whitespace_after_equals = True + index += 1 + if saw_whitespace_after_equals and re.match(r"[A-Za-z][A-Za-z0-9_-]*\s*=", value[index:]): + break + scalar_end = _consume_diagnostic_scalar(value, index, stop_chars=" \t,") + if scalar_end <= index: + break + consumed_any = True + auth_end = scalar_end + index = scalar_end + first_param = False + + return auth_end if consumed_any else start + + +def _redact_authorization_fields(text: str) -> str: + def redact(match: re.Match[str]) -> str: + prefix = match.group("prefix") + value = match.group("value") or "" + if match.group("quote") is not None and value.lstrip(" \t")[:1] in {'"', "'", "{", "["}: + return match.group(0) + if not value.strip(): + return f"{prefix}" + auth_end = _authorization_value_end(value) + if auth_end <= 0: + return f"{prefix}" + return f"{prefix}{value[auth_end:]}" + + return _AUTHORIZATION_FIELD_PATTERN.sub(redact, text) + + +def redact_diagnostic_text(text: str, *, home: Optional[str] = None, limit: int = _PREVIEW_LIMIT) -> str: + """Redact sensitive diagnostics and return a bounded preview. + + Uppercase environment assignments intentionally reuse the fail-closed child + environment contract. Scalar YAML/JSON/log fields use a narrower allowlist + so ordinary fields such as ``token_budget`` and ``session_id`` remain useful + for troubleshooting. + """ + + redacted = _ANSI_ESCAPE_PATTERN.sub("", text or "") home_path = home or os.path.expanduser("~") if home_path: redacted = redacted.replace(home_path, "~") redacted = re.sub(r"([a-zA-Z][a-zA-Z0-9+.-]*://)[^/\s:@]+:[^@\s/]+@", r"\1@", redacted) redacted = _URL_PATTERN.sub(_redact_sensitive_diagnostic_url, redacted) - redacted = re.sub(r"(?i)(authorization\s*[:=]\s*)(bearer\s+)?[^\s]+", r"\1", redacted) - redacted = re.sub(r"(?i)(cookie\s*[:=]\s*)[^\n\r]+", r"\1", redacted) + redacted = _redact_authorization_fields(redacted) + redacted = re.sub(r"(?i)(cookie[ \t]*[:=][ \t]*)[^\n\r]+", r"\1", redacted) + redacted = _redact_sensitive_diagnostic_assignments(redacted) redacted = re.sub(r"(?i)(session[_-]?secret\s*[:=]\s*)[^\s]+", r"\1", redacted) redacted = re.sub(r"\b(sk-[A-Za-z0-9_-]{12,})\b", "", redacted) redacted = re.sub(r"\b(AIza[A-Za-z0-9_-]{16,})\b", "", redacted) @@ -1500,8 +2852,29 @@ def _is_command_not_executable_error(exc: OSError) -> bool: return False +@lru_cache(maxsize=1) +def _registered_sensitive_env_exact_names() -> frozenset[str]: + """Reuse the config registry's secret-field contract without creating an import cycle.""" + + try: + from src.core.config_registry import _FIELD_DEFINITIONS + except Exception: + return frozenset() + + return frozenset( + str(name).upper() + for name, metadata in _FIELD_DEFINITIONS.items() + if isinstance(metadata, Mapping) and metadata.get("is_sensitive") + ) + + def _is_sensitive_env_name(upper_name: str) -> bool: - return any(pattern in upper_name for pattern in _SENSITIVE_ENV_PATTERNS) + return ( + upper_name in _SENSITIVE_ENV_EXACT_NAMES + or upper_name in _registered_sensitive_env_exact_names() + ) or any( + pattern in upper_name for pattern in _SENSITIVE_ENV_PATTERNS + ) def _first_unsafe_token(tokens: Sequence[str]) -> str: diff --git a/tests/test_local_cli_backend.py b/tests/test_local_cli_backend.py index 5222a98c7..621db22d7 100644 --- a/tests/test_local_cli_backend.py +++ b/tests/test_local_cli_backend.py @@ -19,6 +19,7 @@ from tests.litellm_stub import ensure_litellm_stub ensure_litellm_stub() from src.analyzer import GeminiAnalyzer # noqa: E402 +from src.core.config_registry import _FIELD_DEFINITIONS # noqa: E402 from src.llm import local_cli_backend as local_cli_backend_module # noqa: E402 from src.llm.generation_backend import GenerationError, GenerationErrorCode # noqa: E402 from src.llm.local_cli_backend import ( # noqa: E402 @@ -35,6 +36,20 @@ from src.llm.local_cli_backend import ( # noqa: E402 ) +def _registered_sensitive_titles_needing_title_match() -> list[str]: + titles = [] + for field_name, metadata in _FIELD_DEFINITIONS.items(): + if not isinstance(metadata, dict) or not metadata.get("is_sensitive"): + continue + title = metadata.get("title") + if not isinstance(title, str) or not title: + continue + if title.upper() in {field_name.upper(), field_name.replace("_", " ").upper()}: + continue + titles.append(title) + return sorted(set(titles)) + + def _config(**overrides): defaults = { "generation_backend_timeout_seconds": 5, @@ -1372,13 +1387,17 @@ def test_env_allowlist_and_denylist(monkeypatch) -> None: monkeypatch.setenv("CODEX_HOME", "/tmp/codex-home") monkeypatch.setenv("LC_MESSAGES", "C") monkeypatch.setenv("UNRELATED_VALUE", "leak") + monkeypatch.setenv("AIHUBMIX_KEY", "aihubmix-secret") monkeypatch.setenv("CODEX_CLI_TOKEN", "codex-secret") monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-secret") monkeypatch.setenv("ANTHROPIC_MODEL", "claude") monkeypatch.setenv("CLAUDE_CONFIG_DIR", "/tmp/claude") + monkeypatch.setenv("LONGBRIDGE_APP_KEY", "longbridge-secret") monkeypatch.setenv("OPENCODE_CONFIG_CONTENT", "{}") monkeypatch.setenv("OPENAI_API_KEY", "sk-secret") + monkeypatch.setenv("PUSHOVER_USER_KEY", "pushover-secret") monkeypatch.setenv("WEBHOOK_TOKEN", "token") + monkeypatch.setenv("WECOM_ENCODING_AES_KEY", "wecom-secret") monkeypatch.setenv("AUTHORIZATION", "Bearer token") child_env = build_local_cli_env() @@ -1388,13 +1407,17 @@ def test_env_allowlist_and_denylist(monkeypatch) -> None: assert child_env["CODEX_HOME"] == "/tmp/codex-home" assert child_env["LC_MESSAGES"] == "C" assert "UNRELATED_VALUE" not in child_env + assert "AIHUBMIX_KEY" not in child_env assert "CODEX_CLI_TOKEN" not in child_env assert "ANTHROPIC_API_KEY" not in child_env assert "ANTHROPIC_MODEL" not in child_env assert "CLAUDE_CONFIG_DIR" not in child_env + assert "LONGBRIDGE_APP_KEY" not in child_env assert "OPENCODE_CONFIG_CONTENT" not in child_env assert "OPENAI_API_KEY" not in child_env + assert "PUSHOVER_USER_KEY" not in child_env assert "WEBHOOK_TOKEN" not in child_env + assert "WECOM_ENCODING_AES_KEY" not in child_env assert "AUTHORIZATION" not in child_env @@ -1618,6 +1641,1427 @@ def test_diagnostics_redacts_webhook_urls_and_preserves_adjacent_normal_urls() - assert "https://example.com/public/docs?foo=bar" in redacted +@pytest.mark.parametrize( + ("text", "secret"), + [ + ("FEISHU_APP_SECRET=xxy12345abcdef", "xxy12345abcdef"), + ("AIHUBMIX_KEY=short", "short"), + ("CUSTOM_API_KEY=abc123xyz789short", "abc123xyz789short"), + ("LONGBRIDGE_APP_KEY=short", "short"), + ("NTFY_URL=https://ntfy.sh/private-topic", "https://ntfy.sh/private-topic"), + ("API_KEYS=short", "short"), + ("OPENAI_API_KEYS=short", "short"), + ("MYOPENAIKEY=short", "short"), + ("OPENAI_V2_API_KEY=short", "short"), + (r"OPENAI_FOO=\ tiny-secret session_id=ok", "tiny-secret"), + ("OPENAI_API_KEY=\\\ntiny-secret session_id=ok", "tiny-secret"), + ("OPENAI_FOO=$(printf %s tiny-secret) session_id=ok", "tiny-secret"), + ("export OPENAI_FOO=$(printf %s tiny-secret) session_id=ok", "tiny-secret"), + ("PUSHOVER_USER_KEY=short", "short"), + ("R2_SECRET_ACCESS_KEY=short", "short"), + ("My_Api_Key=myvalue", "myvalue"), + ("API Key: tiny-secret session_id=ok", "tiny-secret"), + ("Client Secret: tiny-secret session_id=ok", "tiny-secret"), + ("Secret Access Key: tiny-secret session_id=ok", "tiny-secret"), + ("DingTalk App Key: tiny-secret session_id=ok", "tiny-secret"), + ("Pushover User Key: tiny-secret session_id=ok", "tiny-secret"), + ('{"Database URL":"tiny-secret","session_id":"ok"}', "tiny-secret"), + ("PASSWORD='abc def ghi' next", "abc def ghi"), + ("SESSION_SECRET='abc def ghi' next", "abc def ghi"), + ("Authorization: Bearer tiny", "tiny"), + ('"api_key": "short123"', "short123"), + ('{"accessToken":"short123"}', "short123"), + ("api_keys: short123", "short123"), + ("bot_token: tiny", "tiny"), + ("telegram_bot_token: tiny", "tiny"), + ("client_secret: tiny", "tiny"), + ("clientSecret: tiny", "tiny"), + ("database_url: sqlite-short", "sqlite-short"), + ("aws_secret_access_key: tiny", "tiny"), + ("db_passwd: tiny-secret", "tiny-secret"), + ('{"db_passwd":"tiny-secret"}', "tiny-secret"), + ('{"set-cookie":"session=tiny-secret"}', "tiny-secret"), + (r'{"api\u005fkey":"tiny-secret"}', "tiny-secret"), + (r'{"api\x5fkey":"tiny-secret"}', "tiny-secret"), + ("OPENAI_API_KEY='x'\"'\"'tiny-secret' session_id=ok", "tiny-secret"), + ("OPENAI_API_KEY+=tiny-secret session_id=ok", "tiny-secret"), + ("{'api_key': 'tiny-secret', 'session_id': 'ok'}", "tiny-secret"), + ("'password': tiny-secret session_id=ok", "tiny-secret"), + ], +) +def test_diagnostics_redacts_short_credential_assignments(text: str, secret: str) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert secret not in redacted + assert "" in redacted + + +def test_diagnostics_redacts_yaml_scalars_with_spaces_and_blocks() -> None: + text = ( + "retry: 3 password: correct horse battery staple\n" + "backup_password: 'correct horse''s secret'\n" + "INFO private_key: |\n" + " tiny-secret\n" + " second secret line\n" + "token_budget: 1000\n" + ) + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "correct horse battery staple" not in redacted + assert "correct horse''s secret" not in redacted + assert "tiny-secret" not in redacted + assert "second secret line" not in redacted + assert "retry: 3" in redacted + assert "token_budget: 1000" in redacted + assert "password: " in redacted + assert "backup_password: ''\n" in redacted + assert "''s secret" not in redacted + assert "private_key: " in redacted + + +def test_diagnostics_redacts_yaml_block_scalars_with_node_properties() -> None: + text = ( + "private_key: ! &pem |\n" + " tiny-secret\n" + "session_id: yaml123\n" + ) + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-secret" not in redacted + assert redacted == "private_key: \nsession_id: yaml123\n" + + +def test_diagnostics_preserves_non_sensitive_spaced_key_labels() -> None: + text = "Cache Key: shard-one\nSort Key: created-at\nsession_id: ok\n" + + assert redact_diagnostic_text(text, limit=1000) == text + + +@pytest.mark.parametrize( + ("text", "secrets", "preserved"), + [ + ( + "? api_key\n: tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "? 'api_key'\n: single-quoted-secret\nsession_id: yaml123\n", + ("single-quoted-secret",), + "session_id: yaml123", + ), + ( + '? "api_key"\n: double-quoted-secret\nsession_id: yaml123\n', + ("double-quoted-secret",), + "session_id: yaml123", + ), + ( + '? "api\\x5fkey"\n: tiny-secret\nsession_id: yaml123\n', + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "? credentials\n:\n- tiny-one\n- tiny-two\nsession_id: yaml123\n", + ("tiny-one", "tiny-two"), + "session_id: yaml123", + ), + ( + "? private_key\n: |\n tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "- ? api_key\n : tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "? !!str api_key\n: tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "- ? &cred private_key\n : tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "- ? credentials\n :\n - tiny-one\n - tiny-two\nsession_id: yaml123\n", + ("tiny-one", "tiny-two"), + "session_id: yaml123", + ), + ( + "? api_key\n# note\n: tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "? api_key\n # note\n: tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ( + "? api_key\n\n: tiny-secret\nsession_id: yaml123\n", + ("tiny-secret",), + "session_id: yaml123", + ), + ], +) +def test_diagnostics_redacts_yaml_explicit_sensitive_mappings( + text: str, + secrets: tuple[str, ...], + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + for secret in secrets: + assert secret not in redacted + assert ": " in redacted + assert preserved in redacted + + +def test_diagnostics_redacts_indented_values_under_empty_sensitive_yaml_field() -> None: + text = "api_keys:\n - tiny-one\n - tiny-two\nsession_id: yaml123\n" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-one" not in redacted + assert "tiny-two" not in redacted + assert "api_keys: \n" in redacted + assert "session_id: yaml123\n" in redacted + + +def test_diagnostics_redacts_comment_only_sensitive_yaml_field_blocks() -> None: + text = "api_keys: # configured keys\n - tiny-one\n - tiny-two\nsession_id: yaml123\n" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-one" not in redacted + assert "tiny-two" not in redacted + assert "api_keys: \n" in redacted + assert "session_id: yaml123\n" in redacted + + +def test_diagnostics_redacts_comment_lines_within_sensitive_yaml_field_blocks() -> None: + text = "api_keys: # configured keys\n# nested note\n- tiny-one\n- tiny-two\nsession_id: yaml123\n" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "nested note" not in redacted + assert "tiny-one" not in redacted + assert "tiny-two" not in redacted + assert redacted == "api_keys: \nsession_id: yaml123\n" + + +def test_diagnostics_redacts_indentless_sequences_under_sensitive_yaml_field() -> None: + text = "api_keys:\n- tiny-one\n- tiny-two\nsession_id: yaml123\n" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-one" not in redacted + assert "tiny-two" not in redacted + assert "api_keys: \n" in redacted + assert "session_id: yaml123\n" in redacted + + +def test_diagnostics_redacts_sensitive_collections() -> None: + text = ( + "api_keys: [first-secret, second-secret] token_budget: 1000\n" + '{"credentials":{"username":"alice","value":"tiny-secret"},"session_id":"abc123"}' + ) + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "first-secret" not in redacted + assert "second-secret" not in redacted + assert "tiny-secret" not in redacted + assert "api_keys: token_budget: 1000" in redacted + assert '{"credentials":,"session_id":"abc123"}' in redacted + + +@pytest.mark.parametrize( + ("text", "secret_values", "preserved"), + [ + ( + "api_keys:\n - tiny-one\n - tiny-two\nsession_id: abc123\n", + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + "credentials:\n username: alice\n value: tiny-secret\nsession_id: abc123\n", + ("alice", "tiny-secret"), + "session_id: abc123", + ), + ( + "private_key:\n tiny-secret\nfoo: bar\n", + ("tiny-secret",), + "foo: bar", + ), + ( + "api_keys: # configured keys\n - tiny-one\nsession_id: abc123\n", + ("tiny-one",), + "session_id: abc123", + ), + ( + "api_keys:\n- tiny-one\n- tiny-two\nsession_id: abc123\n", + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + "private_key: &pem |\n tiny-secret\nsession_id: abc123\n", + ("tiny-secret",), + "session_id: abc123", + ), + ( + "credentials: !vault &creds\n value: tiny-secret\nsession_id: abc123\n", + ("tiny-secret",), + "session_id: abc123", + ), + ( + "cookie:\n session: tiny-one\n csrf: tiny-two\nsession_id: abc123\n", + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + "password: correct horse\n battery staple\nsession_id: abc123\n", + ("correct horse", "battery staple"), + "session_id: abc123", + ), + ( + '"password": correct horse\n battery staple\nsession_id: abc123\n', + ("correct horse", "battery staple"), + "session_id: abc123", + ), + ( + '"password":\n tiny-one\n tiny-two\nsession_id: abc123\n', + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + 'password: !secret "tiny-one\n tiny-two"\nsession_id: abc123\n', + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + "authorization:\n scheme: Bearer\n credentials: tiny-auth\nsession_id: abc123\n", + ("Bearer", "tiny-auth"), + "session_id: abc123", + ), + ( + "Authorization: Basic tiny-auth\n continued-secret\nsession_id: abc123\n", + ("tiny-auth", "continued-secret"), + "session_id: abc123", + ), + ( + "API Key:\n - tiny-one\n - tiny-two\nsession_id: abc123\n", + ("tiny-one", "tiny-two"), + "session_id: abc123", + ), + ( + "'credentials':\n username: alice\n value: tiny-secret\nsession_id: abc123\n", + ("alice", "tiny-secret"), + "session_id: abc123", + ), + ], +) +def test_diagnostics_redacts_indented_blocks_under_empty_sensitive_yaml_fields( + text: str, + secret_values: tuple[str, ...], + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + for secret in secret_values: + assert secret not in redacted + assert "" in redacted + assert preserved in redacted + + +@pytest.mark.parametrize( + ("text", "secret", "preserved"), + [ + ( + '{"AIHUBMIX_KEY":"json-short","session_id":"json123"}', + "json-short", + '"session_id":"json123"', + ), + ( + '{"DINGTALK_APP_KEY":"ding-short","session_id":"ding123"}', + "ding-short", + '"session_id":"ding123"', + ), + ( + "WECOM_ENCODING_AES_KEY: wecom-short\nsession_id: wecom123\n", + "wecom-short", + "session_id: wecom123", + ), + ( + "PUSHOVER_USER_KEY: push-short\nsession_id: push123\n", + "push-short", + "session_id: push123", + ), + ( + '{"NTFY_URL":"private-topic","session_id":"ntfy123"}', + "private-topic", + '"session_id":"ntfy123"', + ), + ], +) +def test_diagnostics_applies_registered_sensitive_names_to_structured_fields( + text: str, + secret: str, + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert secret not in redacted + assert "" in redacted + assert preserved in redacted + + +@pytest.mark.parametrize( + "field_name", + sorted( + local_cli_backend_module._SENSITIVE_ENV_EXACT_NAMES + | local_cli_backend_module._registered_sensitive_env_exact_names() + ), +) +def test_all_registered_sensitive_exact_names_are_redacted_in_json( + field_name: str, +) -> None: + redacted = redact_diagnostic_text( + json.dumps({field_name: "tinyZ9", "session_id": "json123"}), + limit=1000, + ) + + assert "tinyZ9" not in redacted + assert '"session_id": "json123"' in redacted + + +@pytest.mark.parametrize( + "field_name", + sorted(local_cli_backend_module._registered_sensitive_env_exact_names()), +) +def test_all_registered_sensitive_exact_names_are_redacted_as_spaced_labels( + field_name: str, +) -> None: + label = field_name.replace("_", " ") + + redacted = redact_diagnostic_text( + f"{label}: tinyZ9 session_id=label123", + limit=1000, + ) + + assert "tinyZ9" not in redacted + assert "" in redacted + + +@pytest.mark.parametrize( + "field_title", + _registered_sensitive_titles_needing_title_match(), +) +def test_registered_sensitive_config_titles_are_redacted_as_structured_labels( + field_title: str, +) -> None: + redacted = redact_diagnostic_text( + f"{field_title}: tiny-secret session_id=label123", + limit=1000, + ) + + assert "tiny-secret" not in redacted + assert "" in redacted + assert "session_id=label123" in redacted + + +@pytest.mark.parametrize( + "field_title", + _registered_sensitive_titles_needing_title_match(), +) +def test_registered_sensitive_config_titles_are_redacted_in_json( + field_title: str, +) -> None: + redacted = redact_diagnostic_text( + json.dumps({field_title: "tiny-secret", "session_id": "json123"}), + limit=1000, + ) + + assert "tiny-secret" not in redacted + assert "" in redacted + assert '"session_id": "json123"' in redacted + + +def test_diagnostics_redacts_ansi_prefixed_sensitive_fields() -> None: + text = "\x1b[31mpassword: tiny\x1b[0m session_id=abc123 \x1b[32mapi_key: short123\x1b[0m" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "\x1b[" not in redacted + assert "tiny" not in redacted + assert "short123" not in redacted + assert redacted == "password: " + + +@pytest.mark.parametrize( + ("text", "secret", "preserved"), + [ + ( + "Authorization: Basic dGlueTpzZWNyZXQ= session_id=abc123", + "dGlueTpzZWNyZXQ=", + "session_id=abc123", + ), + ( + "Authorization: Token tiny-secret token_budget=1000", + "tiny-secret", + "token_budget=1000", + ), + ( + "authorization=Negotiate abc.def.ghi token_budget=1000", + "abc.def.ghi", + "token_budget=1000", + ), + ( + 'Authorization: Digest username="foo", realm="example", response="tiny-secret" session_id=abc123', + "tiny-secret", + "session_id=abc123", + ), + ( + "Proxy-Authorization: Basic tiny-secret session_id=abc123", + "tiny-secret", + "session_id=abc123", + ), + ( + "proxy-authorization=Negotiate abc.def.ghi token_budget=1000", + "abc.def.ghi", + "token_budget=1000", + ), + ( + "proxy_authorization: Basic underscore-secret session_id=proxy123", + "underscore-secret", + "session_id=proxy123", + ), + ( + "'authorization': Bearer tiny-secret session_id=ok", + "tiny-secret", + "session_id=ok", + ), + ( + "'proxy_authorization': Basic tiny-secret session_id=ok", + "tiny-secret", + "session_id=ok", + ), + ( + '"authorization": Bearer tiny-secret session_id=ok', + "tiny-secret", + "session_id=ok", + ), + ( + '"proxy_authorization": Basic tiny-secret session_id=ok', + "tiny-secret", + "session_id=ok", + ), + ( + "proxyAuthorization=Negotiate camel.secret token_budget=1000", + "camel.secret", + "token_budget=1000", + ), + ], +) +def test_diagnostics_redacts_non_bearer_authorization_values( + text: str, + secret: str, + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert secret not in redacted + assert preserved in redacted + assert ( + "Authorization: " in redacted + or "authorization=" in redacted + or "Proxy-Authorization: " in redacted + or "proxy-authorization=" in redacted + or "proxy_authorization: " in redacted + or "'authorization': " in redacted + or "'proxy_authorization': " in redacted + or '"authorization": ' in redacted + or '"proxy_authorization": ' in redacted + or "proxyAuthorization=" in redacted + ) + + +def test_diagnostics_redacts_parameterized_oauth_authorization_values() -> None: + text = ( + 'Authorization: OAuth oauth_consumer_key="client", ' + 'oauth_signature="tiny-secret" session_id=abc123' + ) + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-secret" not in redacted + assert "Authorization: session_id=abc123" in redacted + + +@pytest.mark.parametrize( + ("text", "preserved"), + [ + ( + "Authorization: Bearer first-secret Proxy-Authorization: Basic second-secret session_id=ok", + "session_id=ok", + ), + ( + "Authorization: Bearer first-secret authorization=Basic second-secret session_id=ok", + "session_id=ok", + ), + ], +) +def test_diagnostics_redacts_multiple_authorization_fields_on_one_line( + text: str, + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert "first-secret" not in redacted + assert "second-secret" not in redacted + assert preserved in redacted + assert redacted.count("") == 2 + + +@pytest.mark.parametrize( + ("text", "secret", "preserved"), + [ + ( + "Authorization: AWS4-HMAC-SHA256 Credential=AKIA/test/aws4_request, " + "SignedHeaders=host;x-amz-date, Signature=tiny-secret session_id=aws123", + "tiny-secret", + "session_id=aws123", + ), + ( + 'Authorization: Signature keyId="client",algorithm="hmac-sha256",signature="tiny-secret" ' + "token_budget=1000", + "tiny-secret", + "token_budget=1000", + ), + ], +) +def test_diagnostics_redacts_parameterized_authorization_values_for_any_scheme( + text: str, + secret: str, + preserved: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert secret not in redacted + assert preserved in redacted + assert "Authorization: " in redacted + + +def test_diagnostics_redacts_unclosed_quoted_sensitive_scalar() -> None: + redacted = redact_diagnostic_text('password: "correct horse battery staple', limit=1000) + + assert "correct horse battery staple" not in redacted + assert redacted == "password: " + + +def test_diagnostics_redacts_multiline_quoted_sensitive_scalar() -> None: + redacted = redact_diagnostic_text( + 'password: "correct horse\n battery staple"\nsession_id=abc123\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == "password: \nsession_id=abc123\n" + + +def test_diagnostics_redacts_multiline_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '"password": correct horse\n battery staple\nsession_id: abc123\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '"password": \nsession_id: abc123\n' + + +def test_diagnostics_redacts_single_line_multiword_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '"password": correct horse battery staple\nsession_id: abc123\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '"password": \nsession_id: abc123\n' + + +def test_diagnostics_redacts_tagged_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '!!str "password": correct horse battery staple\nsession_id: ok\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '!!str "password": \nsession_id: ok\n' + + +def test_diagnostics_redacts_anchored_continued_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '&pem "password": correct horse\n battery staple\nsession_id: ok\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '&pem "password": \nsession_id: ok\n' + + +def test_diagnostics_redacts_tagged_uri_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '! "password": correct horse battery staple\nsession_id: ok\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '! "password": \nsession_id: ok\n' + + +def test_diagnostics_redacts_tagged_uri_continued_plain_scalar_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '! "password": correct horse\n battery staple\nsession_id: ok\n', + limit=1000, + ) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == '! "password": \nsession_id: ok\n' + + +def test_diagnostics_redacts_indentless_sequence_under_double_quoted_yaml_key() -> None: + redacted = redact_diagnostic_text( + '"password": # configured\n- tiny-one\n- tiny-two\nsession_id: abc123\n', + limit=1000, + ) + + assert "configured" not in redacted + assert "tiny-one" not in redacted + assert "tiny-two" not in redacted + assert redacted == '"password": \nsession_id: abc123\n' + + +def test_diagnostics_redacts_pretty_printed_json_value_on_following_line() -> None: + redacted = redact_diagnostic_text( + '{\n "api_key":\n "tiny-secret",\n "session_id": "json123"\n}', + limit=1000, + ) + + assert "tiny-secret" not in redacted + assert '"api_key":\n ""' in redacted + assert '"session_id": "json123"' in redacted + + +@pytest.mark.parametrize( + "text", + [ + '{"authorization":"Bearer tiny-secret","session_id":"abc123"}', + '{"cookie":"session=tiny-secret","session_id":"abc123"}', + ], +) +def test_diagnostics_redacts_quoted_json_authentication_fields(text: str) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-secret" not in redacted + assert '""' in redacted + assert '"session_id":"abc123"' in redacted + + +def test_diagnostics_preserves_json_structure_for_quoted_authorization_fields() -> None: + redacted = redact_diagnostic_text( + '{"authorization":"Bearer tiny-secret","session_id":"abc123"}', + limit=1000, + ) + + assert redacted == '{"authorization":"","session_id":"abc123"}' + + +@pytest.mark.parametrize( + ("text", "expected"), + [ + ( + '{"password": correct horse battery staple, "session_id": "ok"}', + '{"password": , "session_id": "ok"}', + ), + ( + '{"api_key": correct horse, "session_id": "ok"}', + '{"api_key": , "session_id": "ok"}', + ), + ( + "{'password': correct horse battery staple, 'session_id': 'ok'}", + "{'password': , 'session_id': 'ok'}", + ), + ( + "{password: correct horse, session_id: ok}", + "{password: , session_id: ok}", + ), + ], +) +def test_diagnostics_redacts_flow_style_sensitive_keys_with_unquoted_multiword_scalars( + text: str, + expected: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert "correct horse" not in redacted + assert "battery staple" not in redacted + assert redacted == expected + + +@pytest.mark.parametrize( + "text", + [ + "password: correct horse=staple session_id=abc123", + "password: correct horse_staple=value session_id=abc123", + ], +) +def test_diagnostics_fails_closed_for_unquoted_yaml_secret_with_assignment( + text: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + assert redacted == "password: " + + +@pytest.mark.parametrize( + ("text", "secret", "expected"), + [ + ("password: abc,def session_id=abc123", "abc,def", "password: "), + ( + "bot_token=tiny]} token_budget: 1000", + "tiny]}", + "bot_token= token_budget: 1000", + ), + ], +) +def test_diagnostics_redacts_sensitive_scalars_with_punctuation( + text: str, + secret: str, + expected: str, +) -> None: + redacted = redact_diagnostic_text(text, limit=1000) + + assert secret not in redacted + assert redacted == expected + + +@pytest.mark.parametrize( + "sensitive_pattern", + local_cli_backend_module._SENSITIVE_ENV_PATTERNS, +) +def test_uppercase_diagnostic_assignment_tracks_child_env_sensitive_contract( + sensitive_pattern: str, +) -> None: + name_segment = sensitive_pattern.strip("_") + text = f"DSA_{name_segment}_VALUE=tiny-value" + + redacted = redact_diagnostic_text(text, limit=1000) + + assert "tiny-value" not in redacted + assert "" in redacted + + +@pytest.mark.parametrize( + "text", + [ + "MONKEY=banana next", + "KEYBOARD_LAYOUT=us next", + "retry: 3 token_budget: 1000", + "docs=https://example.com/public/docs?monkey=banana&foo=bar", + "analysis_key_factor=valuation next", + "sort_key=price primary_key=id cache_key=reports", + "session_id=abc123 user_session: abc123", + 'message: "normal diagnostic value"', + ], +) +def test_diagnostics_preserves_noncredential_assignments(text: str) -> None: + assert redact_diagnostic_text(text, limit=1000) == text + + +def test_nonzero_exit_diagnostic_previews_redact_short_credentials( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("CUSTOM_API_KEY=stdout-short session_id=abc123") +print("password: correct horse battery staple") +print("backup_password: 'correct horse''s secret'") +print("bot_token: tiny,trail token_budget: 1000") +print("Authorization: Basic dGlueTpzZWNyZXQ= session_id=auth123") +print('"api_keys": "stderr-short" token_budget: 1000', file=sys.stderr) +print("private_key: |\\n tiny-secret", file=sys.stderr) +print("telegram_bot_token=tiny]} session_id=stderr123", file=sys.stderr) +print("authorization=Token tiny-secret token_budget=1000", file=sys.stderr) +print("authorization=Negotiate abc.def.ghi token_budget=2000", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + assert "stdout-short" not in stdout_preview + assert "stderr-short" not in stderr_preview + assert "correct horse battery staple" not in stdout_preview + assert "correct horse''s secret" not in stdout_preview + assert "tiny-secret" not in stderr_preview + assert "tiny,trail" not in stdout_preview + assert "tiny]}" not in stderr_preview + assert "dGlueTpzZWNyZXQ=" not in stdout_preview + assert "abc.def.ghi" not in stderr_preview + assert "CUSTOM_API_KEY=" in stdout_preview + assert "password: " in stdout_preview + assert "backup_password: ''" in stdout_preview + assert "bot_token: " in stdout_preview + assert "Authorization: " in stdout_preview + assert '"api_keys": ""' in stderr_preview + assert "private_key: " in stderr_preview + assert "telegram_bot_token=" in stderr_preview + assert "authorization=" in stderr_preview + assert "''s secret" not in stdout_preview + assert "session_id=abc123" in stdout_preview + assert "session_id=auth123" in stdout_preview + assert "session_id=stderr123" in stderr_preview + assert "token_budget: 1000" in stderr_preview + assert "token_budget=1000" in stderr_preview + assert "token_budget=2000" in stderr_preview + + +def test_nonzero_exit_diagnostic_previews_redact_digest_proxy_and_camelcase_credentials( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + f""" +import sys +print('Authorization: Digest username="foo", realm="example", response="tiny-secret" session_id=auth123', file=sys.stderr) +print('Proxy-Authorization: Basic proxy-short session_id=proxy123', file=sys.stderr) +print('{{"accessToken":"json-short","session_id":"camel123"}}', file=sys.stderr) +print('clientSecret: yaml-short token_budget=1000', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stderr_preview = exc_info.value.details["stderr_preview"] + assert "tiny-secret" not in stderr_preview + assert "proxy-short" not in stderr_preview + assert "json-short" not in stderr_preview + assert "yaml-short" not in stderr_preview + assert 'Authorization: session_id=auth123' in stderr_preview + assert 'Proxy-Authorization: session_id=proxy123' in stderr_preview + assert '{"accessToken":"","session_id":"camel123"}' in stderr_preview + assert 'clientSecret: \n' in stderr_preview + + +def test_nonzero_exit_diagnostic_previews_redact_ansi_oauth_and_multiline_quoted_secrets( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("\\x1b[31mpassword: tiny\\x1b[0m session_id=ansi123") +print('Authorization: OAuth oauth_consumer_key="client", oauth_signature="tiny-secret" session_id=oauth123', file=sys.stderr) +print('password: "correct horse', file=sys.stderr) +print(' battery staple"', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + assert "\x1b[" not in stdout_preview + assert "tiny" not in stdout_preview + assert "tiny-secret" not in stderr_preview + assert "correct horse" not in stderr_preview + assert "battery staple" not in stderr_preview + assert "password: \n" in stdout_preview + assert "Authorization: session_id=oauth123" in stderr_preview + assert "password: \n" in stderr_preview + + +def test_nonzero_exit_diagnostic_previews_redact_sensitive_collections( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("api_keys: [first-secret, second-secret] token_budget: 1000") +print('{"credentials":{"username":"alice","value":"tiny-secret"},"session_id":"nested123"}', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + assert "first-secret" not in stdout_preview + assert "second-secret" not in stdout_preview + assert "tiny-secret" not in stderr_preview + assert "api_keys: token_budget: 1000" in stdout_preview + assert '{"credentials":,"session_id":"nested123"}' in stderr_preview + + +def test_nonzero_exit_previews_redact_empty_yaml_blocks_and_registered_fields( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("api_keys:\\n - tiny-one\\n - tiny-two\\nsession_id: yaml123") +print("api_keys: # configured keys\\n# nested note\\n- stdout-short\\n- stdout-short-2\\nsession_id: yaml456") +print("private_key: ! &pem |\\n anchored-secret\\nsession_id: anchor123") +print("cookie:\\n session: cookie-one\\n csrf: cookie-two\\nsession_id: cookie-yaml") +print("password: correct horse\\n battery staple\\nsession_id: plain-yaml") +print('"password": correct horse\\n battery staple\\nsession_id: quoted-key-yaml') +print('"password": correct horse battery staple\\nsession_id: quoted-inline-yaml') +print('"password":\\n quoted-empty-one\\n quoted-empty-two\\nsession_id: quoted-empty-yaml') +print('password: !secret "tagged-one\\n tagged-two"\\nsession_id: tagged-yaml') +print('{"AIHUBMIX_KEY":"json-short","session_id":"json123"}', file=sys.stderr) +print('{"set-cookie":"session=cookie-header","session_id":"header123"}', file=sys.stderr) +print('{"api\\\\u005fkey":"escaped-json","session_id":"escaped123"}', file=sys.stderr) +print("? api_key\\n: explicit-secret\\nsession_id: explicit123", file=sys.stderr) +print("? 'api_key'\\n: quoted-explicit-secret\\nsession_id: quoted-explicit123", file=sys.stderr) +print("- ? api_key\\n : nested-explicit-secret\\nsession_id: nested-explicit123", file=sys.stderr) +print("- ? credentials\\n :\\n - nested-one\\n - nested-two\\nsession_id: nested-list123", file=sys.stderr) +print("OPENAI_API_KEY='x'\\\"'\\\"'shell-quoted' session_id=quoted123", file=sys.stderr) +print("OPENAI_API_KEY+=appended-secret session_id=append123", file=sys.stderr) +print("{'api_key': 'single-quoted', 'session_id': 'single123'}", file=sys.stderr) +print("WECOM_ENCODING_AES_KEY: yaml-short\\nsession_id: wecom123", file=sys.stderr) +print("OPENAI_FOO=\\\\ shell-short session_id=shell123", file=sys.stderr) +print("OPENAI_API_KEY=\\\\\\ncontinued-shell-secret session_id=shell456", file=sys.stderr) +print("API Key: label-short session_id=label123", file=sys.stderr) +print("OpenAI API Keys (Multi): title-short session_id=title123", file=sys.stderr) +print("? api_key\\n# note\\n: commented-explicit-secret\\nsession_id: explicit456", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + for secret in ( + "tiny-one", + "tiny-two", + "nested note", + "stdout-short", + "stdout-short-2", + "anchored-secret", + "cookie-one", + "cookie-two", + "correct horse", + "battery staple", + "quoted-empty-one", + "quoted-empty-two", + "tagged-one", + "tagged-two", + "json-short", + "cookie-header", + "escaped-json", + "explicit-secret", + "quoted-explicit-secret", + "nested-explicit-secret", + "nested-one", + "nested-two", + "shell-quoted", + "appended-secret", + "single-quoted", + "yaml-short", + "shell-short", + "continued-shell-secret", + "label-short", + "title-short", + "commented-explicit-secret", + ): + assert secret not in f"{stdout_preview}\n{stderr_preview}" + assert "api_keys: " in stdout_preview + assert "session_id: yaml123" in stdout_preview + assert "session_id: yaml456" in stdout_preview + assert "private_key: " in stdout_preview + assert "session_id: anchor123" in stdout_preview + assert "cookie: " in stdout_preview + assert "session_id: cookie-yaml" in stdout_preview + assert "session_id: plain-yaml" in stdout_preview + assert "session_id: quoted-key-yaml" in stdout_preview + assert "session_id: quoted-inline-yaml" in stdout_preview + assert "session_id: quoted-empty-yaml" in stdout_preview + assert "session_id: tagged-yaml" in stdout_preview + assert '{"AIHUBMIX_KEY":"","session_id":"json123"}' in stderr_preview + assert '{"set-cookie":"","session_id":"header123"}' in stderr_preview + assert r'{"api\u005fkey":"","session_id":"escaped123"}' in stderr_preview + assert "? api_key\n: \nsession_id: explicit123" in stderr_preview + assert "? 'api_key'\n: \nsession_id: quoted-explicit123" in stderr_preview + assert "- ? api_key\n : \nsession_id: nested-explicit123" in stderr_preview + assert "- ? credentials\n : \nsession_id: nested-list123" in stderr_preview + assert "OPENAI_API_KEY='' session_id=quoted123" in stderr_preview + assert "OPENAI_API_KEY+= session_id=append123" in stderr_preview + assert "{'api_key': '', 'session_id': 'single123'}" in stderr_preview + assert "WECOM_ENCODING_AES_KEY: " in stderr_preview + assert "session_id: wecom123" in stderr_preview + assert "OPENAI_FOO= session_id=shell123" in stderr_preview + assert "OPENAI_API_KEY= session_id=shell456" in stderr_preview + assert "API Key: " in stderr_preview + assert "OpenAI API Keys (Multi): " in stderr_preview + assert "? api_key\n: \nsession_id: explicit456" in stderr_preview + + +def test_nonzero_exit_previews_redact_quoted_authorization_fields( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("'authorization': Bearer quoted-auth-secret session_id=auth456", file=sys.stderr) +print('"proxy_authorization": Basic quoted-proxy-secret session_id=proxy456', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + stderr_preview = exc_info.value.details["stderr_preview"] + + assert "quoted-auth-secret" not in stderr_preview + assert "quoted-proxy-secret" not in stderr_preview + assert "'authorization': session_id=auth456" in stderr_preview + assert '"proxy_authorization": session_id=proxy456' in stderr_preview + + +def test_nonzero_exit_previews_redact_multiple_authorization_fields_on_one_line( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("Authorization: Bearer first-secret Proxy-Authorization: Basic second-secret session_id=auth789", file=sys.stderr) +print("Authorization: Bearer third-secret authorization=Basic fourth-secret session_id=auth790", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + stderr_preview = exc_info.value.details["stderr_preview"] + + for secret in ("first-secret", "second-secret", "third-secret", "fourth-secret"): + assert secret not in stderr_preview + assert ( + "Authorization: Proxy-Authorization: session_id=auth789" + in stderr_preview + ) + assert ( + "Authorization: authorization= session_id=auth790" + in stderr_preview + ) + + +def test_nonzero_exit_previews_redact_explicit_yaml_with_indented_comment( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("? api_key\\n # indented note\\n: indented-explicit-secret\\nsession_id: explicit789", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + stderr_preview = exc_info.value.details["stderr_preview"] + + assert "indented-explicit-secret" not in stderr_preview + assert "? api_key\n: \nsession_id: explicit789" in stderr_preview + + +def test_nonzero_exit_previews_redact_tagged_and_yaml_escaped_explicit_keys( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print('{"api\\\\x5fkey":"escaped-hex-json","session_id":"escapedhex123"}', file=sys.stderr) +print("? !!str api_key\\n: tagged-explicit-secret\\nsession_id: tagged-explicit123", file=sys.stderr) +print('- ? &cred "api\\\\x5fkey"\\n : anchored-escaped-secret\\nsession_id: anchored-explicit123', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stderr_preview = exc_info.value.details["stderr_preview"] + assert "escaped-hex-json" not in stderr_preview + assert "tagged-explicit-secret" not in stderr_preview + assert "anchored-escaped-secret" not in stderr_preview + assert r'{"api\x5fkey":"","session_id":"escapedhex123"}' in stderr_preview + assert "? !!str api_key\n: \nsession_id: tagged-explicit123" in stderr_preview + assert ( + '- ? &cred "api\\x5fkey"\n : \nsession_id: anchored-explicit123' + in stderr_preview + ) + + +def test_nonzero_exit_diagnostic_previews_redact_repo_env_json_and_parameterized_auth( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("AIHUBMIX_KEY=stdout-short session_id=stdout123") +print("LONGBRIDGE_APP_KEY=stderr-short session_id=bridge123", file=sys.stderr) +print("NTFY_URL=https://ntfy.sh/private-topic session_id=ntfy123", file=sys.stderr) +print("PUSHOVER_USER_KEY=notify-short session_id=push123", file=sys.stderr) +print("proxyAuthorization: Basic proxy-short session_id=proxy123", file=sys.stderr) +print('{', file=sys.stderr) +print(' "api_key":', file=sys.stderr) +print(' "json-short",', file=sys.stderr) +print(' "session_id": "json123"', file=sys.stderr) +print('}', file=sys.stderr) +print("Authorization: AWS4-HMAC-SHA256 Credential=AKIA/20240101/test/aws4_request, SignedHeaders=host;x-amz-date, Signature=tiny-secret session_id=aws123", file=sys.stderr) +print('Authorization: Signature keyId="client",algorithm="hmac-sha256",signature="sig-short" token_budget=1000', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + assert "stdout-short" not in stdout_preview + assert "stderr-short" not in stderr_preview + assert "https://ntfy.sh/private-topic" not in stderr_preview + assert "notify-short" not in stderr_preview + assert "proxy-short" not in stderr_preview + assert "json-short" not in stderr_preview + assert "tiny-secret" not in stderr_preview + assert "sig-short" not in stderr_preview + assert "AIHUBMIX_KEY= session_id=stdout123" in stdout_preview + assert "LONGBRIDGE_APP_KEY= session_id=bridge123" in stderr_preview + assert "NTFY_URL= session_id=ntfy123" in stderr_preview + assert "PUSHOVER_USER_KEY= session_id=push123" in stderr_preview + assert "proxyAuthorization: session_id=proxy123" in stderr_preview + assert '"api_key":\n ""' in stderr_preview + assert '"session_id": "json123"' in stderr_preview + assert "Authorization: session_id=aws123" in stderr_preview + assert "Authorization: token_budget=1000" in stderr_preview + + +def test_nonzero_exit_previews_redact_sensitive_env_command_substitutions( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("OPENAI_FOO=$(printf %s stdout-secret) session_id=stdout789") +print("export OPENAI_API_KEY=$(printf '%s %s' stderr tiny-secret) session_id=stderr789", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + + for secret in ("stdout-secret", "tiny-secret"): + assert secret not in f"{stdout_preview}\n{stderr_preview}" + assert "OPENAI_FOO= session_id=stdout789" in stdout_preview + assert "export OPENAI_API_KEY= session_id=stderr789" in stderr_preview + + +def test_redact_diagnostic_text_multi_segment_shell_command_substitutions( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("A=$(echo OPENAI_API_KEY=sk-12345);B=ok; tail $(printenv SECRET_TOKEN)") +print("safe=$(date); echo $(ls)", file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + + for secret in ("OPENAI_API_KEY", "sk-12345", "SECRET_TOKEN"): + assert secret not in f"{stdout_preview}\n{stderr_preview}" + assert "" in stdout_preview + assert stderr_preview.count("") == 0 + + +@pytest.mark.parametrize( + "diagnostic, must_keep, must_redact", + [ + # OR-COR-7c0a5d41: export SENSITIVE=$(...) form must not drop + # trailing non-sensitive fields like session_id when the + # substitution body contains a sensitive uppercase token. + # NOTE: the LHS assignment name (``OPENAI_API_KEY``) is itself + # sensitive and intentionally displayed as the assignment + # target — the leakage vector we guard here is the inner + # command-substitution secret (``SECRET_TOKEN`` / ``sk-12345``) + # and the *trailing* non-sensitive fields that the second-pass + # scan was eaten by overlapping spans. + ( + "export OPENAI_API_KEY=$(printenv SECRET_TOKEN) session_id=dup1 token_budget=1000", + ["session_id=dup1", "token_budget=1000"], + ["SECRET_TOKEN", "printenv SECRET_TOKEN"], + ), + ( + "export OPENAI_API_KEY=$(echo OPENAI_API_KEY=sk-12345) session_id=dup3", + ["session_id=dup3"], + ["sk-12345"], + ), + # Non-export form must continue to preserve trailing fields. + ( + "OPENAI_API_KEY=$(printenv SECRET_TOKEN) session_id=dup4 token_budget=2000", + ["session_id=dup4", "token_budget=2000"], + ["SECRET_TOKEN", "printenv SECRET_TOKEN"], + ), + ], +) +def test_redact_diagnostic_text_export_env_preserves_trailing_fields( + diagnostic: str, must_keep: list[str], must_redact: list[str] +) -> None: + """``export SENSITIVE_ENV=$(printenv OTHER_SECRET) session_id=...`` + must redact the secret substitution while preserving the trailing + non-sensitive diagnostics (session_id, token_budget, …). Regression + for OR-COR-7c0a5d41. + """ + redacted = redact_diagnostic_text(diagnostic, limit=1000) + for secret in must_redact: + assert secret not in redacted, f"leaked {secret!r}: {redacted!r}" + for kept in must_keep: + assert kept in redacted, f"dropped {kept!r}: {redacted!r}" + # Sanity: the LHS assignment name (e.g. ``OPENAI_API_KEY=``) is the + # redaction *target* marker and should remain visible so users + # can see which env was scrubbed. + assert "" in redacted + + +def test_nonzero_exit_previews_redact_json_auth_and_embedded_assignments( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print("password: correct horse=staple session_id=abc123") +print('{"authorization":"Bearer tiny-secret","session_id":"auth123"}', file=sys.stderr) +print('{"cookie":"session=cookie-secret","session_id":"cookie123"}', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stdout_preview = exc_info.value.details["stdout_preview"] + stderr_preview = exc_info.value.details["stderr_preview"] + assert stdout_preview == "password: \n" + assert "tiny-secret" not in stderr_preview + assert "cookie-secret" not in stderr_preview + assert '{"authorization":"","session_id":"auth123"}' in stderr_preview + assert '{"cookie":"","session_id":"cookie123"}' in stderr_preview + + +def test_nonzero_exit_previews_redact_flow_style_unquoted_multitoken_secrets( + tmp_path: Path, +) -> None: + backend = _backend( + tmp_path, + """ +import sys +print('{"password": correct horse battery staple, "session_id": "json123"}', file=sys.stderr) +print('{"api_key": correct horse, "session_id": "api123"}', file=sys.stderr) +print("{'password': correct horse battery staple, 'session_id': 'yaml123'}", file=sys.stderr) +print('{password: correct horse, session_id: yaml456}', file=sys.stderr) +raise SystemExit(2) +""", + ) + + with pytest.raises(GenerationError) as exc_info: + backend.generate("prompt", {}) + + assert exc_info.value.error_code is GenerationErrorCode.NON_ZERO_EXIT + stderr_preview = exc_info.value.details["stderr_preview"] + assert "correct horse" not in stderr_preview + assert "battery staple" not in stderr_preview + assert '{"password": , "session_id": "json123"}' in stderr_preview + assert '{"api_key": , "session_id": "api123"}' in stderr_preview + assert "{'password': , 'session_id': 'yaml123'}" in stderr_preview + assert "{password: , session_id: yaml456}" in stderr_preview + + +def test_preview_diagnostics_from_files_redacts_truncated_quoted_sensitive_scalar( + tmp_path: Path, +) -> None: + long_password = "correct horse battery staple " * 130 + stderr_text = f'password: "{long_password}"\n' + assert len(stderr_text.encode("utf-8")) > local_cli_backend_module._PREVIEW_LIMIT * 4 + + stdout_path = tmp_path / "stdout.txt" + stderr_path = tmp_path / "stderr.txt" + stdout_path.write_text("", encoding="utf-8") + stderr_path.write_text(stderr_text, encoding="utf-8") + + previews = local_cli_backend_module._preview_diagnostics_from_files(stdout_path, stderr_path) + + assert "correct horse battery staple" not in previews["stderr_preview"] + assert previews["stderr_preview"] == "password: " + + def test_effective_local_cli_concurrency_uses_minimum() -> None: assert effective_local_cli_concurrency(_config()) == 1 assert effective_local_cli_concurrency(