fix: 过滤低质量股票新闻结果 (#1674)

* fix: filter low-quality stock news results

* fix: filter adult spam news results

* fix(review-feedback-1674): Require content evidence before dropping URL-only hits and Match

* fix(review-feedback-1674): Move admission filtering before dimension limiting

* fix(review-feedback-1674): Match app/download terms as tokens before filtering

* fix(review-feedback-1674): Recognize Chinese numeric ratings in app-page filter

* fix(review-feedback-1674): 收敛后再合入

* fix(review-feedback-1674): 收敛后再合入

* fix: tighten stock news admission signals

* fix: narrow noisy news admission heuristics

* fix: handle app metric news admission

* fix: require context for app listing spam

* fix: preserve app metric news results

* fix: require strong app spam evidence

* fix: honor official source labels

* fix: preserve official labels and app metrics

* fix: detect separated adult contact ids

* fix: preserve app metrics and service news

* fix: match alphanumeric adult contacts

* fix: require adult spam solicitation signals

* fix: preserve neutral mobile app metrics

* fix: reduce app and adult false positives
This commit is contained in:
mumu
2026-06-13 15:19:44 +08:00
committed by GitHub
parent 3902670f02
commit 89fab774f6
5 changed files with 1411 additions and 16 deletions

View File

@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
- [改进] #1390 P1 补充 Web decision-signals typed API wrapper 与契约隔离测试,暂不接入 UI。
- [修复] #1390 收紧建议动作 legacy fallback英文 `not to ...``avoid selling/reducing/trimming ...` 等否定/回避表达不再误判为买卖动作Web 旧记录不再把中文金融上下文、`buy or sell`、多 guard 歧义文本或 `buyback` / `buy-back` / `buy back` / `selloff` / `sell-off` / `sell off` 等英文复合词渲染成 action badge并在有结构化 `action` 时让回测/历史趋势等入口按界面语言显示 action 标签。
- [改进] 完善运行时日志上下文,补充 logger name、触发来源、市场统计与实时行情预取链路状态便于排查调度、API、Bot 和数据源降级路径。
- [修复] #1662 股票新闻与多维情报搜索在相关度排序后新增域名无关的准入过滤,剔除下载/安装包/应用评分页及成人/招嫖服务垃圾页,并在同批已有有效标的/行业候选时移除 `score=0` 背景填充项,避免无关或低质量页面进入 Prompt、Agent 工具输出和历史情报缓存。
- [新功能] 新增分析任务与历史报告运行流快照 API提供 lanes、nodes、edges、events、summary 等统一契约,并从任务队列、运行诊断和 AnalysisContextPack overview 构建脱敏数据流/信息流。
- [新功能] Web 端为活跃任务、历史报告和大盘复盘报告补充运行流视图入口,支持查看运行摘要、拓扑节点、事件流和基础排障详情。
- [修复] 修复历史报告运行流快照在混合时区事件时间戳下返回 500 的问题。

View File

@@ -352,6 +352,8 @@ daily_stock_analysis/
排序策略为先按类别优先级direct > sector > macro排序再按语言偏好中文优先再按分数排序因此当同一时窗内存在明确标的命中的新闻时会优先展示。
排序后还会执行一层域名无关的准入过滤:明显的下载/安装包/应用评分页、成人/招嫖服务垃圾页会被剔除;当同一批次已经存在直接标的或有分数的行业/市场候选时,`score=0` 的背景填充项不会进入 `news_context`、Agent 工具输出或历史情报缓存。该规则不内置具体网站黑名单,避免靠穷举域名维护。
调试入口:
- 每条返回会保留 `relevance_score` / `relevance_category` / `relevance_reasons` 元数据,最终 `to_text()` 与情报上下文会附带对应「关联度」说明;

View File

@@ -1407,6 +1407,8 @@ A: Check if Actions is enabled, and if cron expression is correct (note it's UTC
- If Agent asks for more days than the local cache contains, the tool returns the available records and marks the response with `partial_cache=true`, `requested_days`, and `actual_records`.
- When the cache is missing or stale, the tool keeps the original data-source fetch path; successful fetches are written back to `stock_daily` on a best-effort basis, and write failures do not block the Agent response.
- `search_stock_news` and `search_comprehensive_intel` persist successful results to `news_intel` on a best-effort basis, reusing the existing URL / fallback-key deduplication logic.
- Stock news search now applies a domain-agnostic admission filter after relevance ranking: obvious download/install/app-rating pages and adult/escort spam pages are removed, and zero-score filler results are dropped when the same batch already has direct-stock or scored sector/market candidates. This is not a hard-coded website blocklist.
- This admission-filter change is isolated to retrieval post-filtering and does not alter model names, provider settings, Base URL, LiteLLM route semantics, or runtime config migration/cleanup behavior.
- `get_realtime_quote` does not use `stock_daily` as a realtime-quote cache and does not write intraday quotes into the daily-bar table; realtime quote caching should use a dedicated realtime store if needed.
## Agent Event Monitor

View File

@@ -2163,6 +2163,96 @@ class SearchService:
"cninfo", "sse.com", "szse.cn", "hkexnews", "sec.gov", "nasdaq.com",
"nyse.com", "上交所", "深交所", "港交所", "证券交易所",
)
_OFFICIAL_SOURCE_HOSTS = (
"cninfo.com.cn", "sse.com", "sse.com.cn", "szse.cn", "hkexnews.hk",
"sec.gov", "nasdaq.com", "nyse.com",
)
_OFFICIAL_SOURCE_LABELS = (
"cninfo", "hkexnews", "巨潮资讯", "巨潮资讯网",
"上交所", "深交所", "港交所", "证券交易所",
"上海证券交易所", "深圳证券交易所", "香港交易所", "香港联合交易所",
)
_LOW_QUALITY_DOWNLOAD_ACTION_TERMS = (
"下载", "安装", "下载安装", "下载安装到手机", "下载链接",
"免费下载", "客户端下载", "应用下载", "官方app下载",
"安装包", "apk", "download", "install", "installer",
)
_LOW_QUALITY_DOWNLOAD_INTENT_TERMS = (
"安装包", "客户端下载", "应用下载", "下载安装", "下载安装到手机",
"下载链接", "免费下载", "旧版下载", "极速版下载", "官方app下载",
)
_LOW_QUALITY_APP_CONTEXT_TERMS = (
"好评", "评分", "版本", "大小", "适用年龄", "开发者", "应用",
"ratings", "reviews", "stars", "version", "developer", "package",
)
_LOW_QUALITY_APP_METADATA_TERMS = (
"版本", "大小", "适用年龄", "开发者", "应用", "应用商店",
"安卓版", "苹果版", "官方版", "最新版", "version", "developer",
"package", "mobile app",
)
_LOW_QUALITY_APP_PAGE_DETAIL_TERMS = (
"客户端", "安卓版", "苹果版", "官方版", "最新版", "应用商店",
"下载安装到手机", "一键下载", "旧版下载", "极速版下载",
)
_LOW_QUALITY_FILE_SIZE_RE = re.compile(r"\b\d+(?:\.\d+)?\s*(?:kb|mb|gb)\b", re.IGNORECASE)
_LOW_QUALITY_RATING_RE = re.compile(
r"(?:\d{1,3}\s*%\s*好评|好评率|用户评分|"
r"(?:用户)?评分\s*[:]?\s*(?:10|[0-9])(?:\.\d{1,2})?|"
r"\b\d(?:\.\d)?\s*(?:stars?|ratings?|reviews?)\b)",
re.IGNORECASE,
)
_LOW_QUALITY_URL_RE = re.compile(
r"(?:^|[/_.=-])(?:download|downloads|apk|ipa|exe|dmg|installer|"
r"software|soft|game|games|app|apps|package)(?:$|[/_.?&=-])",
re.IGNORECASE,
)
_BUSINESS_APP_METRIC_RE = re.compile(
r"(?:(?:下载量|安装量|装机量|应用下载|应用安装|app下载|app安装).{0,12}"
r"(?:增长|同比|环比|上升|增加|提升|突破|达到|达|超过|超|累计|接近|保持|创新高|下降|下滑|减少|回落|放缓|持平|承压|低迷)|"
r"(?:增长|同比|环比|上升|增加|提升|突破|达到|达|超过|超|累计|接近|保持|创新高|下降|下滑|减少|回落|放缓|持平|承压|低迷)"
r".{0,12}(?:下载量|安装量|装机量|应用下载|应用安装|app下载|app安装)|"
r"\b(?:downloads?|installs?)\b.{0,16}"
r"\b(?:grew|growth|rose|increase|increased|surged|reached|reach|reaches|"
r"hit|hits|topped|totaled|totalled|exceeded|exceeds|surpassed|surpasses|"
r"fell|fall|declined|decline|decreased|dropped|drop|slowed|flat|weakened)\b|"
r"\b(?:grew|growth|rose|increase|increased|surged|reached|reach|reaches|"
r"hit|hits|topped|totaled|totalled|exceeded|exceeds|surpassed|surpasses|"
r"fell|fall|declined|decline|decreased|dropped|drop|slowed|flat|weakened)\b"
r".{0,16}\b(?:downloads?|installs?)\b)",
re.IGNORECASE,
)
_ADULT_SERVICE_SPAM_STRONG_TERMS = (
"上门特殊服务", "同城约", "约炮", "援交", "楼凤", "外围女",
"外围服务", "包夜", "大保健", "莞式", "推油",
"成人服务", "adult service", "escort service",
"sex service", "call girl",
)
_ADULT_SERVICE_SPAM_AMBIGUOUS_TERMS = (
"全套服务", "色情",
)
_ADULT_SERVICE_SPAM_CONTEXT_TERMS = (
"小姐", "上门", "预约", "同城", "按摩", "保健", "足浴", "桑拿",
"会所", "技师", "全套", "套餐", "vip",
)
_ADULT_SERVICE_SPAM_CONTACT_RE = re.compile(
r"(?:^|[^a-z0-9])(?:yue|vx|wx|qq|wechat|weixin|微信号?|微[信讯]|"
r"电话|手机|联系电话|tel|phone)"
r"[-_:\s]*[a-z0-9][a-z0-9_-]{2,}(?:[^a-z0-9]|$)",
re.IGNORECASE,
)
_ADULT_SERVICE_SPAM_CONTACT_CONTEXT_TERMS = (
"小姐", "上门", "同城", "预约",
"全套", "包夜", "大保健", "推油",
"约炮", "援交", "成人", "色情",
)
_ADULT_SERVICE_REMEDIATION_TERMS = (
"治理", "整治", "下架", "处罚", "监管", "打击", "清理",
"封禁", "整改", "内容安全", "低俗内容", "平台风险",
)
_ADULT_SERVICE_SOLICITATION_TERMS = (
"上门", "同城", "预约", "套餐", "包夜", "大保健",
"推油", "联系", "咨询", "加微信", "加qq", "vip",
)
def __init__(
self,
@@ -2635,6 +2725,229 @@ class SearchService:
lower = (text or "").lower()
return any(term.lower() in lower for term in terms)
@classmethod
def _contains_any_low_quality_news_term(cls, text: str, terms: Tuple[str, ...]) -> bool:
lower = (text or "").lower()
if not lower:
return False
for term in terms:
normalized_term = term.lower()
if not normalized_term:
continue
if normalized_term.isascii() and re.search(r"[a-z0-9]", normalized_term):
pattern = r"(?<![A-Za-z0-9])" + re.escape(normalized_term) + r"(?![A-Za-z0-9])"
if re.search(pattern, lower):
return True
continue
if normalized_term in lower:
return True
return False
@staticmethod
def _candidate_hostname(value: Any) -> str:
raw = str(value or "").strip().lower()
if not raw or re.search(r"\s", raw):
return ""
parse_value = (
raw
if re.match(r"^[a-z][a-z0-9+.-]*://", raw) or raw.startswith("//")
else f"//{raw}"
)
return (urlparse(parse_value).hostname or "").rstrip(".")
@staticmethod
def _source_resembles_hostname(value: Any) -> bool:
raw = str(value or "").strip().lower()
if not raw or re.search(r"\s", raw):
return False
if re.match(r"^[a-z][a-z0-9+.-]*://", raw) or raw.startswith("//"):
return True
return bool(re.search(r"\.[a-z0-9-]{2,}(?::\d+)?/?$", raw))
@classmethod
def _is_trusted_official_news_source(cls, item: SearchResult) -> bool:
"""Only trust official exemptions from trusted hosts; fallback to labels only when URL host is absent."""
url_host = cls._candidate_hostname(item.url)
source_label = str(item.source or "").strip().lower()
source_host = (
cls._candidate_hostname(item.source)
if cls._source_resembles_hostname(item.source)
else ""
)
if url_host:
# 有 URL 时以 URL 主机为准,避免 source label/host 伪装的官方放行。
return any(
url_host == official_host or url_host.endswith(f".{official_host}")
for official_host in cls._OFFICIAL_SOURCE_HOSTS
)
if source_host:
return any(
source_host == official_host or source_host.endswith(f".{official_host}")
for official_host in cls._OFFICIAL_SOURCE_HOSTS
)
return source_label in cls._OFFICIAL_SOURCE_LABELS
@classmethod
def _has_low_quality_news_page_signal(cls, item: SearchResult) -> bool:
"""Detect app/download/listing pages without relying on a domain blocklist."""
content_text = " ".join(filter(None, [item.title, item.snippet])).lower()
parsed_url = urlparse(item.url or "")
url_surface = unquote(
" ".join(filter(None, [parsed_url.netloc, parsed_url.path, parsed_url.query]))
).lower()
has_app_context = cls._contains_any_low_quality_news_term(
content_text,
cls._LOW_QUALITY_APP_CONTEXT_TERMS,
)
has_app_metadata = cls._contains_any_low_quality_news_term(
content_text,
cls._LOW_QUALITY_APP_METADATA_TERMS,
)
has_download_action = cls._contains_any_low_quality_news_term(
content_text,
cls._LOW_QUALITY_DOWNLOAD_ACTION_TERMS,
)
has_download_intent = cls._contains_any_low_quality_news_term(
content_text,
cls._LOW_QUALITY_DOWNLOAD_INTENT_TERMS,
)
has_app_page_detail = cls._contains_any_low_quality_news_term(
content_text,
cls._LOW_QUALITY_APP_PAGE_DETAIL_TERMS,
)
has_file_size = bool(cls._LOW_QUALITY_FILE_SIZE_RE.search(content_text))
has_rating = bool(cls._LOW_QUALITY_RATING_RE.search(content_text))
has_url_signal = bool(cls._LOW_QUALITY_URL_RE.search(url_surface))
has_business_app_metric = bool(cls._BUSINESS_APP_METRIC_RE.search(content_text))
has_app_listing_detail = (
has_file_size
or has_rating
or cls._contains_any_low_quality_news_term(
content_text,
(
"版本", "适用年龄", "开发者", "应用商店", "安卓版",
"苹果版", "官方版", "最新版", "version", "developer",
"package",
),
)
)
has_strong_app_page_evidence = (
has_app_listing_detail
and (
has_url_signal
or has_download_intent
or (has_download_action and has_app_metadata)
)
)
has_business_app_metric_only = (
has_business_app_metric
and not has_strong_app_page_evidence
)
has_app_listing_context = (
not has_business_app_metric_only
and has_app_context
and has_app_metadata
and (has_download_action or has_download_intent)
and (has_file_size or has_rating)
)
has_content_download_page = (
not has_business_app_metric_only
and (
(has_download_intent and (has_app_page_detail or has_file_size or has_rating))
or (has_download_action and (has_app_metadata or has_file_size))
)
)
has_url_backed_download_page = (
not has_business_app_metric_only
and has_url_signal
and (
has_file_size
or has_download_intent
or (has_download_action and has_app_metadata)
or (has_app_metadata and has_rating)
)
)
return (
has_content_download_page
or has_app_listing_context
or has_url_backed_download_page
)
@classmethod
def _has_adult_service_spam_news_page_signal(cls, item: SearchResult) -> bool:
"""Detect adult-service spam by content signals instead of domain names."""
combined_text = " ".join(
filter(None, [item.title, item.snippet, item.source, item.url])
).lower()
if cls._contains_any_news_term(
combined_text,
cls._ADULT_SERVICE_SPAM_STRONG_TERMS,
):
return True
has_contact_signal = bool(cls._ADULT_SERVICE_SPAM_CONTACT_RE.search(combined_text))
has_remediation_context = cls._contains_any_news_term(
combined_text,
cls._ADULT_SERVICE_REMEDIATION_TERMS,
)
if has_remediation_context and not has_contact_signal:
return False
if (
"外围" in combined_text
and cls._contains_any_news_term(
combined_text,
("上门", "同城", "约炮", "援交", "包夜", "大保健", "推油", "小姐", "技师"),
)
):
return True
context_hits = sum(
1
for term in cls._ADULT_SERVICE_SPAM_CONTEXT_TERMS
if term.lower() in combined_text
)
has_service_anchor = cls._contains_any_news_term(
combined_text,
("小姐", "按摩", "足浴", "桑拿", "会所", "技师"),
)
has_adult_specific_anchor = cls._contains_any_news_term(
combined_text,
(
"小姐", "约炮", "援交", "楼凤", "外围", "包夜",
"大保健", "莞式", "推油", "成人", "色情",
),
)
if has_contact_signal:
return has_adult_specific_anchor and cls._contains_any_news_term(
combined_text,
cls._ADULT_SERVICE_SPAM_CONTACT_CONTEXT_TERMS,
)
has_solicitation_signal = cls._contains_any_news_term(
combined_text,
cls._ADULT_SERVICE_SOLICITATION_TERMS,
)
has_ambiguous_adult_phrase = cls._contains_any_news_term(
combined_text,
cls._ADULT_SERVICE_SPAM_AMBIGUOUS_TERMS,
)
if has_ambiguous_adult_phrase:
return has_service_anchor and has_solicitation_signal
return (
has_adult_specific_anchor
and has_service_anchor
and has_solicitation_signal
and context_hits >= 3
)
@classmethod
def _score_news_relevance(
cls,
@@ -2727,7 +3040,7 @@ class SearchService:
direct_signal += 12
add_reason("命中公告/财报/交易等公司事件词")
if cls._contains_any_news_term(f"{source} {url}", cls._OFFICIAL_SOURCE_TERMS):
if cls._is_trusted_official_news_source(item):
score += 8
add_reason("来源接近公告或交易所渠道")
@@ -2824,6 +3137,72 @@ class SearchService:
search_time=response.search_time,
)
@classmethod
def _filter_ranked_news_for_context(
cls,
response: SearchResponse,
*,
log_scope: str,
) -> SearchResponse:
"""Drop obvious non-news pages and zero-relevance fillers from ranked results."""
if not response.success or not response.results:
return response
candidates: List[SearchResult] = []
dropped_low_quality = 0
dropped_adult_spam = 0
dropped_zero_relevance = 0
for item in response.results:
is_official_source = cls._is_trusted_official_news_source(item)
if (
not is_official_source
and cls._has_low_quality_news_page_signal(item)
):
dropped_low_quality += 1
continue
if (
not is_official_source
and cls._has_adult_service_spam_news_page_signal(item)
):
dropped_adult_spam += 1
continue
candidates.append(item)
meaningful_candidates = [
item
for item in candidates
if item.relevance_category == cls._DIRECT_NEWS_CATEGORY
or (item.relevance_score or 0) > 0
]
if meaningful_candidates:
dropped_zero_relevance = len(candidates) - len(meaningful_candidates)
filtered_results = meaningful_candidates
else:
filtered_results = candidates
if dropped_low_quality or dropped_adult_spam or dropped_zero_relevance:
logger.info(
"[新闻准入] %s: provider=%s, total=%s, kept=%s, "
"drop_low_quality=%s, drop_adult_spam=%s, drop_zero_relevance=%s",
log_scope,
response.provider,
len(response.results),
len(filtered_results),
dropped_low_quality,
dropped_adult_spam,
dropped_zero_relevance,
)
return SearchResponse(
query=response.query,
results=filtered_results,
provider=response.provider,
success=response.success,
error_message=response.error_message,
search_time=response.search_time,
)
@classmethod
def _news_relevance_stats(
cls,
@@ -3343,16 +3722,6 @@ class SearchService:
log_scope=f"{stock_code}:{provider.name}:stock_news",
)
had_provider_success = had_provider_success or bool(response.success)
filtered_count = len(filtered_response.results or []) if filtered_response.success else 0
self._record_news_search_run(
provider=provider.name,
operation="search_stock_news",
success=bool(filtered_response.success and filtered_response.results),
latency_ms=self._elapsed_ms(started_at),
record_count=filtered_count,
error_type=None if filtered_count else "NoUsableNews",
error_message=None if filtered_count else (response.error_message or "过滤后无有效新闻"),
)
if filtered_response.success and filtered_response.results:
language_response, _preferred_count = self._prioritize_news_language(
@@ -3367,10 +3736,33 @@ class SearchService:
max_results=provider_max_results,
log_scope=f"{stock_code}:{provider.name}:stock_news",
)
limited_response = self._limit_search_response(
admitted_response = self._filter_ranked_news_for_context(
ranked_response,
log_scope=f"{stock_code}:{provider.name}:stock_news",
)
limited_response = self._limit_search_response(
admitted_response,
max_results=max_results,
)
admitted_count = len(limited_response.results or [])
self._record_news_search_run(
provider=provider.name,
operation="search_stock_news",
success=bool(limited_response.success and limited_response.results),
latency_ms=self._elapsed_ms(started_at),
record_count=admitted_count,
error_type=None if admitted_count else "NoUsableNews",
error_message=None if admitted_count else (
response.error_message or "过滤后无有效新闻"
),
)
if not admitted_count:
logger.info(
"%s 搜索成功但准入过滤后无有效新闻,继续尝试下一引擎",
provider.name,
)
continue
stats = self._news_relevance_stats(
limited_response,
prefer_chinese=prefer_chinese,
@@ -3424,6 +3816,18 @@ class SearchService:
provider.name,
)
else:
filtered_count = len(filtered_response.results or []) if filtered_response.success else 0
self._record_news_search_run(
provider=provider.name,
operation="search_stock_news",
success=bool(filtered_response.success and filtered_response.results),
latency_ms=self._elapsed_ms(started_at),
record_count=filtered_count,
error_type=None if filtered_count else "NoUsableNews",
error_message=None if filtered_count else (
response.error_message or "过滤后无有效新闻"
),
)
if response.success and not filtered_response.results:
logger.info(
"%s 搜索成功但过滤后无有效新闻,继续尝试下一引擎",
@@ -3727,9 +4131,17 @@ class SearchService:
stock_code=stock_code,
stock_name=stock_name,
prefer_chinese=self._should_prefer_chinese_news(stock_code, stock_name),
max_results=target_per_dimension,
max_results=provider_max_results,
log_scope=f"{stock_code}:{provider.name}:{dim['name']}:rank",
)
filtered_response = self._filter_ranked_news_for_context(
filtered_response,
log_scope=f"{stock_code}:{provider.name}:{dim['name']}:admission",
)
filtered_response = self._limit_search_response(
filtered_response,
max_results=target_per_dimension,
)
results[dim['name']] = filtered_response
search_count += 1

File diff suppressed because it is too large Load Diff