mirror of
https://github.com/ZhuLinsen/daily_stock_analysis
synced 2026-09-20 10:53:33 +08:00
fix: 过滤低质量股票新闻结果 (#1674)
* fix: filter low-quality stock news results * fix: filter adult spam news results * fix(review-feedback-1674): Require content evidence before dropping URL-only hits and Match * fix(review-feedback-1674): Move admission filtering before dimension limiting * fix(review-feedback-1674): Match app/download terms as tokens before filtering * fix(review-feedback-1674): Recognize Chinese numeric ratings in app-page filter * fix(review-feedback-1674): 收敛后再合入 * fix(review-feedback-1674): 收敛后再合入 * fix: tighten stock news admission signals * fix: narrow noisy news admission heuristics * fix: handle app metric news admission * fix: require context for app listing spam * fix: preserve app metric news results * fix: require strong app spam evidence * fix: honor official source labels * fix: preserve official labels and app metrics * fix: detect separated adult contact ids * fix: preserve app metrics and service news * fix: match alphanumeric adult contacts * fix: require adult spam solicitation signals * fix: preserve neutral mobile app metrics * fix: reduce app and adult false positives
This commit is contained in:
@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||
- [改进] #1390 P1 补充 Web decision-signals typed API wrapper 与契约隔离测试,暂不接入 UI。
|
||||
- [修复] #1390 收紧建议动作 legacy fallback:英文 `not to ...` 与 `avoid selling/reducing/trimming ...` 等否定/回避表达不再误判为买卖动作,Web 旧记录不再把中文金融上下文、`buy or sell`、多 guard 歧义文本或 `buyback` / `buy-back` / `buy back` / `selloff` / `sell-off` / `sell off` 等英文复合词渲染成 action badge,并在有结构化 `action` 时让回测/历史趋势等入口按界面语言显示 action 标签。
|
||||
- [改进] 完善运行时日志上下文,补充 logger name、触发来源、市场统计与实时行情预取链路状态,便于排查调度、API、Bot 和数据源降级路径。
|
||||
- [修复] #1662 股票新闻与多维情报搜索在相关度排序后新增域名无关的准入过滤,剔除下载/安装包/应用评分页及成人/招嫖服务垃圾页,并在同批已有有效标的/行业候选时移除 `score=0` 背景填充项,避免无关或低质量页面进入 Prompt、Agent 工具输出和历史情报缓存。
|
||||
- [新功能] 新增分析任务与历史报告运行流快照 API,提供 lanes、nodes、edges、events、summary 等统一契约,并从任务队列、运行诊断和 AnalysisContextPack overview 构建脱敏数据流/信息流。
|
||||
- [新功能] Web 端为活跃任务、历史报告和大盘复盘报告补充运行流视图入口,支持查看运行摘要、拓扑节点、事件流和基础排障详情。
|
||||
- [修复] 修复历史报告运行流快照在混合时区事件时间戳下返回 500 的问题。
|
||||
|
||||
@@ -352,6 +352,8 @@ daily_stock_analysis/
|
||||
|
||||
排序策略为:先按类别优先级(direct > sector > macro)排序,再按语言偏好(中文优先)再按分数排序,因此当同一时窗内存在明确标的命中的新闻时会优先展示。
|
||||
|
||||
排序后还会执行一层域名无关的准入过滤:明显的下载/安装包/应用评分页、成人/招嫖服务垃圾页会被剔除;当同一批次已经存在直接标的或有分数的行业/市场候选时,`score=0` 的背景填充项不会进入 `news_context`、Agent 工具输出或历史情报缓存。该规则不内置具体网站黑名单,避免靠穷举域名维护。
|
||||
|
||||
调试入口:
|
||||
|
||||
- 每条返回会保留 `relevance_score` / `relevance_category` / `relevance_reasons` 元数据,最终 `to_text()` 与情报上下文会附带对应「关联度」说明;
|
||||
|
||||
@@ -1407,6 +1407,8 @@ A: Check if Actions is enabled, and if cron expression is correct (note it's UTC
|
||||
- If Agent asks for more days than the local cache contains, the tool returns the available records and marks the response with `partial_cache=true`, `requested_days`, and `actual_records`.
|
||||
- When the cache is missing or stale, the tool keeps the original data-source fetch path; successful fetches are written back to `stock_daily` on a best-effort basis, and write failures do not block the Agent response.
|
||||
- `search_stock_news` and `search_comprehensive_intel` persist successful results to `news_intel` on a best-effort basis, reusing the existing URL / fallback-key deduplication logic.
|
||||
- Stock news search now applies a domain-agnostic admission filter after relevance ranking: obvious download/install/app-rating pages and adult/escort spam pages are removed, and zero-score filler results are dropped when the same batch already has direct-stock or scored sector/market candidates. This is not a hard-coded website blocklist.
|
||||
- This admission-filter change is isolated to retrieval post-filtering and does not alter model names, provider settings, Base URL, LiteLLM route semantics, or runtime config migration/cleanup behavior.
|
||||
- `get_realtime_quote` does not use `stock_daily` as a realtime-quote cache and does not write intraday quotes into the daily-bar table; realtime quote caching should use a dedicated realtime store if needed.
|
||||
|
||||
## Agent Event Monitor
|
||||
|
||||
@@ -2163,6 +2163,96 @@ class SearchService:
|
||||
"cninfo", "sse.com", "szse.cn", "hkexnews", "sec.gov", "nasdaq.com",
|
||||
"nyse.com", "上交所", "深交所", "港交所", "证券交易所",
|
||||
)
|
||||
_OFFICIAL_SOURCE_HOSTS = (
|
||||
"cninfo.com.cn", "sse.com", "sse.com.cn", "szse.cn", "hkexnews.hk",
|
||||
"sec.gov", "nasdaq.com", "nyse.com",
|
||||
)
|
||||
_OFFICIAL_SOURCE_LABELS = (
|
||||
"cninfo", "hkexnews", "巨潮资讯", "巨潮资讯网",
|
||||
"上交所", "深交所", "港交所", "证券交易所",
|
||||
"上海证券交易所", "深圳证券交易所", "香港交易所", "香港联合交易所",
|
||||
)
|
||||
_LOW_QUALITY_DOWNLOAD_ACTION_TERMS = (
|
||||
"下载", "安装", "下载安装", "下载安装到手机", "下载链接",
|
||||
"免费下载", "客户端下载", "应用下载", "官方app下载",
|
||||
"安装包", "apk", "download", "install", "installer",
|
||||
)
|
||||
_LOW_QUALITY_DOWNLOAD_INTENT_TERMS = (
|
||||
"安装包", "客户端下载", "应用下载", "下载安装", "下载安装到手机",
|
||||
"下载链接", "免费下载", "旧版下载", "极速版下载", "官方app下载",
|
||||
)
|
||||
_LOW_QUALITY_APP_CONTEXT_TERMS = (
|
||||
"好评", "评分", "版本", "大小", "适用年龄", "开发者", "应用",
|
||||
"ratings", "reviews", "stars", "version", "developer", "package",
|
||||
)
|
||||
_LOW_QUALITY_APP_METADATA_TERMS = (
|
||||
"版本", "大小", "适用年龄", "开发者", "应用", "应用商店",
|
||||
"安卓版", "苹果版", "官方版", "最新版", "version", "developer",
|
||||
"package", "mobile app",
|
||||
)
|
||||
_LOW_QUALITY_APP_PAGE_DETAIL_TERMS = (
|
||||
"客户端", "安卓版", "苹果版", "官方版", "最新版", "应用商店",
|
||||
"下载安装到手机", "一键下载", "旧版下载", "极速版下载",
|
||||
)
|
||||
_LOW_QUALITY_FILE_SIZE_RE = re.compile(r"\b\d+(?:\.\d+)?\s*(?:kb|mb|gb)\b", re.IGNORECASE)
|
||||
_LOW_QUALITY_RATING_RE = re.compile(
|
||||
r"(?:\d{1,3}\s*%\s*好评|好评率|用户评分|"
|
||||
r"(?:用户)?评分\s*[::]?\s*(?:10|[0-9])(?:\.\d{1,2})?|"
|
||||
r"\b\d(?:\.\d)?\s*(?:stars?|ratings?|reviews?)\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LOW_QUALITY_URL_RE = re.compile(
|
||||
r"(?:^|[/_.=-])(?:download|downloads|apk|ipa|exe|dmg|installer|"
|
||||
r"software|soft|game|games|app|apps|package)(?:$|[/_.?&=-])",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_BUSINESS_APP_METRIC_RE = re.compile(
|
||||
r"(?:(?:下载量|安装量|装机量|应用下载|应用安装|app下载|app安装).{0,12}"
|
||||
r"(?:增长|同比|环比|上升|增加|提升|突破|达到|达|超过|超|累计|接近|保持|创新高|下降|下滑|减少|回落|放缓|持平|承压|低迷)|"
|
||||
r"(?:增长|同比|环比|上升|增加|提升|突破|达到|达|超过|超|累计|接近|保持|创新高|下降|下滑|减少|回落|放缓|持平|承压|低迷)"
|
||||
r".{0,12}(?:下载量|安装量|装机量|应用下载|应用安装|app下载|app安装)|"
|
||||
r"\b(?:downloads?|installs?)\b.{0,16}"
|
||||
r"\b(?:grew|growth|rose|increase|increased|surged|reached|reach|reaches|"
|
||||
r"hit|hits|topped|totaled|totalled|exceeded|exceeds|surpassed|surpasses|"
|
||||
r"fell|fall|declined|decline|decreased|dropped|drop|slowed|flat|weakened)\b|"
|
||||
r"\b(?:grew|growth|rose|increase|increased|surged|reached|reach|reaches|"
|
||||
r"hit|hits|topped|totaled|totalled|exceeded|exceeds|surpassed|surpasses|"
|
||||
r"fell|fall|declined|decline|decreased|dropped|drop|slowed|flat|weakened)\b"
|
||||
r".{0,16}\b(?:downloads?|installs?)\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_ADULT_SERVICE_SPAM_STRONG_TERMS = (
|
||||
"上门特殊服务", "同城约", "约炮", "援交", "楼凤", "外围女",
|
||||
"外围服务", "包夜", "大保健", "莞式", "推油",
|
||||
"成人服务", "adult service", "escort service",
|
||||
"sex service", "call girl",
|
||||
)
|
||||
_ADULT_SERVICE_SPAM_AMBIGUOUS_TERMS = (
|
||||
"全套服务", "色情",
|
||||
)
|
||||
_ADULT_SERVICE_SPAM_CONTEXT_TERMS = (
|
||||
"小姐", "上门", "预约", "同城", "按摩", "保健", "足浴", "桑拿",
|
||||
"会所", "技师", "全套", "套餐", "vip",
|
||||
)
|
||||
_ADULT_SERVICE_SPAM_CONTACT_RE = re.compile(
|
||||
r"(?:^|[^a-z0-9])(?:yue|vx|wx|qq|wechat|weixin|微信号?|微[信讯]|"
|
||||
r"电话|手机|联系电话|tel|phone)"
|
||||
r"[-_:\s:]*[a-z0-9][a-z0-9_-]{2,}(?:[^a-z0-9]|$)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_ADULT_SERVICE_SPAM_CONTACT_CONTEXT_TERMS = (
|
||||
"小姐", "上门", "同城", "预约",
|
||||
"全套", "包夜", "大保健", "推油",
|
||||
"约炮", "援交", "成人", "色情",
|
||||
)
|
||||
_ADULT_SERVICE_REMEDIATION_TERMS = (
|
||||
"治理", "整治", "下架", "处罚", "监管", "打击", "清理",
|
||||
"封禁", "整改", "内容安全", "低俗内容", "平台风险",
|
||||
)
|
||||
_ADULT_SERVICE_SOLICITATION_TERMS = (
|
||||
"上门", "同城", "预约", "套餐", "包夜", "大保健",
|
||||
"推油", "联系", "咨询", "加微信", "加qq", "vip",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -2635,6 +2725,229 @@ class SearchService:
|
||||
lower = (text or "").lower()
|
||||
return any(term.lower() in lower for term in terms)
|
||||
|
||||
@classmethod
|
||||
def _contains_any_low_quality_news_term(cls, text: str, terms: Tuple[str, ...]) -> bool:
|
||||
lower = (text or "").lower()
|
||||
if not lower:
|
||||
return False
|
||||
|
||||
for term in terms:
|
||||
normalized_term = term.lower()
|
||||
if not normalized_term:
|
||||
continue
|
||||
if normalized_term.isascii() and re.search(r"[a-z0-9]", normalized_term):
|
||||
pattern = r"(?<![A-Za-z0-9])" + re.escape(normalized_term) + r"(?![A-Za-z0-9])"
|
||||
if re.search(pattern, lower):
|
||||
return True
|
||||
continue
|
||||
if normalized_term in lower:
|
||||
return True
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _candidate_hostname(value: Any) -> str:
|
||||
raw = str(value or "").strip().lower()
|
||||
if not raw or re.search(r"\s", raw):
|
||||
return ""
|
||||
|
||||
parse_value = (
|
||||
raw
|
||||
if re.match(r"^[a-z][a-z0-9+.-]*://", raw) or raw.startswith("//")
|
||||
else f"//{raw}"
|
||||
)
|
||||
return (urlparse(parse_value).hostname or "").rstrip(".")
|
||||
|
||||
@staticmethod
|
||||
def _source_resembles_hostname(value: Any) -> bool:
|
||||
raw = str(value or "").strip().lower()
|
||||
if not raw or re.search(r"\s", raw):
|
||||
return False
|
||||
if re.match(r"^[a-z][a-z0-9+.-]*://", raw) or raw.startswith("//"):
|
||||
return True
|
||||
return bool(re.search(r"\.[a-z0-9-]{2,}(?::\d+)?/?$", raw))
|
||||
|
||||
@classmethod
|
||||
def _is_trusted_official_news_source(cls, item: SearchResult) -> bool:
|
||||
"""Only trust official exemptions from trusted hosts; fallback to labels only when URL host is absent."""
|
||||
url_host = cls._candidate_hostname(item.url)
|
||||
source_label = str(item.source or "").strip().lower()
|
||||
source_host = (
|
||||
cls._candidate_hostname(item.source)
|
||||
if cls._source_resembles_hostname(item.source)
|
||||
else ""
|
||||
)
|
||||
|
||||
if url_host:
|
||||
# 有 URL 时以 URL 主机为准,避免 source label/host 伪装的官方放行。
|
||||
return any(
|
||||
url_host == official_host or url_host.endswith(f".{official_host}")
|
||||
for official_host in cls._OFFICIAL_SOURCE_HOSTS
|
||||
)
|
||||
|
||||
if source_host:
|
||||
return any(
|
||||
source_host == official_host or source_host.endswith(f".{official_host}")
|
||||
for official_host in cls._OFFICIAL_SOURCE_HOSTS
|
||||
)
|
||||
|
||||
return source_label in cls._OFFICIAL_SOURCE_LABELS
|
||||
|
||||
@classmethod
|
||||
def _has_low_quality_news_page_signal(cls, item: SearchResult) -> bool:
|
||||
"""Detect app/download/listing pages without relying on a domain blocklist."""
|
||||
content_text = " ".join(filter(None, [item.title, item.snippet])).lower()
|
||||
parsed_url = urlparse(item.url or "")
|
||||
url_surface = unquote(
|
||||
" ".join(filter(None, [parsed_url.netloc, parsed_url.path, parsed_url.query]))
|
||||
).lower()
|
||||
|
||||
has_app_context = cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
cls._LOW_QUALITY_APP_CONTEXT_TERMS,
|
||||
)
|
||||
has_app_metadata = cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
cls._LOW_QUALITY_APP_METADATA_TERMS,
|
||||
)
|
||||
has_download_action = cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
cls._LOW_QUALITY_DOWNLOAD_ACTION_TERMS,
|
||||
)
|
||||
has_download_intent = cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
cls._LOW_QUALITY_DOWNLOAD_INTENT_TERMS,
|
||||
)
|
||||
has_app_page_detail = cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
cls._LOW_QUALITY_APP_PAGE_DETAIL_TERMS,
|
||||
)
|
||||
has_file_size = bool(cls._LOW_QUALITY_FILE_SIZE_RE.search(content_text))
|
||||
has_rating = bool(cls._LOW_QUALITY_RATING_RE.search(content_text))
|
||||
has_url_signal = bool(cls._LOW_QUALITY_URL_RE.search(url_surface))
|
||||
has_business_app_metric = bool(cls._BUSINESS_APP_METRIC_RE.search(content_text))
|
||||
has_app_listing_detail = (
|
||||
has_file_size
|
||||
or has_rating
|
||||
or cls._contains_any_low_quality_news_term(
|
||||
content_text,
|
||||
(
|
||||
"版本", "适用年龄", "开发者", "应用商店", "安卓版",
|
||||
"苹果版", "官方版", "最新版", "version", "developer",
|
||||
"package",
|
||||
),
|
||||
)
|
||||
)
|
||||
has_strong_app_page_evidence = (
|
||||
has_app_listing_detail
|
||||
and (
|
||||
has_url_signal
|
||||
or has_download_intent
|
||||
or (has_download_action and has_app_metadata)
|
||||
)
|
||||
)
|
||||
has_business_app_metric_only = (
|
||||
has_business_app_metric
|
||||
and not has_strong_app_page_evidence
|
||||
)
|
||||
has_app_listing_context = (
|
||||
not has_business_app_metric_only
|
||||
and has_app_context
|
||||
and has_app_metadata
|
||||
and (has_download_action or has_download_intent)
|
||||
and (has_file_size or has_rating)
|
||||
)
|
||||
has_content_download_page = (
|
||||
not has_business_app_metric_only
|
||||
and (
|
||||
(has_download_intent and (has_app_page_detail or has_file_size or has_rating))
|
||||
or (has_download_action and (has_app_metadata or has_file_size))
|
||||
)
|
||||
)
|
||||
has_url_backed_download_page = (
|
||||
not has_business_app_metric_only
|
||||
and has_url_signal
|
||||
and (
|
||||
has_file_size
|
||||
or has_download_intent
|
||||
or (has_download_action and has_app_metadata)
|
||||
or (has_app_metadata and has_rating)
|
||||
)
|
||||
)
|
||||
|
||||
return (
|
||||
has_content_download_page
|
||||
or has_app_listing_context
|
||||
or has_url_backed_download_page
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _has_adult_service_spam_news_page_signal(cls, item: SearchResult) -> bool:
|
||||
"""Detect adult-service spam by content signals instead of domain names."""
|
||||
combined_text = " ".join(
|
||||
filter(None, [item.title, item.snippet, item.source, item.url])
|
||||
).lower()
|
||||
|
||||
if cls._contains_any_news_term(
|
||||
combined_text,
|
||||
cls._ADULT_SERVICE_SPAM_STRONG_TERMS,
|
||||
):
|
||||
return True
|
||||
has_contact_signal = bool(cls._ADULT_SERVICE_SPAM_CONTACT_RE.search(combined_text))
|
||||
has_remediation_context = cls._contains_any_news_term(
|
||||
combined_text,
|
||||
cls._ADULT_SERVICE_REMEDIATION_TERMS,
|
||||
)
|
||||
if has_remediation_context and not has_contact_signal:
|
||||
return False
|
||||
|
||||
if (
|
||||
"外围" in combined_text
|
||||
and cls._contains_any_news_term(
|
||||
combined_text,
|
||||
("上门", "同城", "约炮", "援交", "包夜", "大保健", "推油", "小姐", "技师"),
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
context_hits = sum(
|
||||
1
|
||||
for term in cls._ADULT_SERVICE_SPAM_CONTEXT_TERMS
|
||||
if term.lower() in combined_text
|
||||
)
|
||||
has_service_anchor = cls._contains_any_news_term(
|
||||
combined_text,
|
||||
("小姐", "按摩", "足浴", "桑拿", "会所", "技师"),
|
||||
)
|
||||
has_adult_specific_anchor = cls._contains_any_news_term(
|
||||
combined_text,
|
||||
(
|
||||
"小姐", "约炮", "援交", "楼凤", "外围", "包夜",
|
||||
"大保健", "莞式", "推油", "成人", "色情",
|
||||
),
|
||||
)
|
||||
if has_contact_signal:
|
||||
return has_adult_specific_anchor and cls._contains_any_news_term(
|
||||
combined_text,
|
||||
cls._ADULT_SERVICE_SPAM_CONTACT_CONTEXT_TERMS,
|
||||
)
|
||||
has_solicitation_signal = cls._contains_any_news_term(
|
||||
combined_text,
|
||||
cls._ADULT_SERVICE_SOLICITATION_TERMS,
|
||||
)
|
||||
has_ambiguous_adult_phrase = cls._contains_any_news_term(
|
||||
combined_text,
|
||||
cls._ADULT_SERVICE_SPAM_AMBIGUOUS_TERMS,
|
||||
)
|
||||
if has_ambiguous_adult_phrase:
|
||||
return has_service_anchor and has_solicitation_signal
|
||||
|
||||
return (
|
||||
has_adult_specific_anchor
|
||||
and has_service_anchor
|
||||
and has_solicitation_signal
|
||||
and context_hits >= 3
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _score_news_relevance(
|
||||
cls,
|
||||
@@ -2727,7 +3040,7 @@ class SearchService:
|
||||
direct_signal += 12
|
||||
add_reason("命中公告/财报/交易等公司事件词")
|
||||
|
||||
if cls._contains_any_news_term(f"{source} {url}", cls._OFFICIAL_SOURCE_TERMS):
|
||||
if cls._is_trusted_official_news_source(item):
|
||||
score += 8
|
||||
add_reason("来源接近公告或交易所渠道")
|
||||
|
||||
@@ -2824,6 +3137,72 @@ class SearchService:
|
||||
search_time=response.search_time,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _filter_ranked_news_for_context(
|
||||
cls,
|
||||
response: SearchResponse,
|
||||
*,
|
||||
log_scope: str,
|
||||
) -> SearchResponse:
|
||||
"""Drop obvious non-news pages and zero-relevance fillers from ranked results."""
|
||||
if not response.success or not response.results:
|
||||
return response
|
||||
|
||||
candidates: List[SearchResult] = []
|
||||
dropped_low_quality = 0
|
||||
dropped_adult_spam = 0
|
||||
dropped_zero_relevance = 0
|
||||
|
||||
for item in response.results:
|
||||
is_official_source = cls._is_trusted_official_news_source(item)
|
||||
if (
|
||||
not is_official_source
|
||||
and cls._has_low_quality_news_page_signal(item)
|
||||
):
|
||||
dropped_low_quality += 1
|
||||
continue
|
||||
if (
|
||||
not is_official_source
|
||||
and cls._has_adult_service_spam_news_page_signal(item)
|
||||
):
|
||||
dropped_adult_spam += 1
|
||||
continue
|
||||
candidates.append(item)
|
||||
|
||||
meaningful_candidates = [
|
||||
item
|
||||
for item in candidates
|
||||
if item.relevance_category == cls._DIRECT_NEWS_CATEGORY
|
||||
or (item.relevance_score or 0) > 0
|
||||
]
|
||||
if meaningful_candidates:
|
||||
dropped_zero_relevance = len(candidates) - len(meaningful_candidates)
|
||||
filtered_results = meaningful_candidates
|
||||
else:
|
||||
filtered_results = candidates
|
||||
|
||||
if dropped_low_quality or dropped_adult_spam or dropped_zero_relevance:
|
||||
logger.info(
|
||||
"[新闻准入] %s: provider=%s, total=%s, kept=%s, "
|
||||
"drop_low_quality=%s, drop_adult_spam=%s, drop_zero_relevance=%s",
|
||||
log_scope,
|
||||
response.provider,
|
||||
len(response.results),
|
||||
len(filtered_results),
|
||||
dropped_low_quality,
|
||||
dropped_adult_spam,
|
||||
dropped_zero_relevance,
|
||||
)
|
||||
|
||||
return SearchResponse(
|
||||
query=response.query,
|
||||
results=filtered_results,
|
||||
provider=response.provider,
|
||||
success=response.success,
|
||||
error_message=response.error_message,
|
||||
search_time=response.search_time,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _news_relevance_stats(
|
||||
cls,
|
||||
@@ -3343,16 +3722,6 @@ class SearchService:
|
||||
log_scope=f"{stock_code}:{provider.name}:stock_news",
|
||||
)
|
||||
had_provider_success = had_provider_success or bool(response.success)
|
||||
filtered_count = len(filtered_response.results or []) if filtered_response.success else 0
|
||||
self._record_news_search_run(
|
||||
provider=provider.name,
|
||||
operation="search_stock_news",
|
||||
success=bool(filtered_response.success and filtered_response.results),
|
||||
latency_ms=self._elapsed_ms(started_at),
|
||||
record_count=filtered_count,
|
||||
error_type=None if filtered_count else "NoUsableNews",
|
||||
error_message=None if filtered_count else (response.error_message or "过滤后无有效新闻"),
|
||||
)
|
||||
|
||||
if filtered_response.success and filtered_response.results:
|
||||
language_response, _preferred_count = self._prioritize_news_language(
|
||||
@@ -3367,10 +3736,33 @@ class SearchService:
|
||||
max_results=provider_max_results,
|
||||
log_scope=f"{stock_code}:{provider.name}:stock_news",
|
||||
)
|
||||
limited_response = self._limit_search_response(
|
||||
admitted_response = self._filter_ranked_news_for_context(
|
||||
ranked_response,
|
||||
log_scope=f"{stock_code}:{provider.name}:stock_news",
|
||||
)
|
||||
limited_response = self._limit_search_response(
|
||||
admitted_response,
|
||||
max_results=max_results,
|
||||
)
|
||||
admitted_count = len(limited_response.results or [])
|
||||
self._record_news_search_run(
|
||||
provider=provider.name,
|
||||
operation="search_stock_news",
|
||||
success=bool(limited_response.success and limited_response.results),
|
||||
latency_ms=self._elapsed_ms(started_at),
|
||||
record_count=admitted_count,
|
||||
error_type=None if admitted_count else "NoUsableNews",
|
||||
error_message=None if admitted_count else (
|
||||
response.error_message or "过滤后无有效新闻"
|
||||
),
|
||||
)
|
||||
if not admitted_count:
|
||||
logger.info(
|
||||
"%s 搜索成功但准入过滤后无有效新闻,继续尝试下一引擎",
|
||||
provider.name,
|
||||
)
|
||||
continue
|
||||
|
||||
stats = self._news_relevance_stats(
|
||||
limited_response,
|
||||
prefer_chinese=prefer_chinese,
|
||||
@@ -3424,6 +3816,18 @@ class SearchService:
|
||||
provider.name,
|
||||
)
|
||||
else:
|
||||
filtered_count = len(filtered_response.results or []) if filtered_response.success else 0
|
||||
self._record_news_search_run(
|
||||
provider=provider.name,
|
||||
operation="search_stock_news",
|
||||
success=bool(filtered_response.success and filtered_response.results),
|
||||
latency_ms=self._elapsed_ms(started_at),
|
||||
record_count=filtered_count,
|
||||
error_type=None if filtered_count else "NoUsableNews",
|
||||
error_message=None if filtered_count else (
|
||||
response.error_message or "过滤后无有效新闻"
|
||||
),
|
||||
)
|
||||
if response.success and not filtered_response.results:
|
||||
logger.info(
|
||||
"%s 搜索成功但过滤后无有效新闻,继续尝试下一引擎",
|
||||
@@ -3727,9 +4131,17 @@ class SearchService:
|
||||
stock_code=stock_code,
|
||||
stock_name=stock_name,
|
||||
prefer_chinese=self._should_prefer_chinese_news(stock_code, stock_name),
|
||||
max_results=target_per_dimension,
|
||||
max_results=provider_max_results,
|
||||
log_scope=f"{stock_code}:{provider.name}:{dim['name']}:rank",
|
||||
)
|
||||
filtered_response = self._filter_ranked_news_for_context(
|
||||
filtered_response,
|
||||
log_scope=f"{stock_code}:{provider.name}:{dim['name']}:admission",
|
||||
)
|
||||
filtered_response = self._limit_search_response(
|
||||
filtered_response,
|
||||
max_results=target_per_dimension,
|
||||
)
|
||||
results[dim['name']] = filtered_response
|
||||
search_count += 1
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user