_CLOSING_AUCTION_WINDOW_MINUTES had no tw entry, so `.get(market, 0)` gave a
zero-width window and infer_market_phase("tw") could never return CLOSING_AUCTION:
the last tick before 13:30 stayed INTRADAY and 13:30 flipped straight to POSTMARKET.
TWSE/TPEx run a 5-minute closing call auction (13:25-13:30), so add "tw": 5.
+ a phase-boundary regression test (no lunch break, 13:30 half-hour close):
13:24 INTRADAY / 13:25-13:29 CLOSING_AUCTION / 13:30 POSTMARKET.
* feat(market): v2.1 — consume tw 三大法人 in report + LLM prompt, fix TWD currency label
The merged 三大法人 (institutional flows) data (#1829/#1841/#1855/#1863) reached
get_fundamental_context but was never surfaced: the report renderer and the LLM
prompt-builder never read the institution block, so a real analysis run showed a tw
report with no 三大法人 figures. This wires the last mile (tw-only, additive,
fail-open) and fixes a TWD currency mislabel found in the same run:
- report render (src/notification.py): _append_institutional_flow renders a 三大法人
net-buy/sell table (外资/投信/自营/合计 in 万股/亿股) when the institution block is
status='ok'; _get_fundamental_blocks now extracts it. + zh/en/ko labels.
- LLM prompt (src/analyzer.py): _format_prompt injects a 三大法人 section (raw net
figures + a qualitative 台股筹码过滤器 note, mirroring the capital_flow section)
when status='ok' and all four core nets are present.
- fetch availability (data_provider/base.py): the tw institution fetch (a ~4-5s
whole-market download) now uses the remaining stage budget instead of the ~3s
per-symbol fetch cap that starved it and made the first/only stock of a run
coin-flip to not_supported. Still bounded by the stage deadline (fail-open).
- currency (src/notification.py): _CURRENCY_SUFFIX gains TWD -> 新台币 so TWD amounts
(revenue/profit/dividend) no longer render as the A-share default 元 (reads as RMB).
Verified end-to-end via a real analysis run: the tw report now shows the 三大法人
table and 新台币-labelled financials, and institution is reliably 'ok' on a
single-stock run. Strictly additive — cn/hk/us/jp/kr byte-identical (gated on
market=='tw' / status=='ok'); no capital_flow_signal / scoring / schema. + 11 tests.
Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter caught a
prompt-gate asymmetry (only total_net checked vs all four in the render gate);
tightened to require all four core nets, matching the render/base.py gate.
* fix(market): honour FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 for the tw institution fetch
Review on #1866: the v2.1 institution-timeout change (use the remaining stage budget
instead of the ~3s per-fetch cap) inadvertently bypassed the existing
FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 semantic — with fetch_timeout=0, valuation and
bundle are disabled (they gate on fetch_timeout) but the institution fetch still ran.
Gate the institution fetch on `fetch_timeout > 0` as well, so fetch_timeout=0 disables
it like the other fundamental fetches; when enabled it still uses the remaining stage
budget (the whole-market download needs more than the per-symbol cap). + a regression
test asserting fetch_timeout=0 -> institution not_supported and the fetcher is not called.
---------
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
Two tw-only, strictly-additive, fail-open hardening changes to TwInstitutionalFetcher
(benchmarked against docs/data-source-stability.md):
- C1 circuit breaker: reuse data_provider.realtime_types.CircuitBreaker (the same class
DataFetcherManager uses), keyed per market (twse/tpex), 3-fail / ~5-min cooldown /
half-open probe. When an endpoint is unreachable the fetch is skipped fast and fails
open, instead of paying timeout + throttle on every stock during an outage. The breaker
tracks REACHABILITY: a hard network/HTTP error trips it; an empty / stat!=OK body still
means the endpoint responded, so it counts as success (resets the failure streak and
closes the breaker during recovery -- a no-data day mid-recovery can never strand it open).
- C2 TPEx date guard: TPEx OpenAPI serves only the latest trading day; an explicit date
that does not match the served record now fails open (None) instead of a wrong-day record.
Tests: +6 (breaker opens after 3 failures & skips the 4th fetch; recovers after cooldown;
3 empties don't trip it; TPEx date match / mismatch / no-date). + CHANGELOG.
Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter flagged a
HALF_OPEN-recovery edge in the shared CircuitBreaker; resolved tw-side by treating an
empty response as success (reachability), without modifying the shared class.
* feat(market): surface tw 三大法人 (institutional flows) in the tw report institution block
Wire the merged TwInstitutionalFetcher (#1829/#1841/#1855) into the tw analysis
report: in _build_offshore_fundamental_context, for market == "tw" only, the
institution block now carries the raw 三大法人 net buy/sell figures (foreign /
trust / dealer / total, unit=shares) from TWSE T86 / TPEx instead of the hardcoded
not_supported. v2 scope confirmed by the maintainer on issue #1777.
- tw-only + strictly additive: cn/hk/us/jp/kr offshore flows are byte-identical
(the not_supported loop just excludes institution; non-tw still gets the same
not_supported block). Pinned by tests for us/hk/jp/kr.
- fail-open + default-on: any error or no-data -> not_supported, never interrupts
the analysis. Wiring (import/construct) failures log at error (visible) but still
fail open; fetch failures log at warning.
- status "ok" only when all four core nets are present (a genuine 0 is kept).
- raw figures only: no capital_flow_signal / scoring / weight / schema / Web.
Tests: tests/test_tw_institution_report_wiring.py (9) covers data->ok, genuine-zero,
None/raise/init-raise fail-open, missing-core-net->not_supported, us/hk/jp/kr
byte-identical + fetcher-unused, and no-derived-key. + CHANGELOG.
Dual review (code-reviewer + silent-failure-hunter): the hunter caught the combined
try swallowing wiring (import/construct) errors silently at info -> split into
loud-but-fail-open (error log + still not_supported, honoring #1777's
never-interrupt rule) and guarded "ok" against a null core net.
* fix(market): address Codex review on tw institution wiring (#1863) — stage timeout, overall status, docs
- P1: run the tw institution fetch under the fundamental stage/fetch budget via
_run_with_retry (like the other offshore blocks), so a slow / rate-limited TWSE/TPEx
call fails open at the deadline instead of pushing the analysis past it.
- P2: count a present tw institution toward the OVERALL fundamental status so a report
with only 三大法人 data still surfaces fundamentals (consumers key off the top-level
status). missing_fields keeps the original three blocks, so offshore markets without
institution data stay byte-identical.
- P2 (docs): sync docs/market-support.md (drop the now-false "data-layer only / not
wired into report" clause) + add a tw institution note to docs/full-guide.md and
docs/full-guide_EN.md (AGENTS.md requires user-visible report changes to update guides).
Tests: + stage-timeout (a 2s fetch is abandoned at the ~0.3s budget, fail-open) +
overall-status surfacing assertion. 10 tw-wiring tests + offshore/tw/jp-kr suites green.
* test(tw): add live-network drift smoke + network-marked tests for 三大法人 fetcher
The offline tests (tests/test_tw_institutional_fetcher.py) pin the parser to
frozen fixtures, so they cannot notice an upstream TWSE T86 / TPEx feed change.
Add two additive, tw-only drift detectors that hit the real public endpoints:
- tests/tw_institutional_live_smoke.py: manual non-pytest smoke (mirrors
tests/longbridge_live_smoke.py). Checks endpoint liveness + core-column-name
presence, and cross-checks the fetcher's foreign/trust/dealer/total against
the raw columns plus the always-true reconstruction
total == foreign + foreign_dealer + trust + dealer (the 3-term identity only
holds when the foreign-dealer sub-component is 0).
- tests/test_tw_institutional_network.py: @pytest.mark.network, run only by the
non-blocking Network Smoke cron (pytest -m network); the blocking backend gate
runs pytest -m "not network" (scripts/ci_gate.sh) so these never gate a PR.
Two self-contained tests cross-check the fetcher against the raw feed.
No production code touched (data_provider/base.py unchanged); fail-open and
no-silent-pass preserved. A renamed core/foreign-dealer column or a non-JSON
response (maintenance page / URL migration) is reported LOUD as drift; only a
transport error or non-trading-day soft-skips. Verified live against today's
feeds and via a negative-path simulation of every drift/transient case.
Dual review (code-reviewer APPROVE + silent-failure-hunter PROCEED) caught and
fixed soft-skip paths that had masked feed drift (non-JSON body swallowed as a
blip; foreign-dealer rename fabricated via `or 0`).
* test(tw): fail loud when a stock is present in the raw feed but the fetcher returns None
Addresses the review correctness-blocker on #1855. The TWSE/TPEx drift tests
called the fetcher first and soft-skipped on a None result as "transient /
suspended" WITHOUT checking the raw feed — so an upstream parse-prerequisite
drift (e.g. a 民國->ISO date-format change that _parse_tpex_row / _parse_t86_row
cannot convert) would make get_institutional_net() fail-open to None and be
silently skipped, the exact fail-open these tests exist to catch (and contrary
to the PR's stated "民國->ISO date switch fails loud").
Reorder both the smoke and the network tests: resolve the target stock's raw
row FIRST, then if the fetcher returns None — fail LOUD when the row IS present
in the raw feed (parse/date drift), and only soft-skip when the row is genuinely
absent (non-trading day / suspended / transient).
Verified: a negative-path simulation (raw row present + an unconvertible payload
date) now reports drift and fails, the live happy path still passes, and the
network tests are stable across repeated runs (transient blips hit both the raw
fetch and the fetcher together -> consistent skip, never a false fail).
---------
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
* feat(i18n): add Korean to report language label maps
Extend the report language layer with a third code `ko` so reports can
render deterministic labels in Korean. Adds `ko` to the supported list,
aliases (korean/kr/ko-kr), every translation map, the full report label
set, sentiment bands, the config registry enum and `.env.example`.
`zh`/`en` behavior is unchanged and unknown languages still fall back to
the default. Prompt and Web surfaces follow in later changes.
Refs #1614
* feat(i18n): emit Korean output directives in analysis prompts
Make the analysis and market-review prompts produce Korean output when
REPORT_LANGUAGE=ko. The decision agent now appends a Korean
output-language directive (JSON keys and decision_type enum unchanged),
and the market-review prompt reuses the English structural scaffolding
while instructing the model to write the shell, headings and conclusion
in Korean. Market-phase and context-pack prompt sections route ko to the
English structural base. The market-review payload keeps the truthful
language code via a dedicated output-language helper, leaving zh/en
behavior unchanged.
Refs #1614
* fix(i18n): localize phase and market-context guardrails for Korean
Route Korean reports through the English structural scaffolding for
market-phase and market-context prompt sections, and add Korean output,
detection markers, negations and recap patterns to the phase-decision and
daily-market-context guardrails so they operate on Korean model output.
Confidence and operation labels now flow through the shared localize
helpers. zh/en behavior is unchanged.
Refs #1614
* feat(i18n): localize Korean fallback output across analysis pipeline
Add Korean output to the deterministic strings emitted outside the LLM:
per-stock and executor output-language directives, no-API-key / backend
/ parse error fallbacks, hold-watch advice and reasons, market-review
titles and summaries, and history and notification report labels. Fund
flow, trend and confidence values now flow through the shared localize
helpers, and language-keyed advice tables no longer raise KeyError for a
third language. Structural prompt sections route ko to the English
scaffolding. zh/en output is unchanged.
Refs #1614
* feat(web): add Korean report language rendering
Extend the Web ReportLanguage type with ko and add Korean copy to the
report detail surfaces: report text, sentiment labels, market-phase
labels, analysis-context summary, market-review view, diagnostics and
news source. Run-flow chrome that is keyed by UI language falls back to
English for ko. The report-language selector is driven by the backend
config schema, so Korean appears automatically. zh/en rendering is
unchanged.
Refs #1614
* docs(i18n): document Korean report language support
Note that REPORT_LANGUAGE accepts ko in the bilingual guides and the
analyze / market-review request-language parameters, and add a
CHANGELOG entry for Korean report output.
Refs #1614
* fix(i18n): canonicalize Korean values and accept ko in API schemas
Add Korean aliases to the operation-advice, trend, confidence, chip and
bias canonical maps so Korean model output (매수/매도/보유/관망 etc.)
resolves to the correct decision_type and signal level instead of
falling back to hold or a score-band signal. Accept ko in the
analyze, market-review and decision-signal request schemas (and the
static API spec) so the typed client and backend agree and per-request
Korean analysis is not rejected with 422.
Refs #1614
Hardens the merged #1777 TwInstitutionalFetcher. Under concurrent callers for the
same (market, date) the previous check-then-fetch could issue N duplicate upstream
requests -- which for the TWSE T86 RWD endpoint risks tripping its informal
~3 req/5s ban. A per-(market,date) in-flight lock now coalesces same-key callers
into a single fetch (double-checked against the cache); different keys still fetch
in parallel, and the master lock is never held across network I/O.
- data_provider/tw_institutional_fetcher.py: per-key in-flight lock + _read_cache /
_key_lock helpers. Behavior is otherwise unchanged -- still fail-open, still
caches only non-empty results, tw-only, no data_provider/base.py change.
- tests: + concurrent-same-key-coalesces-to-single-fetch (8 threads -> 1 request),
different-keys-not-coalesced, and HTTP-error (429) fail-open
- docs/CHANGELOG.md: [改进] entry
Refs #1777
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
* fix: stabilize related board rendering
* fix(review-feedback-1836): Recognize existing Chinese sector tables before appending
* fix(review-feedback-1836): Use unique segment keys for combined reviews
* fix(review-feedback-1836): return or otherwise propagate the rendered markdown for the merge path
* fix(review-feedback-1836): Insert sector fallback inside each market segment
* fix: preserve market review segment boundaries
Implements the #1777 maintainer-greenlit Phase-2 data layer: a self-contained,
tw-only fetcher for Taiwan per-stock institutional (外資/投信/自營商) net
buy/sell. Strictly additive -- no change to the existing cn/hk/us/jp/kr flows in
data_provider/base.py, and not yet wired into the report/Web/scoring path (a
deliberate follow-up per the maintainer's scope).
- data_provider/tw_institutional_fetcher.py (NEW): TwInstitutionalFetcher
- 上市 .TW -> TWSE T86 legacy rwd JSON endpoint (西元 date, comma values)
- 上櫃 .TWO -> TPEx OpenAPI tpex_3insti_daily_trading (民國 date, plain ints)
- T86 columns are read by NAME (validated against the payload `fields` header),
so a TWSE column rename / reorder fails open instead of silently shipping
misaligned numbers under stale indices
- foreign_net = 外陸資 (ex 外資自營商, T86) / dealer-excluded foreign (TPEx) so
the breakdown matches the official 三大法人 total; total_net is the official
figure; unit = shares, signs preserved
- whole-market single-day cache keyed by (market, date), filtered per stock;
~3 req/5s throttle (own lock) for the T86 endpoint
- fail-open: any network/rate-limit/empty/unknown-stock returns None; a missing
or renamed column drops the row (never a fabricated 0); a row whose trading
date cannot be attributed (TPEx 民國 unconvertible) is dropped; empty/failed
fetches are not cached (no TTL-long blackout)
- tests/test_tw_institutional_fetcher.py (NEW): 20 offline tests with fixtures
trimmed from real T86 (2330) / TPEx (3105) responses; pins the net breakdown +
sign, 民國->西元 conversion, routing, caching, and fail-open -- including
column reorder (parsed by name), column rename / missing header (fail-open),
the missing-column-vs-genuine-zero distinction, and unconvertible TPEx dates
- docs/market-support.md + docs/CHANGELOG.md: data-source capability boundary +
OGDL v1 license note; no new config (.env.example untouched)
Addresses the #1829 review (read T86 by field name; drop TPEx rows with an
unconvertible date; foreign_net excludes foreign-dealers).
Sources are 政府開放資料 under 政府資料開放授權條款第 1 版 (OGDL v1, commercial-safe).
Refs #1777
* fix: prefetch portfolio snapshot quotes
* fix: keep portfolio quote prefetch concurrent
* fix: stop sharing fetcher manager across portfolio quote workers
Previously _prefetch_realtime_position_prices reused a single
DataFetcherManager for all workers when prefetch_realtime_quotes
returned a full count. That count only signals a successful bulk
fetch on the underlying fetcher's module-level cache; it does not
guarantee every per-symbol get_realtime_quote() is a cache hit
(mixed-market portfolios, cache misses, or providers that report
counts loosely). With a shared manager, manager-owned per-fetcher
locks (_call_fetcher_method) re-serialize the cache-miss reads
and re-trigger the #1803 30s timeout the prefetch was meant to
avoid.
Discard the prefetch manager after warming the module-level
fetcher cache and let each worker own its DataFetcherManager.
Independent locks restore true concurrency, while cache hits
still work because the realtime cache is fetcher-module-level.
Add a regression test that exercises len>=5, full-count bulk
prefetch, and a per-symbol fetch path that still blocks; it
verifies a fresh manager per worker and max_active>=2.
Follow-up to the #1773 data-layer MVP (Taiwan suffix-only detection + routing,
merged in 2086e3c). That MVP deferred the service/API/frontend layers, leaving a
live defect: tw was absent from the DecisionSignal/Portfolio service VALID_MARKETS,
so _normalize_market("tw") raised ValueError on the decision-signal write path.
The analysis pipeline auto-extracts a DecisionSignal after history save
(_extract_decision_signal_after_history_save), so every tw analysis silently
failed to persist a signal while jp/kr succeeded -- tw was the only
yfinance-supported market that could be analyzed but never produced a signal.
Converge the tw market contract for DecisionSignal + Portfolio + Intelligence in
one pass (mirroring jp/kr #1720), per the human review on #1801 asking not to
land it piecemeal:
Backend service + API:
- src/services/{portfolio,intelligence}_service.py: VALID_MARKETS /
_ALLOWED_MARKETS + _normalize_market error strings accept tw
- src/services/decision_signal_service.py: _normalize_market error string
(VALID_MARKETS is imported from portfolio_service, so the set change propagates)
- src/services/decision_signal_extractor.py: drop the now-stale "(e.g. tw)" guard
comment (tw is supported; the guard still protects genuinely-unsupported markets)
- api/v1/schemas/{decision_signals,intelligence,portfolio}.py: Pydantic Literals + tw
- api/v1/endpoints/decision_signals.py + docs/architecture/api_spec.json: market
filter description + DecisionSignalMarket enum gain tw; test_api_schema_pydantic
exact-match vs create_app().openapi() passes (api_spec kept CRLF)
Frontend (DecisionSignal + Portfolio typed consumers only; tsc + vitest pass):
- apps/dsa-web/src/types/{decisionSignals,portfolio}.ts + pages/{DecisionSignalsPage,
PortfolioPage}.tsx + utils/{decisionSignalLabels,stockCode}.ts + i18n/uiText.ts:
add tw to the DecisionSignalMarket / portfolio market unions, the market filter
options, the tw display label, and .TW/.TWO stock-code normalization
- the alert Market-Light surface (types/alerts.ts MarketRegion, featureText
ALERT_MARKET_REGION_*) is intentionally LEFT OUT: the backend market_light_service
is cn/hk/us only, so exposing tw there would be a front/back mismatch
Tests:
- flip the two #1773 graceful-skip regressions to first-class assertions and add
test_extract_and_persist_writes_tw_signal (end-to-end persist guard)
- frontend: PortfolioPage + stockCode vitest gain tw cases
Docs (reconcile the tw contract so changelog/topic docs/code state one fact):
- docs/CHANGELOG.md: rewrite the #1772 [Unreleased] entries so they no longer say
"service/API deferred" + "tw gracefully skipped" alongside "tw now supported"
- docs/market-support.md, docs/decision-signals.md, docs/intelligence-sources.md:
sync the tw market enum / filter / examples; keep the boundary note
Still deferred (separate follow-ups): the Taiwan stock-index/seed + Web autocomplete,
and the alert (大盘红绿灯) Market-Light tw support (needs a market_light backend change).
Refs #1772
Per the maintainer review on #1772/#1773, scope this PR to market detection
and data routing only — no Taiwan stock-index / seed / pool-resolution, so a
bare code can never be silently rewritten to a .TW suffix (the index path is a
deferred follow-up with the jp/kr-style honest doc + entry-point tests).
- data_provider/base.py: _is_tw_market / _is_tw_suffix_stock, _market_tag,
normalize_stock_code, get_daily_data + get_realtime_quote routing,
_DAILY_MARKET_FETCHER_SUPPORT (YfinanceFetcher only), offshore fundamental path
- data_provider/yfinance_fetcher.py: _get_tw_main_indices (^TWII / ^TWOII) +
.TW/.TWO suffix passthrough
- src/core/trading_calendar.py: XTAI / Asia/Taipei + get_market_for_stock
- src/market_context.py: detect_market + Taiwan prompt guidelines
- src/{market_phase_summary, services/stock_code_utils, core/pipeline}: tw plumbing
(stock_code_utils only recognizes .TW/.TWO codes; no bare->suffix rewrite)
- src/services/decision_signal_extractor.py: gracefully skip tw (recognized by
the data layer but not yet supported by the signal service) — no swallowed
ValueError + traceback
- tests: test_tw_market_support.py + test_decision_signal_extractor.py
(test_build_payload_skips_tw_market_gracefully — real tw skip regression)
- docs/market-support.md + CHANGELOG
Deferred follow-ups: stock index/seed + autocomplete + frontend, API market
enums + api_spec, and the service-layer market enums.
Bare numeric codes keep A-share semantics; only an explicit .TW/.TWO suffix
opts into Taiwan.
Refs #1772
main.py --serve-only passes the app to uvicorn as the import string
"api.app:app", so uvicorn imports the app lazily inside the server
thread. That import (litellm + the full app tree, ~10s+ on constrained
hosts) runs inside the 3.0s startup self-check window, so the probe
times out and --serve-only returns 1, causing a container restart loop
on slower machines (e.g. a 2 vCPU VPS). Faster hosts finish the import
under 3s and pass, which is why the issue was spec-dependent and
long unreported.
Import the ASGI app object in the calling thread before the probe and
hand the object to uvicorn, so the heavy import stays out of the probe
window. Genuine import failures still surface immediately. Update the
start_api_server unit tests to stub api.app the same way they already
stub uvicorn.
Fixes#1759
Co-authored-by: gang.wu@ximalaya.com <gang.wu@ximalaya.com>
Co-authored-by: mumu <42829555+ZhuLinsen@users.noreply.github.com>
efinance writes its search cache (search-cache.json) into its own
package directory (site-packages/efinance/data), which is root-owned
in the image. Since the container runs as the non-root dsa user
(uid 1000), every A-share fetch failed with PermissionError, forcing
EfinanceFetcher (the default priority-0 source) to fail and fall back.
Pre-create and chown the efinance data dir to dsa at build time. The
path is resolved dynamically to avoid hardcoding the Python version.
This is image-internal static content, so the fix belongs at build
time rather than the runtime mount-repair in the entrypoint (#1263),
which only covers user-mounted volumes.
Fixes#1748
Co-authored-by: gang.wu@ximalaya.com <gang.wu@ximalaya.com>
Co-authored-by: mumu <42829555+ZhuLinsen@users.noreply.github.com>
* fix(issue-1739): [bug]-docker重新部署时，.env设置里的通知变量$content_j
* fix(review-feedback-1741): escape/unescape the braced forms too, or reject them before persisting
* fix(review-feedback-1741): preserve braced templates during env import
* fix(review-feedback-1741): 澄清外部模型/API 配置风险提示是否命中真实 diff
* fix(review-feedback-1741): limit raw parsing to CUSTOM WEBHOOK BODY TEMPLATE or re-apply the
* fix: limit decision signals to AI page
* fix: preserve report signal lookup
* docs: sync decision signal guide
* fix(review-feedback-1737): add a report-id filter or deep link on /decision-signals before
