yfinance 1.2.x returns Ticker.dividends as a single-column DataFrame instead of a
Series. YfinanceFundamentalAdapter iterated it with `.items()`, which for a DataFrame
yields (column_name, Series) — so `_safe_float(Series)` returned None, every dividend
event was dropped, and the TTM cash/count silently fell back to
`info.trailingAnnualDividendRate` (e.g. 24.0 with "0 次") instead of the true TTM sum.
Coerce to a Series before iterating.
Affects every yfinance-backed market (US/HK/JP/KR/TW); surfaced on a live TW report
(2330.TW showed 24.0 / 0 payouts vs the real ~22 across 4). + a regression test that
feeds a single-column DataFrame and asserts the events + TTM sum are parsed.
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
_CLOSING_AUCTION_WINDOW_MINUTES had no tw entry, so `.get(market, 0)` gave a
zero-width window and infer_market_phase("tw") could never return CLOSING_AUCTION:
the last tick before 13:30 stayed INTRADAY and 13:30 flipped straight to POSTMARKET.
TWSE/TPEx run a 5-minute closing call auction (13:25-13:30), so add "tw": 5.
+ a phase-boundary regression test (no lunch break, 13:30 half-hour close):
13:24 INTRADAY / 13:25-13:29 CLOSING_AUCTION / 13:30 POSTMARKET.
* feat(market): v2.1 — consume tw 三大法人 in report + LLM prompt, fix TWD currency label
The merged 三大法人 (institutional flows) data (#1829/#1841/#1855/#1863) reached
get_fundamental_context but was never surfaced: the report renderer and the LLM
prompt-builder never read the institution block, so a real analysis run showed a tw
report with no 三大法人 figures. This wires the last mile (tw-only, additive,
fail-open) and fixes a TWD currency mislabel found in the same run:
- report render (src/notification.py): _append_institutional_flow renders a 三大法人
net-buy/sell table (外资/投信/自营/合计 in 万股/亿股) when the institution block is
status='ok'; _get_fundamental_blocks now extracts it. + zh/en/ko labels.
- LLM prompt (src/analyzer.py): _format_prompt injects a 三大法人 section (raw net
figures + a qualitative 台股筹码过滤器 note, mirroring the capital_flow section)
when status='ok' and all four core nets are present.
- fetch availability (data_provider/base.py): the tw institution fetch (a ~4-5s
whole-market download) now uses the remaining stage budget instead of the ~3s
per-symbol fetch cap that starved it and made the first/only stock of a run
coin-flip to not_supported. Still bounded by the stage deadline (fail-open).
- currency (src/notification.py): _CURRENCY_SUFFIX gains TWD -> 新台币 so TWD amounts
(revenue/profit/dividend) no longer render as the A-share default 元 (reads as RMB).
Verified end-to-end via a real analysis run: the tw report now shows the 三大法人
table and 新台币-labelled financials, and institution is reliably 'ok' on a
single-stock run. Strictly additive — cn/hk/us/jp/kr byte-identical (gated on
market=='tw' / status=='ok'); no capital_flow_signal / scoring / schema. + 11 tests.
Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter caught a
prompt-gate asymmetry (only total_net checked vs all four in the render gate);
tightened to require all four core nets, matching the render/base.py gate.
* fix(market): honour FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 for the tw institution fetch
Review on #1866: the v2.1 institution-timeout change (use the remaining stage budget
instead of the ~3s per-fetch cap) inadvertently bypassed the existing
FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 semantic — with fetch_timeout=0, valuation and
bundle are disabled (they gate on fetch_timeout) but the institution fetch still ran.
Gate the institution fetch on `fetch_timeout > 0` as well, so fetch_timeout=0 disables
it like the other fundamental fetches; when enabled it still uses the remaining stage
budget (the whole-market download needs more than the per-symbol cap). + a regression
test asserting fetch_timeout=0 -> institution not_supported and the fetcher is not called.
---------
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
Two tw-only, strictly-additive, fail-open hardening changes to TwInstitutionalFetcher
(benchmarked against docs/data-source-stability.md):
- C1 circuit breaker: reuse data_provider.realtime_types.CircuitBreaker (the same class
DataFetcherManager uses), keyed per market (twse/tpex), 3-fail / ~5-min cooldown /
half-open probe. When an endpoint is unreachable the fetch is skipped fast and fails
open, instead of paying timeout + throttle on every stock during an outage. The breaker
tracks REACHABILITY: a hard network/HTTP error trips it; an empty / stat!=OK body still
means the endpoint responded, so it counts as success (resets the failure streak and
closes the breaker during recovery -- a no-data day mid-recovery can never strand it open).
- C2 TPEx date guard: TPEx OpenAPI serves only the latest trading day; an explicit date
that does not match the served record now fails open (None) instead of a wrong-day record.
Tests: +6 (breaker opens after 3 failures & skips the 4th fetch; recovers after cooldown;
3 empties don't trip it; TPEx date match / mismatch / no-date). + CHANGELOG.
Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter flagged a
HALF_OPEN-recovery edge in the shared CircuitBreaker; resolved tw-side by treating an
empty response as success (reachability), without modifying the shared class.
* feat(market): surface tw 三大法人 (institutional flows) in the tw report institution block
Wire the merged TwInstitutionalFetcher (#1829/#1841/#1855) into the tw analysis
report: in _build_offshore_fundamental_context, for market == "tw" only, the
institution block now carries the raw 三大法人 net buy/sell figures (foreign /
trust / dealer / total, unit=shares) from TWSE T86 / TPEx instead of the hardcoded
not_supported. v2 scope confirmed by the maintainer on issue #1777.
- tw-only + strictly additive: cn/hk/us/jp/kr offshore flows are byte-identical
(the not_supported loop just excludes institution; non-tw still gets the same
not_supported block). Pinned by tests for us/hk/jp/kr.
- fail-open + default-on: any error or no-data -> not_supported, never interrupts
the analysis. Wiring (import/construct) failures log at error (visible) but still
fail open; fetch failures log at warning.
- status "ok" only when all four core nets are present (a genuine 0 is kept).
- raw figures only: no capital_flow_signal / scoring / weight / schema / Web.
Tests: tests/test_tw_institution_report_wiring.py (9) covers data->ok, genuine-zero,
None/raise/init-raise fail-open, missing-core-net->not_supported, us/hk/jp/kr
byte-identical + fetcher-unused, and no-derived-key. + CHANGELOG.
Dual review (code-reviewer + silent-failure-hunter): the hunter caught the combined
try swallowing wiring (import/construct) errors silently at info -> split into
loud-but-fail-open (error log + still not_supported, honoring #1777's
never-interrupt rule) and guarded "ok" against a null core net.
* fix(market): address Codex review on tw institution wiring (#1863) — stage timeout, overall status, docs
- P1: run the tw institution fetch under the fundamental stage/fetch budget via
_run_with_retry (like the other offshore blocks), so a slow / rate-limited TWSE/TPEx
call fails open at the deadline instead of pushing the analysis past it.
- P2: count a present tw institution toward the OVERALL fundamental status so a report
with only 三大法人 data still surfaces fundamentals (consumers key off the top-level
status). missing_fields keeps the original three blocks, so offshore markets without
institution data stay byte-identical.
- P2 (docs): sync docs/market-support.md (drop the now-false "data-layer only / not
wired into report" clause) + add a tw institution note to docs/full-guide.md and
docs/full-guide_EN.md (AGENTS.md requires user-visible report changes to update guides).
Tests: + stage-timeout (a 2s fetch is abandoned at the ~0.3s budget, fail-open) +
overall-status surfacing assertion. 10 tw-wiring tests + offshore/tw/jp-kr suites green.
* test(tw): add live-network drift smoke + network-marked tests for 三大法人 fetcher
The offline tests (tests/test_tw_institutional_fetcher.py) pin the parser to
frozen fixtures, so they cannot notice an upstream TWSE T86 / TPEx feed change.
Add two additive, tw-only drift detectors that hit the real public endpoints:
- tests/tw_institutional_live_smoke.py: manual non-pytest smoke (mirrors
tests/longbridge_live_smoke.py). Checks endpoint liveness + core-column-name
presence, and cross-checks the fetcher's foreign/trust/dealer/total against
the raw columns plus the always-true reconstruction
total == foreign + foreign_dealer + trust + dealer (the 3-term identity only
holds when the foreign-dealer sub-component is 0).
- tests/test_tw_institutional_network.py: @pytest.mark.network, run only by the
non-blocking Network Smoke cron (pytest -m network); the blocking backend gate
runs pytest -m "not network" (scripts/ci_gate.sh) so these never gate a PR.
Two self-contained tests cross-check the fetcher against the raw feed.
No production code touched (data_provider/base.py unchanged); fail-open and
no-silent-pass preserved. A renamed core/foreign-dealer column or a non-JSON
response (maintenance page / URL migration) is reported LOUD as drift; only a
transport error or non-trading-day soft-skips. Verified live against today's
feeds and via a negative-path simulation of every drift/transient case.
Dual review (code-reviewer APPROVE + silent-failure-hunter PROCEED) caught and
fixed soft-skip paths that had masked feed drift (non-JSON body swallowed as a
blip; foreign-dealer rename fabricated via `or 0`).
* test(tw): fail loud when a stock is present in the raw feed but the fetcher returns None
Addresses the review correctness-blocker on #1855. The TWSE/TPEx drift tests
called the fetcher first and soft-skipped on a None result as "transient /
suspended" WITHOUT checking the raw feed — so an upstream parse-prerequisite
drift (e.g. a 民國->ISO date-format change that _parse_tpex_row / _parse_t86_row
cannot convert) would make get_institutional_net() fail-open to None and be
silently skipped, the exact fail-open these tests exist to catch (and contrary
to the PR's stated "民國->ISO date switch fails loud").
Reorder both the smoke and the network tests: resolve the target stock's raw
row FIRST, then if the fetcher returns None — fail LOUD when the row IS present
in the raw feed (parse/date drift), and only soft-skip when the row is genuinely
absent (non-trading day / suspended / transient).
Verified: a negative-path simulation (raw row present + an unconvertible payload
date) now reports drift and fails, the live happy path still passes, and the
network tests are stable across repeated runs (transient blips hit both the raw
fetch and the fetcher together -> consistent skip, never a false fail).
---------
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
* feat(i18n): add Korean to report language label maps
Extend the report language layer with a third code `ko` so reports can
render deterministic labels in Korean. Adds `ko` to the supported list,
aliases (korean/kr/ko-kr), every translation map, the full report label
set, sentiment bands, the config registry enum and `.env.example`.
`zh`/`en` behavior is unchanged and unknown languages still fall back to
the default. Prompt and Web surfaces follow in later changes.
Refs #1614
* feat(i18n): emit Korean output directives in analysis prompts
Make the analysis and market-review prompts produce Korean output when
REPORT_LANGUAGE=ko. The decision agent now appends a Korean
output-language directive (JSON keys and decision_type enum unchanged),
and the market-review prompt reuses the English structural scaffolding
while instructing the model to write the shell, headings and conclusion
in Korean. Market-phase and context-pack prompt sections route ko to the
English structural base. The market-review payload keeps the truthful
language code via a dedicated output-language helper, leaving zh/en
behavior unchanged.
Refs #1614
* fix(i18n): localize phase and market-context guardrails for Korean
Route Korean reports through the English structural scaffolding for
market-phase and market-context prompt sections, and add Korean output,
detection markers, negations and recap patterns to the phase-decision and
daily-market-context guardrails so they operate on Korean model output.
Confidence and operation labels now flow through the shared localize
helpers. zh/en behavior is unchanged.
Refs #1614
* feat(i18n): localize Korean fallback output across analysis pipeline
Add Korean output to the deterministic strings emitted outside the LLM:
per-stock and executor output-language directives, no-API-key / backend
/ parse error fallbacks, hold-watch advice and reasons, market-review
titles and summaries, and history and notification report labels. Fund
flow, trend and confidence values now flow through the shared localize
helpers, and language-keyed advice tables no longer raise KeyError for a
third language. Structural prompt sections route ko to the English
scaffolding. zh/en output is unchanged.
Refs #1614
* feat(web): add Korean report language rendering
Extend the Web ReportLanguage type with ko and add Korean copy to the
report detail surfaces: report text, sentiment labels, market-phase
labels, analysis-context summary, market-review view, diagnostics and
news source. Run-flow chrome that is keyed by UI language falls back to
English for ko. The report-language selector is driven by the backend
config schema, so Korean appears automatically. zh/en rendering is
unchanged.
Refs #1614
* docs(i18n): document Korean report language support
Note that REPORT_LANGUAGE accepts ko in the bilingual guides and the
analyze / market-review request-language parameters, and add a
CHANGELOG entry for Korean report output.
Refs #1614
* fix(i18n): canonicalize Korean values and accept ko in API schemas
Add Korean aliases to the operation-advice, trend, confidence, chip and
bias canonical maps so Korean model output (매수/매도/보유/관망 etc.)
resolves to the correct decision_type and signal level instead of
falling back to hold or a score-band signal. Accept ko in the
analyze, market-review and decision-signal request schemas (and the
static API spec) so the typed client and backend agree and per-request
Korean analysis is not rejected with 422.
Refs #1614
Hardens the merged #1777 TwInstitutionalFetcher. Under concurrent callers for the
same (market, date) the previous check-then-fetch could issue N duplicate upstream
requests -- which for the TWSE T86 RWD endpoint risks tripping its informal
~3 req/5s ban. A per-(market,date) in-flight lock now coalesces same-key callers
into a single fetch (double-checked against the cache); different keys still fetch
in parallel, and the master lock is never held across network I/O.
- data_provider/tw_institutional_fetcher.py: per-key in-flight lock + _read_cache /
_key_lock helpers. Behavior is otherwise unchanged -- still fail-open, still
caches only non-empty results, tw-only, no data_provider/base.py change.
- tests: + concurrent-same-key-coalesces-to-single-fetch (8 threads -> 1 request),
different-keys-not-coalesced, and HTTP-error (429) fail-open
- docs/CHANGELOG.md: [改进] entry
Refs #1777
Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com>
* fix: stabilize related board rendering
* fix(review-feedback-1836): Recognize existing Chinese sector tables before appending
* fix(review-feedback-1836): Use unique segment keys for combined reviews
* fix(review-feedback-1836): return or otherwise propagate the rendered markdown for the merge path
* fix(review-feedback-1836): Insert sector fallback inside each market segment
* fix: preserve market review segment boundaries
Implements the #1777 maintainer-greenlit Phase-2 data layer: a self-contained,
tw-only fetcher for Taiwan per-stock institutional (外資/投信/自營商) net
buy/sell. Strictly additive -- no change to the existing cn/hk/us/jp/kr flows in
data_provider/base.py, and not yet wired into the report/Web/scoring path (a
deliberate follow-up per the maintainer's scope).
- data_provider/tw_institutional_fetcher.py (NEW): TwInstitutionalFetcher
- 上市 .TW -> TWSE T86 legacy rwd JSON endpoint (西元 date, comma values)
- 上櫃 .TWO -> TPEx OpenAPI tpex_3insti_daily_trading (民國 date, plain ints)
- T86 columns are read by NAME (validated against the payload `fields` header),
so a TWSE column rename / reorder fails open instead of silently shipping
misaligned numbers under stale indices
- foreign_net = 外陸資 (ex 外資自營商, T86) / dealer-excluded foreign (TPEx) so
the breakdown matches the official 三大法人 total; total_net is the official
figure; unit = shares, signs preserved
- whole-market single-day cache keyed by (market, date), filtered per stock;
~3 req/5s throttle (own lock) for the T86 endpoint
- fail-open: any network/rate-limit/empty/unknown-stock returns None; a missing
or renamed column drops the row (never a fabricated 0); a row whose trading
date cannot be attributed (TPEx 民國 unconvertible) is dropped; empty/failed
fetches are not cached (no TTL-long blackout)
- tests/test_tw_institutional_fetcher.py (NEW): 20 offline tests with fixtures
trimmed from real T86 (2330) / TPEx (3105) responses; pins the net breakdown +
sign, 民國->西元 conversion, routing, caching, and fail-open -- including
column reorder (parsed by name), column rename / missing header (fail-open),
the missing-column-vs-genuine-zero distinction, and unconvertible TPEx dates
- docs/market-support.md + docs/CHANGELOG.md: data-source capability boundary +
OGDL v1 license note; no new config (.env.example untouched)
Addresses the #1829 review (read T86 by field name; drop TPEx rows with an
unconvertible date; foreign_net excludes foreign-dealers).
Sources are 政府開放資料 under 政府資料開放授權條款第 1 版 (OGDL v1, commercial-safe).
Refs #1777
