From ff11940a880ccb2ead92045a7736d557a6f326f9 Mon Sep 17 00:00:00 2001 From: Emil Lerch Date: Wed, 19 Aug 2026 14:59:42 -0700 Subject: [PATCH] correct caching docs for adjustment-basis restatement --- docs/dev/caching-implementation.md | 140 +++++++++++++++++++++++++---- docs/explanation/caching.md | 16 ++++ src/commands/diagnose.zig | 5 +- 3 files changed, 142 insertions(+), 19 deletions(-) diff --git a/docs/dev/caching-implementation.md b/docs/dev/caching-implementation.md index 78034c3..e77ae1e 100644 --- a/docs/dev/caching-implementation.md +++ b/docs/dev/caching-implementation.md @@ -20,8 +20,8 @@ directly. It reads and writes a per-symbol, per-type SRF file cache via ``` {cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR AAPL/ - candles_daily.srf OHLCV bars (append-only) - candles_meta.srf last_close, last_date, provider + freshness + candles_daily.srf OHLCV bars (appended; replaced on restatement) + candles_meta.srf last_close, last_date, provider, adj_basis + freshness dividends.srf splits.srf options.srf @@ -43,12 +43,14 @@ format. Files carry `#!`-prefixed directives (`#!expires=`, Candles are stored as **two** files, and the split is load-bearing: -- `candles_daily.srf` holds the actual OHLCV records and grows - append-only: on a cache miss only bars newer than `last_date` are - fetched and appended, never the full history. +- `candles_daily.srf` holds the actual OHLCV records and normally grows + by appending: on a cache miss only bars newer than `last_date` are + fetched and appended, not the full history. The exception is a + **restatement**, which replaces the file wholesale - see + [The adjustment basis](#the-adjustment-basis). - `candles_meta.srf` holds a single small record (`last_close`, - `last_date`, `provider`, `fail_count`) plus the `#!expires=` and - `#!created=` directives. + `last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`, + `adj_basis`) plus the `#!expires=` and `#!created=` directives. Keeping the metadata separate lets every freshness check and last-price read touch a ~100-byte file instead of deserializing a multi-megabyte @@ -61,6 +63,62 @@ self-heal clear both together, and a negative cache entry for a candle-less symbol is keyed off `candles_daily.srf` (see [Negative caching](#negative-caching)). +### The adjustment basis + +Appending bars cannot restate the bars already on disk, and sometimes +they need it. + +Providers compute `adj_close` by scaling raw `close` by the product of +the adjustment factors for every distribution *after* that bar. So a +series' adjustment basis is only as current as the fetch that produced +it. Newly appended bars arrive with `adj_close == close`, because +nothing has gone ex after them yet, while every previously cached bar +keeps the basis it was originally fetched with. When the next +distribution goes ex, the bars behind it should be marked down by its +factor - and an append does not do that. Every total return spanning +that ex-date then reads low by roughly the missed yield. + +`CandleMeta.adj_basis` records how current a series' basis is. It is +**the date of the newest bar present at the last full fetch**: + +- Not a wall-clock timestamp. A fetch that runs before the day's bar is + published gets a basis one bar behind - which is the honest claim, + since a provider's adjusted series is only ever as complete as its + newest bar. +- Not an ex-date. It is compared against ex-dates but is never one. +- Never advanced by an append. `Store.appendCandles` takes the existing + meta and overrides only `last_close` / `last_date`; only + `Store.cacheCandles`, which replaces the whole file, sets a new + basis. That asymmetry is the mechanism, so keep it. + +`getCandles` escalates from append to full refetch when +`freshness.adjustmentBasisStale` says the basis predates a corporate +action that has **already gone ex**: + +```zig +newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date) +stale = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex) +``` + +The already-ex bound lives in `newestCorporateAction`, not in the +verdict, and that placement is load-bearing in both directions. A +declared-but-not-yet-ex distribution is reflected in no provider's +adjusted series, so treating it as something to catch up to would +refetch on every pass forever. But bounding the *verdict* instead - +"is the newest action of all still in the future? then nothing to do" - +lets one forward announcement hide every older unapplied action behind +it. That shipped: a quarterly payer announcing a quarter ahead was +permanently unable to restate. + +The check runs on the stale path only, not on the fresh-cache +early-return. Detection is therefore at most one trading day behind, +which the candle TTL guarantees, and the hot portfolio-pricing path +pays nothing. + +Steady state for a quarterly payer is four full-history fetches a year, +each landing within about a trading day of an ex-date. `zfin diagnose +SYMBOL` reports the basis against the newest already-ex action. + ## Freshness is the `#!expires=` directive, not mtime A cache entry is fresh when the wall clock is earlier than the @@ -146,7 +204,21 @@ The `--refresh-data` policy maps to `FetchOptions`: ### `getCandles` (single symbol) This is the most involved path because of the daily/meta split, the -incremental-update logic, and the TwelveData carve-out. +incremental-update logic, the adjustment-basis escalation, and the +TwelveData carve-out. + +Note also that `force` re-asks the provider but does **not** rebuild +candle history: it skips the TTL and the server tier, then takes the +same incremental top-up. Replacing the series is the restatement path's +job (or `zfin cache clear`). + +Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off +`provider`. `provider` is pure provenance - "where did these bars come +from" - and using it to route made a single non-transient Tiingo failure +permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`, +and Tiingo was never consulted again. The backoff is armed only by a +genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter, +and clears the moment Tiingo serves the symbol again. ```mermaid flowchart TD @@ -154,31 +226,61 @@ flowchart TD NG -->|yes| FF["return FetchFailed, no network"] NG -->|no| RM{"candles_meta exists?"} - RM -->|yes| TW{"provider is twelvedata?"} + RM -->|yes| SK{"skip_network?"} + SK -->|yes| RETS["return cached even if stale
(FetchFailed if unreadable)"] + SK -->|no| TW{"provider is twelvedata?"} TW -->|yes| FULL TW -->|no| FR{"meta fresh and not force_refresh?"} FR -->|yes| RET["return cached candles"] FR -->|no| SS1["syncCandlesFromServer"] - SS1 --> SF1{"fresh now?"} + SS1 --> SF1{"fresh AND adj_basis current?"} SF1 -->|yes| RET - SF1 -->|no| INC{"shouldRefresh?"} + SF1 -->|no| AB{"adj_basis predates an already-ex action?"} + AB -->|yes| REST["refetchFullHistory: restate whole series"] + REST --> RR{"ok?"} + RR -->|yes| RET2["return fetched"] + RR -->|no| INC + AB -->|no| INC{"shouldRefresh?"} INC -->|no| BUMP["bump TTL, return cached"] - INC -->|yes| INCF["incremental fetch from last_date+1"] + INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"] RM -->|no| SN{"skip_network?"} SN -->|yes| FF SN -->|no| SS2["syncCandlesFromServer"] SS2 --> SF2{"fresh now?"} SF2 -->|yes| RET - SF2 -->|no| FULL["populateAllFromTiingo, full history"] + SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"] FULL --> RES{"result?"} - RES -->|ok| RET2["return fetched"] - RES -->|NotFound| WN["writeNegative candles_daily"] + RES -->|ok| RET2 + RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"] WN --> FF RES -->|transient| TR["bump fail_count, TransientError"] RES -->|other| FF ``` +Two things about this shape are easy to get wrong. + +**The basis check precedes `shouldRefresh`.** A symbol that needs both a +top-up and a restatement costs one full fetch, not an append followed by +a second pass. And a failed restatement falls through to the ordinary +top-up rather than erroring: the existing series is untouched and still +usable, just understated by the missed adjustment, so it retries on the +next stale pass. + +**Only a unanimous `NotFound` earns a negative entry.** `writeNegative` +*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such +symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and +`refetchFullHistory` returns `error.NotFound` only when every provider +disclaims the symbol. Anything else (auth trouble, a malformed body, a +network blip) fails the call but leaves the cache alone. The restatement +path above never writes a negative entry at all, for the same reason: it +is reached while holding a working series. + +The no-prior-cache branch does not re-check the basis after a server +sync, only freshness. A stale basis inherited from the server is caught +on the next invocation, which takes the meta-exists branch. One +invocation of understated returns, then it self-corrects. + Key invariant: the negative marker for a candle-less symbol lives in `candles_daily.srf`, and **every** candle decision honors it there - `isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the @@ -255,11 +357,13 @@ re-run the dead lookup on every invocation. The entry is the sentinel: ## Candle-less symbols (crypto and friends) -Some held symbols have **no daily candles available from the candle -provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` / +Some held symbols have **no daily candles available from any candle +provider** - cryptocurrencies on the Yahoo `DOGE-USD` / `BTC-USD` shape are the common case, and delisted or invalid tickers behave identically. For these symbols `getCandles` writes a negative -entry and never produces a price from history. +entry and never produces a price from history. "Any" is literal: Tiingo +and Yahoo must both disclaim the symbol, because the marker overwrites +`candles_daily.srf`. Such symbols are still priced, through two mechanisms that do **not** touch the candle cache: diff --git a/docs/explanation/caching.md b/docs/explanation/caching.md index e04d990..3575216 100644 --- a/docs/explanation/caching.md +++ b/docs/explanation/caching.md @@ -115,6 +115,22 @@ using a small `candles_meta.srf` companion file to track the last date and source provider. A ten-year history costs one big fetch the first time and tiny top-ups thereafter. +With one exception. A provider's *adjusted* close prices bake in every +dividend and split that happened after each bar, so when a distribution +goes ex, all the bars behind it need marking down - and appending new +bars can't do that to bars already on disk. Left alone, total returns +spanning that ex-date read low by roughly the missed dividend. + +So zfin tracks how current each series' adjustment basis is, and when a +dividend or split has gone ex behind it, re-downloads that symbol's full +history once to pick up the corrected values. For a quarterly dividend +payer that's about four full fetches a year, each within a day or so of +an ex-date. It's why a refresh run occasionally takes noticeably longer +than the usual top-up. + +`zfin diagnose SYMBOL` reports the basis for one symbol and says whether +a restatement is pending. + ## Negative caching When a provider permanently fails for a symbol -- a nonexistent diff --git a/src/commands/diagnose.zig b/src/commands/diagnose.zig index 922cf00..88c59a3 100644 --- a/src/commands/diagnose.zig +++ b/src/commands/diagnose.zig @@ -45,7 +45,10 @@ pub const meta: framework.Meta = .{ \\Usage: zfin diagnose SYMBOL \\ \\Reports, in order: - \\ local newest cached bar, TTL state, provider, failure count + \\ local newest cached bar, TTL state, provider, failure count, + \\ and any active Tiingo backoff + \\ adj basis how current the cached adj_close values are, against the + \\ newest dividend or split that has already gone ex \\ peers newest bar held by other symbols of the same kind \\ server what ZFIN_SERVER offers, whether it is ahead or behind, and \\ whether it refreshes this symbol at all