correct caching docs for adjustment-basis restatement

docs/dev/caching-implementation.md

@@ -20,8 +20,8 @@ directly. It reads and writes a per-symbol, per-type SRF file cache via
 ```
 {cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
   AAPL/
-    candles_daily.srf             OHLCV bars (append-only)
+    candles_daily.srf             OHLCV bars (appended; replaced on restatement)
-    candles_meta.srf              last_close, last_date, provider + freshness
+    candles_meta.srf              last_close, last_date, provider, adj_basis + freshness
     dividends.srf
     splits.srf
     options.srf
@@ -43,12 +43,14 @@ format. Files carry `#!`-prefixed directives (`#!expires=`,
 
 Candles are stored as **two** files, and the split is load-bearing:
 
-- `candles_daily.srf` holds the actual OHLCV records and grows
+- `candles_daily.srf` holds the actual OHLCV records and normally grows
-  append-only: on a cache miss only bars newer than `last_date` are
+  by appending: on a cache miss only bars newer than `last_date` are
-  fetched and appended, never the full history.
+  fetched and appended, not the full history. The exception is a
+  **restatement**, which replaces the file wholesale - see
+  [The adjustment basis](#the-adjustment-basis).
 - `candles_meta.srf` holds a single small record (`last_close`,
-  `last_date`, `provider`, `fail_count`) plus the `#!expires=` and
+  `last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`,
-  `#!created=` directives.
+  `adj_basis`) plus the `#!expires=` and `#!created=` directives.
 
 Keeping the metadata separate lets every freshness check and last-price
 read touch a ~100-byte file instead of deserializing a multi-megabyte
@@ -61,6 +63,62 @@ self-heal clear both together, and a negative cache entry for a
 candle-less symbol is keyed off `candles_daily.srf` (see
 [Negative caching](#negative-caching)).
 
+### The adjustment basis
+
+Appending bars cannot restate the bars already on disk, and sometimes
+they need it.
+
+Providers compute `adj_close` by scaling raw `close` by the product of
+the adjustment factors for every distribution *after* that bar. So a
+series' adjustment basis is only as current as the fetch that produced
+it. Newly appended bars arrive with `adj_close == close`, because
+nothing has gone ex after them yet, while every previously cached bar
+keeps the basis it was originally fetched with. When the next
+distribution goes ex, the bars behind it should be marked down by its
+factor - and an append does not do that. Every total return spanning
+that ex-date then reads low by roughly the missed yield.
+
+`CandleMeta.adj_basis` records how current a series' basis is. It is
+**the date of the newest bar present at the last full fetch**:
+
+- Not a wall-clock timestamp. A fetch that runs before the day's bar is
+  published gets a basis one bar behind - which is the honest claim,
+  since a provider's adjusted series is only ever as complete as its
+  newest bar.
+- Not an ex-date. It is compared against ex-dates but is never one.
+- Never advanced by an append. `Store.appendCandles` takes the existing
+  meta and overrides only `last_close` / `last_date`; only
+  `Store.cacheCandles`, which replaces the whole file, sets a new
+  basis. That asymmetry is the mechanism, so keep it.
+
+`getCandles` escalates from append to full refetch when
+`freshness.adjustmentBasisStale` says the basis predates a corporate
+action that has **already gone ex**:
+
+```zig
+newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
+stale     = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)
+```
+
+The already-ex bound lives in `newestCorporateAction`, not in the
+verdict, and that placement is load-bearing in both directions. A
+declared-but-not-yet-ex distribution is reflected in no provider's
+adjusted series, so treating it as something to catch up to would
+refetch on every pass forever. But bounding the *verdict* instead -
+"is the newest action of all still in the future? then nothing to do" -
+lets one forward announcement hide every older unapplied action behind
+it. That shipped: a quarterly payer announcing a quarter ahead was
+permanently unable to restate.
+
+The check runs on the stale path only, not on the fresh-cache
+early-return. Detection is therefore at most one trading day behind,
+which the candle TTL guarantees, and the hot portfolio-pricing path
+pays nothing.
+
+Steady state for a quarterly payer is four full-history fetches a year,
+each landing within about a trading day of an ex-date. `zfin diagnose
+SYMBOL` reports the basis against the newest already-ex action.
+
 ## Freshness is the `#!expires=` directive, not mtime
 
 A cache entry is fresh when the wall clock is earlier than the
@@ -146,7 +204,21 @@ The `--refresh-data` policy maps to `FetchOptions`:
 ### `getCandles` (single symbol)
 
 This is the most involved path because of the daily/meta split, the
-incremental-update logic, and the TwelveData carve-out.
+incremental-update logic, the adjustment-basis escalation, and the
+TwelveData carve-out.
+
+Note also that `force` re-asks the provider but does **not** rebuild
+candle history: it skips the TTL and the server tier, then takes the
+same incremental top-up. Replacing the series is the restatement path's
+job (or `zfin cache clear`).
+
+Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off
+`provider`. `provider` is pure provenance - "where did these bars come
+from" - and using it to route made a single non-transient Tiingo failure
+permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`,
+and Tiingo was never consulted again. The backoff is armed only by a
+genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
+and clears the moment Tiingo serves the symbol again.
 
 ```mermaid
 flowchart TD
@@ -154,31 +226,61 @@ flowchart TD
     NG -->|yes| FF["return FetchFailed, no network"]
     NG -->|no| RM{"candles_meta exists?"}
 
-    RM -->|yes| TW{"provider is twelvedata?"}
+    RM -->|yes| SK{"skip_network?"}
+    SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
+    SK -->|no| TW{"provider is twelvedata?"}
     TW -->|yes| FULL
     TW -->|no| FR{"meta fresh and not force_refresh?"}
     FR -->|yes| RET["return cached candles"]
     FR -->|no| SS1["syncCandlesFromServer"]
-    SS1 --> SF1{"fresh now?"}
+    SS1 --> SF1{"fresh AND adj_basis current?"}
     SF1 -->|yes| RET
-    SF1 -->|no| INC{"shouldRefresh?"}
+    SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
+    AB -->|yes| REST["refetchFullHistory: restate whole series"]
+    REST --> RR{"ok?"}
+    RR -->|yes| RET2["return fetched"]
+    RR -->|no| INC
+    AB -->|no| INC{"shouldRefresh?"}
     INC -->|no| BUMP["bump TTL, return cached"]
-    INC -->|yes| INCF["incremental fetch from last_date+1"]
+    INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]
 
     RM -->|no| SN{"skip_network?"}
     SN -->|yes| FF
     SN -->|no| SS2["syncCandlesFromServer"]
     SS2 --> SF2{"fresh now?"}
     SF2 -->|yes| RET
-    SF2 -->|no| FULL["populateAllFromTiingo, full history"]
+    SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
     FULL --> RES{"result?"}
-    RES -->|ok| RET2["return fetched"]
+    RES -->|ok| RET2
-    RES -->|NotFound| WN["writeNegative candles_daily"]
+    RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"]
     WN --> FF
     RES -->|transient| TR["bump fail_count, TransientError"]
     RES -->|other| FF
 ```
 
+Two things about this shape are easy to get wrong.
+
+**The basis check precedes `shouldRefresh`.** A symbol that needs both a
+top-up and a restatement costs one full fetch, not an append followed by
+a second pass. And a failed restatement falls through to the ordinary
+top-up rather than erroring: the existing series is untouched and still
+usable, just understated by the missed adjustment, so it retries on the
+next stale pass.
+
+**Only a unanimous `NotFound` earns a negative entry.** `writeNegative`
+*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such
+symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and
+`refetchFullHistory` returns `error.NotFound` only when every provider
+disclaims the symbol. Anything else (auth trouble, a malformed body, a
+network blip) fails the call but leaves the cache alone. The restatement
+path above never writes a negative entry at all, for the same reason: it
+is reached while holding a working series.
+
+The no-prior-cache branch does not re-check the basis after a server
+sync, only freshness. A stale basis inherited from the server is caught
+on the next invocation, which takes the meta-exists branch. One
+invocation of understated returns, then it self-corrects.
+
 Key invariant: the negative marker for a candle-less symbol lives in
 `candles_daily.srf`, and **every** candle decision honors it there -
 `isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
@@ -255,11 +357,13 @@ re-run the dead lookup on every invocation. The entry is the sentinel:
 
 ## Candle-less symbols (crypto and friends)
 
-Some held symbols have **no daily candles available from the candle
+Some held symbols have **no daily candles available from any candle
-provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` /
+provider** - cryptocurrencies on the Yahoo `DOGE-USD` /
 `BTC-USD` shape are the common case, and delisted or invalid tickers
 behave identically. For these symbols `getCandles` writes a negative
-entry and never produces a price from history.
+entry and never produces a price from history. "Any" is literal: Tiingo
+and Yahoo must both disclaim the symbol, because the marker overwrites
+`candles_daily.srf`.
 
 Such symbols are still priced, through two mechanisms that do **not**
 touch the candle cache:

docs/explanation/caching.md

@@ -115,6 +115,22 @@ using a small `candles_meta.srf` companion file to track the last date
 and source provider. A ten-year history costs one big fetch the first
 time and tiny top-ups thereafter.
 
+With one exception. A provider's *adjusted* close prices bake in every
+dividend and split that happened after each bar, so when a distribution
+goes ex, all the bars behind it need marking down - and appending new
+bars can't do that to bars already on disk. Left alone, total returns
+spanning that ex-date read low by roughly the missed dividend.
+
+So zfin tracks how current each series' adjustment basis is, and when a
+dividend or split has gone ex behind it, re-downloads that symbol's full
+history once to pick up the corrected values. For a quarterly dividend
+payer that's about four full fetches a year, each within a day or so of
+an ex-date. It's why a refresh run occasionally takes noticeably longer
+than the usual top-up.
+
+`zfin diagnose SYMBOL` reports the basis for one symbol and says whether
+a restatement is pending.
+
 ## Negative caching
 
 When a provider permanently fails for a symbol -- a nonexistent

src/commands/diagnose.zig

@@ -45,7 +45,10 @@ pub const meta: framework.Meta = .{
     \\Usage: zfin diagnose SYMBOL
     \\
     \\Reports, in order:
-    \\  local      newest cached bar, TTL state, provider, failure count
+    \\  local      newest cached bar, TTL state, provider, failure count,
+    \\             and any active Tiingo backoff
+    \\  adj basis  how current the cached adj_close values are, against the
+    \\             newest dividend or split that has already gone ex
     \\  peers      newest bar held by other symbols of the same kind
     \\  server     what ZFIN_SERVER offers, whether it is ahead or behind, and
     \\             whether it refreshes this symbol at all