lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

correct caching docs for adjustment-basis restatement

Browse source
This commit is contained in:
Emil Lerch 2026-08-19 14:59:42 -07:00
parent c15ea92b0d
commit ff11940a88
Signed by: lobo
GPG key ID: A7B62D657EF764F8
3 changed files with 142 additions and 19 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

140
docs/dev/caching-implementation.md
View file

@ -20,8 +20,8 @@ directly. It reads and writes a per-symbol, per-type SRF file cache via
```
{cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR
AAPL/
candles_daily.srf OHLCV bars (append-only)
candles_meta.srf last_close, last_date, provider + freshness
candles_daily.srf OHLCV bars (appended; replaced on restatement)
candles_meta.srf last_close, last_date, provider, adj_basis + freshness
dividends.srf
splits.srf
options.srf
@ -43,12 +43,14 @@ format. Files carry `#!`-prefixed directives (`#!expires=`,
Candles are stored as **two** files, and the split is load-bearing:
- `candles_daily.srf` holds the actual OHLCV records and grows
append-only: on a cache miss only bars newer than `last_date` are
fetched and appended, never the full history.
- `candles_daily.srf` holds the actual OHLCV records and normally grows
by appending: on a cache miss only bars newer than `last_date` are
fetched and appended, not the full history. The exception is a
**restatement**, which replaces the file wholesale - see
[The adjustment basis](#the-adjustment-basis).
- `candles_meta.srf` holds a single small record (`last_close`,
`last_date`, `provider`, `fail_count`) plus the `#!expires=` and
`#!created=` directives.
`last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`,
`adj_basis`) plus the `#!expires=` and `#!created=` directives.
Keeping the metadata separate lets every freshness check and last-price
read touch a ~100-byte file instead of deserializing a multi-megabyte
@ -61,6 +63,62 @@ self-heal clear both together, and a negative cache entry for a
candle-less symbol is keyed off `candles_daily.srf` (see
[Negative caching](#negative-caching)).
### The adjustment basis
Appending bars cannot restate the bars already on disk, and sometimes
they need it.
Providers compute `adj_close` by scaling raw `close` by the product of
the adjustment factors for every distribution *after* that bar. So a
series' adjustment basis is only as current as the fetch that produced
it. Newly appended bars arrive with `adj_close == close`, because
nothing has gone ex after them yet, while every previously cached bar
keeps the basis it was originally fetched with. When the next
distribution goes ex, the bars behind it should be marked down by its
factor - and an append does not do that. Every total return spanning
that ex-date then reads low by roughly the missed yield.
`CandleMeta.adj_basis` records how current a series' basis is. It is
**the date of the newest bar present at the last full fetch**:
- Not a wall-clock timestamp. A fetch that runs before the day's bar is
published gets a basis one bar behind - which is the honest claim,
since a provider's adjusted series is only ever as complete as its
newest bar.
- Not an ex-date. It is compared against ex-dates but is never one.
- Never advanced by an append. `Store.appendCandles` takes the existing
meta and overrides only `last_close` / `last_date`; only
`Store.cacheCandles`, which replaces the whole file, sets a new
basis. That asymmetry is the mechanism, so keep it.
`getCandles` escalates from append to full refetch when
`freshness.adjustmentBasisStale` says the basis predates a corporate
action that has **already gone ex**:
```zig
newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)
```
The already-ex bound lives in `newestCorporateAction`, not in the
verdict, and that placement is load-bearing in both directions. A
declared-but-not-yet-ex distribution is reflected in no provider's
adjusted series, so treating it as something to catch up to would
refetch on every pass forever. But bounding the *verdict* instead -
"is the newest action of all still in the future? then nothing to do" -
lets one forward announcement hide every older unapplied action behind
it. That shipped: a quarterly payer announcing a quarter ahead was
permanently unable to restate.
The check runs on the stale path only, not on the fresh-cache
early-return. Detection is therefore at most one trading day behind,
which the candle TTL guarantees, and the hot portfolio-pricing path
pays nothing.
Steady state for a quarterly payer is four full-history fetches a year,
each landing within about a trading day of an ex-date. `zfin diagnose
SYMBOL` reports the basis against the newest already-ex action.
## Freshness is the `#!expires=` directive, not mtime
A cache entry is fresh when the wall clock is earlier than the
@ -146,7 +204,21 @@ The `--refresh-data` policy maps to `FetchOptions`:
### `getCandles` (single symbol)
This is the most involved path because of the daily/meta split, the
incremental-update logic, and the TwelveData carve-out.
incremental-update logic, the adjustment-basis escalation, and the
TwelveData carve-out.
Note also that `force` re-asks the provider but does **not** rebuild
candle history: it skips the TTL and the server tier, then takes the
same incremental top-up. Replacing the series is the restatement path's
job (or `zfin cache clear`).
Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off
`provider`. `provider` is pure provenance - "where did these bars come
from" - and using it to route made a single non-transient Tiingo failure
permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`,
and Tiingo was never consulted again. The backoff is armed only by a
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
and clears the moment Tiingo serves the symbol again.
```mermaid
flowchart TD
@ -154,31 +226,61 @@ flowchart TD
NG -->|yes| FF["return FetchFailed, no network"]
NG -->|no| RM{"candles_meta exists?"}
RM -->|yes| TW{"provider is twelvedata?"}
RM -->|yes| SK{"skip_network?"}
SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
SK -->|no| TW{"provider is twelvedata?"}
TW -->|yes| FULL
TW -->|no| FR{"meta fresh and not force_refresh?"}
FR -->|yes| RET["return cached candles"]
FR -->|no| SS1["syncCandlesFromServer"]
SS1 --> SF1{"fresh now?"}
SS1 --> SF1{"fresh AND adj_basis current?"}
SF1 -->|yes| RET
SF1 -->|no| INC{"shouldRefresh?"}
SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
AB -->|yes| REST["refetchFullHistory: restate whole series"]
REST --> RR{"ok?"}
RR -->|yes| RET2["return fetched"]
RR -->|no| INC
AB -->|no| INC{"shouldRefresh?"}
INC -->|no| BUMP["bump TTL, return cached"]
INC -->|yes| INCF["incremental fetch from last_date+1"]
INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]
RM -->|no| SN{"skip_network?"}
SN -->|yes| FF
SN -->|no| SS2["syncCandlesFromServer"]
SS2 --> SF2{"fresh now?"}
SF2 -->|yes| RET
SF2 -->|no| FULL["populateAllFromTiingo, full history"]
SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
FULL --> RES{"result?"}
RES -->|ok| RET2["return fetched"]
RES -->|NotFound| WN["writeNegative candles_daily"]
RES -->|ok| RET2
RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"]
WN --> FF
RES -->|transient| TR["bump fail_count, TransientError"]
RES -->|other| FF
```
Two things about this shape are easy to get wrong.
**The basis check precedes `shouldRefresh`.** A symbol that needs both a
top-up and a restatement costs one full fetch, not an append followed by
a second pass. And a failed restatement falls through to the ordinary
top-up rather than erroring: the existing series is untouched and still
usable, just understated by the missed adjustment, so it retries on the
next stale pass.
**Only a unanimous `NotFound` earns a negative entry.** `writeNegative`
*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such
symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and
`refetchFullHistory` returns `error.NotFound` only when every provider
disclaims the symbol. Anything else (auth trouble, a malformed body, a
network blip) fails the call but leaves the cache alone. The restatement
path above never writes a negative entry at all, for the same reason: it
is reached while holding a working series.
The no-prior-cache branch does not re-check the basis after a server
sync, only freshness. A stale basis inherited from the server is caught
on the next invocation, which takes the meta-exists branch. One
invocation of understated returns, then it self-corrects.
Key invariant: the negative marker for a candle-less symbol lives in
`candles_daily.srf`, and **every** candle decision honors it there -
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
@ -255,11 +357,13 @@ re-run the dead lookup on every invocation. The entry is the sentinel:
## Candle-less symbols (crypto and friends)
Some held symbols have **no daily candles available from the candle
provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` /
Some held symbols have **no daily candles available from any candle
provider** - cryptocurrencies on the Yahoo `DOGE-USD` /
`BTC-USD` shape are the common case, and delisted or invalid tickers
behave identically. For these symbols `getCandles` writes a negative
entry and never produces a price from history.
entry and never produces a price from history. "Any" is literal: Tiingo
and Yahoo must both disclaim the symbol, because the marker overwrites
`candles_daily.srf`.
Such symbols are still priced, through two mechanisms that do **not**
touch the candle cache:

16
docs/explanation/caching.md
View file

@ -115,6 +115,22 @@ using a small `candles_meta.srf` companion file to track the last date
and source provider. A ten-year history costs one big fetch the first
time and tiny top-ups thereafter.
With one exception. A provider's *adjusted* close prices bake in every
dividend and split that happened after each bar, so when a distribution
goes ex, all the bars behind it need marking down - and appending new
bars can't do that to bars already on disk. Left alone, total returns
spanning that ex-date read low by roughly the missed dividend.
So zfin tracks how current each series' adjustment basis is, and when a
dividend or split has gone ex behind it, re-downloads that symbol's full
history once to pick up the corrected values. For a quarterly dividend
payer that's about four full fetches a year, each within a day or so of
an ex-date. It's why a refresh run occasionally takes noticeably longer
than the usual top-up.
`zfin diagnose SYMBOL` reports the basis for one symbol and says whether
a restatement is pending.
## Negative caching
When a provider permanently fails for a symbol -- a nonexistent

5
src/commands/diagnose.zig
View file

@ -45,7 +45,10 @@ pub const meta: framework.Meta = .{
\\Usage: zfin diagnose SYMBOL
\\
\\Reports, in order:
\\ local newest cached bar, TTL state, provider, failure count
\\ local newest cached bar, TTL state, provider, failure count,
\\ and any active Tiingo backoff
\\ adj basis how current the cached adj_close values are, against the
\\ newest dividend or split that has already gone ex
\\ peers newest bar held by other symbols of the same kind
\\ server what ZFIN_SERVER offers, whether it is ahead or behind, and
\\ whether it refreshes this symbol at all