diff --git a/AGENTS.md b/AGENTS.md index cb2bbd7..5c42b0b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -441,7 +441,7 @@ User input -> main.zig (CLI dispatch) or tui.zig (TUI event loop) - **Providers are lazily initialized.** `DataService` fields like `td`, `pg`, `fh` start as `null` and are created on first use via `getProvider()`. The provider field name is derived from the type name at comptime. -- **Cache uses SRF format.** [SRF](https://git.lerch.org/lobo/srf) (Simple Record Format) is a line-oriented key-value format. Cache layout: `{cache_dir}/{SYMBOL}/{data_type}.srf`. Freshness is determined by file mtime vs TTL. +- **Cache uses SRF format.** [SRF](https://git.lerch.org/lobo/srf) (Simple Record Format) is a line-oriented key-value format. Cache layout: `{cache_dir}/{SYMBOL}/{data_type}.srf`. Freshness is determined by an `#!expires=` directive written into each cache file at fetch time (`now + TTL`) vs wall-clock time - NOT file mtime. - **Candles use incremental updates.** On cache miss, only candles newer than the last cached date are fetched (not the full 10-year history). The `candles_meta.srf` file tracks the last date and provider without deserializing the full candle file. diff --git a/README.md b/README.md index 40662fb..fb69353 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,10 @@ caching/TTL model, and the complete environment-variable list, see ## Architecture +The data/cache/fetch flow -- freshness, negative caching, incremental +candle updates, and the optional server tier -- is documented in +[Cache implementation](docs/dev/caching-implementation.md). + ``` src/ main.zig CLI dispatch + TUI entry point diff --git a/docs/README.md b/docs/README.md index e18a419..2de097c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -75,6 +75,7 @@ Background and the "why" behind zfin's behavior. - [Core concepts](explanation/concepts.md) - [Caching and data freshness](explanation/caching.md) + - [Cache implementation](dev/caching-implementation.md) -- contributor-level internals: layout, freshness model, fetch flowcharts - [Why multiple data providers](explanation/data-providers.md) - [Returns and performance](explanation/returns-and-performance.md) - [The retirement projection model](explanation/projections-model.md) diff --git a/docs/dev/caching-implementation.md b/docs/dev/caching-implementation.md new file mode 100644 index 0000000..78034c3 --- /dev/null +++ b/docs/dev/caching-implementation.md @@ -0,0 +1,352 @@ +# Cache implementation (developer reference) + +This is the low-level, contributor-facing companion to the user-facing +[Caching and data freshness](../explanation/caching.md) page. It +documents *how* the cache is built: the on-disk layout, the freshness +model, the fetch-decision flow, negative caching, and the optional +server (L2) tier. If you are changing anything in `src/cache/store.zig` +or the fetch paths in `src/service.zig`, read this first. + +Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders +natively. + +## Where the data lives + +`DataService` (`src/service.zig`) is the sole data-access layer. Both +the CLI and the TUI go through it; nothing else calls a provider +directly. It reads and writes a per-symbol, per-type SRF file cache via +`Store` (`src/cache/store.zig`): + +``` +{cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR + AAPL/ + candles_daily.srf OHLCV bars (append-only) + candles_meta.srf last_close, last_date, provider + freshness + dividends.srf + splits.srf + options.srf + earnings.srf + classification.srf + etf_metrics.srf + _edgar/ + tickers_companies.srf shared EDGAR ticker -> CIK maps + tickers_funds.srf + 0000320193/ + entity_facts.srf per-CIK XBRL facts +``` + +[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value +format. Files carry `#!`-prefixed directives (`#!expires=`, +`#!created=`) ahead of their records. + +### The candle two-file split + +Candles are stored as **two** files, and the split is load-bearing: + +- `candles_daily.srf` holds the actual OHLCV records and grows + append-only: on a cache miss only bars newer than `last_date` are + fetched and appended, never the full history. +- `candles_meta.srf` holds a single small record (`last_close`, + `last_date`, `provider`, `fail_count`) plus the `#!expires=` and + `#!created=` directives. + +Keeping the metadata separate lets every freshness check and last-price +read touch a ~100-byte file instead of deserializing a multi-megabyte +candle history. The price fast-path in `loadAllPrices` never +deserializes `candles_daily.srf` - at most it peeks at the first bytes +to detect a negative entry. + +The two files are a unit: `DataService.invalidate` and the torn-file +self-heal clear both together, and a negative cache entry for a +candle-less symbol is keyed off `candles_daily.srf` (see +[Negative caching](#negative-caching)). + +## Freshness is the `#!expires=` directive, not mtime + +A cache entry is fresh when the wall clock is earlier than the +`#!expires=` epoch-seconds directive embedded in the file. File +modification time is **not** consulted for freshness anywhere - mtime is +a fragile signal (it changes on copy, restore, rsync, and filesystem +quirks), so the expiry is written into the content itself. + +- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL` + for the data type, optionally offset by a per-key deterministic jitter + to avoid thundering-herd refreshes. +- On read, the SRF iterator parses `#!expires=` and `isFresh` compares + it to `Timestamp.now(io, .real)`. +- An entry with **no** `#!expires=` is treated as stale by zfin's + `.fresh_only` reads (a deliberate override of SRF's "no expiry = + always fresh" default), except for negative entries, which are always + fresh. + +### TTLs by data type + +Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site +in `DataType.ttl()`. + +| Data type | TTL | Jitter | Notes | +|----------------------|------------------|--------|--------------------------------------------------| +| Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)| +| Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable | +| Dividends | 14 days | 11% | Declared well in advance | +| Splits | 14 days | 11% | Rare corporate events | +| Options | 1 hour | 0 | Move continuously during market hours | +| Earnings | 30 days | 8% | Smart-refresh after an announcement date passes | +| Classification | 90 days | 8% | Sector/industry/country from Wikidata | +| ETF metrics | 90 days | 8% | NPORT-P profile, quarterly cadence | +| Entity facts (XBRL) | 30 days | 8% | Per-CIK, quarterly filing cadence | +| EDGAR ticker maps | 30 days | 8% | ticker -> CIK; very stable upstream | +| Quotes | never cached | n/a | Live by definition (see below) | + +### Market-aware candle freshness + +Daily bars are only meaningful once the session settles, so candle +expiry is keyed to the market clock rather than a rolling window +(`market.nextCandleExpiry` / `market.staleCandleExpiry`, +`market.shouldRefresh`): + +- Equities/ETFs expire at **16:55 ET** on the next trading day. +- Mutual funds (NAV) expire at **03:25 ET** the next morning. + +If a refresh fires but the provider has not posted the just-closed bar +yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes +overdue, the code concludes the session was an un-modeled closure (Good +Friday, weather) and falls back to the next normal boundary instead of +thrashing all day. See the user page for the cron-timing rationale. + +## The fetch decision + +### Tiers (the big picture) + +```mermaid +flowchart TD + A["Data request via DataService"] --> B{"Local cache fresh?"} + B -->|yes| C["Deserialize and return, no network"] + B -->|no| D{"ZFIN_SERVER set and not force_refresh?"} + D -->|yes| E["GET server, write bytes verbatim"] + E --> F{"Synced entry fresh?"} + F -->|yes| C + F -->|no| G["Provider fetch"] + D -->|no| G + G --> H{"Result?"} + H -->|ok| I["Write cache with new expiry, return"] + H -->|NotFound| J["Write negative cache, return FetchFailed"] + H -->|transient| K["Return error, retry next run"] +``` + +The `--refresh-data` policy maps to `FetchOptions`: + +- `auto` (default): all tiers, honor TTL. +- `force` -> `force_refresh = true`: skip the local-cache and server + tiers, go straight to the provider, re-stamp the cache. Bypasses + negative entries (so it retries dead lookups). +- `never` -> `skip_network = true`: stop at the local cache; return + stale data if present, never touch the network. + +### `getCandles` (single symbol) + +This is the most involved path because of the daily/meta split, the +incremental-update logic, and the TwelveData carve-out. + +```mermaid +flowchart TD + S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"} + NG -->|yes| FF["return FetchFailed, no network"] + NG -->|no| RM{"candles_meta exists?"} + + RM -->|yes| TW{"provider is twelvedata?"} + TW -->|yes| FULL + TW -->|no| FR{"meta fresh and not force_refresh?"} + FR -->|yes| RET["return cached candles"] + FR -->|no| SS1["syncCandlesFromServer"] + SS1 --> SF1{"fresh now?"} + SF1 -->|yes| RET + SF1 -->|no| INC{"shouldRefresh?"} + INC -->|no| BUMP["bump TTL, return cached"] + INC -->|yes| INCF["incremental fetch from last_date+1"] + + RM -->|no| SN{"skip_network?"} + SN -->|yes| FF + SN -->|no| SS2["syncCandlesFromServer"] + SS2 --> SF2{"fresh now?"} + SF2 -->|yes| RET + SF2 -->|no| FULL["populateAllFromTiingo, full history"] + FULL --> RES{"result?"} + RES -->|ok| RET2["return fetched"] + RES -->|NotFound| WN["writeNegative candles_daily"] + WN --> FF + RES -->|transient| TR["bump fail_count, TransientError"] + RES -->|other| FF +``` + +Key invariant: the negative marker for a candle-less symbol lives in +`candles_daily.srf`, and **every** candle decision honors it there - +`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the +cache-only display path), and the `getCandles` short-circuit above. This +matters because `candles_meta.srf` is never created for a symbol that +has no candles, so anything keying freshness off the meta file alone +would treat such a symbol as perpetually stale and re-fetch it forever. + +### `fetchCached` (dividends, splits, options, earnings, ...) + +Everything that is not candles flows through the generic `fetchCached`, +which is simpler because each type is a single file: + +1. `.fresh_only` read; a fresh entry (including a negative one) returns + immediately. +2. `skip_network`: return any cached entry, even stale; else + `FetchFailed`. +3. Server sync (if configured); a fresh synced entry returns. +4. Provider fetch; on success write with the type's TTL; on + `NotFound` write a negative entry; on transient error return + `FetchFailed` without poisoning the cache. + +### `loadAllPrices` (portfolio + watchlist price load) + +The portfolio price load batches all symbols through three phases. +Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider +fallback that calls `getCandles`. + +```mermaid +flowchart TD + ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"] + P1 --> CF{"cache fresh and not force_refresh?"} + CF -->|yes| HIT["use cached last close, cached_count++"] + CF -->|no| ADD["add to needs_fetch"] + HIT --> CHK + ADD --> CHK{"needs_fetch empty?"} + CHK -->|yes| DONE["return"] + CHK -->|no| OFF{"skip_network?"} + OFF -->|yes| STALE["stale-cache fallback or failed_count++"] + OFF -->|no| HASSRV{"ZFIN_SERVER set?"} + HASSRV -->|yes| P2["Phase 2: parallelServerSync"] + HASSRV -->|no| ALLF["all needs_fetch to server_failures"] + P2 --> REM["unsynced to server_failures"] + ALLF --> P3 + REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"] + P3 --> END["return prices + counts"] +``` + +## Negative caching + +When a provider says a symbol genuinely has no data of a type - an +`error.NotFound` - zfin writes a **negative cache entry** so it does not +re-run the dead lookup on every invocation. The entry is the sentinel: + +``` +#!srfv1 +# fetch_failed +``` + +(`Store.negative_cache_content`). Rules: + +- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the + write. Rate-limit, 5xx, connection, auth, and parse failures are + transient - they fail the call but leave the cache untouched so the + next run retries. (Auth/parse looking permanent but being transient is + exactly why they must not poison a now-sticky negative cache.) +- **Negative entries are always fresh.** They have no `#!expires=`; + `readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel + as fresh, so they stick until `--refresh-data=force` or `cache clear`. +- **Candles key the negative off `candles_daily.srf`.** `writeNegative` + writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the + `getCandles` short-circuit all recognize it there. `candles_meta.srf` + is intentionally not created for a no-data symbol. + +## Candle-less symbols (crypto and friends) + +Some held symbols have **no daily candles available from the candle +provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` / +`BTC-USD` shape are the common case, and delisted or invalid tickers +behave identically. For these symbols `getCandles` writes a negative +entry and never produces a price from history. + +Such symbols are still priced, through two mechanisms that do **not** +touch the candle cache: + +- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an + intraday price from Yahoo, which *does* serve crypto. The TUI overlays + these live quotes on top of the candle-close price map on refresh and + on every streaming tick, so a candle-less holding shows its real + current price there. Live quotes are never cached. +- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a + value. When neither a candle close nor a live quote is available, + `buildFallbackPrices` (`analytics/valuation.zig`) falls back to the + position's average cost and flags it as a manual/estimated price + (rendered in a warning color). + +Practical consequence by surface: the plain CLI `portfolio` command does +not apply the live-quote overlay, so a candle-less holding shows its +average-cost fallback (break-even, warning color); the TUI shows the +live Yahoo price. This is expected - historical-candle commands (`perf`, +charts) simply have no data for these symbols, while quote-driven views +do. + +If you want a candle-less symbol to be re-checked against the provider +(for example a ticker that has since started trading), clear its +negative entry with `cache clear` or `--refresh-data=force`; the live +quote and manual-price paths are unaffected by the negative cache. + +## Server sync (the optional L2 tier) + +`ZFIN_SERVER` points zfin at a +[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a +shared cache between your local cache and the upstream providers. When +unset, every server-sync path silently no-ops. + +Client side (`syncFromServer`, `syncCandlesFromServer`, +`parallelServerSync` in `service.zig`): + +- Triggered on a local miss/stale entry, before any provider call + (skipped under `force_refresh`). +- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated + (sha256 ETag, completeness check) and written to the local cache + **verbatim** via `writeRaw` - the client does not re-stamp + `#!expires=`, so it inherits the server's freshness boundary. +- `parallelServerSync` fans out one task per symbol for the portfolio + price load (each worker uses its own HTTP client; the allocator is + thread-safe). + +Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`): + +- A **present** file is served as-is, even if stale - the server's cron + is the freshness authority; reads never trigger a refetch. +- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the + same `getCandles` / `fetchCached` code through the shared zfin + library), then re-reads; if still absent, it returns 404. + +Because the server runs the same library, the negative-cache rules above +apply there too: a candle-less symbol gets a negative `candles_daily.srf` +on first miss and is served from it thereafter, rather than re-hitting +the upstream provider on every request. + +## Invalidation and atomicity + +- `DataService.invalidate(symbol)` clears a symbol's entries; for + candles it removes the `candles_daily` + `candles_meta` pair together. +- `cache clear` wipes the whole cache directory; everything re-fetches + next run. +- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs, + and renames into place, so a reader never sees a torn file. A + defensively detected torn candle file self-heals by wiping the pair. + +## Key code references + +| Concern | Location | +|---------------------------------|-----------------------------------------------------| +| Data-access entry point | `DataService` - `src/service.zig` | +| Per-type generic fetch | `fetchCached` - `src/service.zig` | +| Candle fetch + incremental | `getCandles` - `src/service.zig` | +| Batch price load | `loadAllPrices` - `src/service.zig` | +| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` | +| Cache store, read/write | `Store` - `src/cache/store.zig` | +| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` | +| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` | +| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` | +| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` | +| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` | +| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` | +| Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` | + +For the user-facing summary and the `--refresh-data` walkthrough, see +[Caching and data freshness](../explanation/caching.md). diff --git a/docs/explanation/caching.md b/docs/explanation/caching.md index 3973b83..e04d990 100644 --- a/docs/explanation/caching.md +++ b/docs/explanation/caching.md @@ -21,9 +21,11 @@ can satisfy it: 3. **Provider.** Otherwise zfin fetches from the upstream provider, writes the result to the cache, and returns it. -Freshness is decided by the cache file's modification time versus the -TTL for that data type. The cache directory defaults to `~/.cache/zfin` -and is set with `ZFIN_CACHE_DIR`. +Freshness is decided by an expiry timestamp written into each cache +file when it is fetched (an `#!expires=` line), compared against the +current time -- not the file's modification time. Each data type sets +its own TTL, which determines that expiry (see below). The cache +directory defaults to `~/.cache/zfin` and is set with `ZFIN_CACHE_DIR`. The `--refresh-data` policy decides which tiers run: @@ -120,6 +122,32 @@ ticker, say -- zfin records a negative cache entry so it doesn't retry the same dead lookup on every run. (Transient failures like rate limits are not cached this way; they're retried.) +## Symbols without candle data (crypto) + +A few holdings have no daily price history available from zfin's candle +provider -- cryptocurrencies (held as `DOGE-USD`, `BTC-USD`, and the +like) are the common case, and long-delisted tickers behave the same +way. zfin negative-caches these so it stops asking for candles that will +never arrive. + +Such holdings are still valued, just not from price history: + +- **In the TUI** they are priced from **live quotes** (Yahoo serves + crypto), refreshed on load and on every streaming tick. +- A **manual price** wins when set: a `price::` field on the lot in + `portfolio.srf` pins the value. +- Otherwise zfin falls back to your **average cost** for the holding and + flags it in a warning color, so a candle-less position shows at + break-even rather than disappearing. + +Because the plain `portfolio` CLI command doesn't fetch live quotes, a +candle-less holding shows its average-cost fallback there; open the TUI +(or set a `price::`) to see a live value. History-based commands like +[`perf`](../reference/cli/perf.md) and the price charts simply have no +data for these symbols. To force a re-check (say a ticker started +trading), clear the entry with `--refresh-data=force` or `zfin cache +clear`. + ## Rate limiting Each provider has a client-side token-bucket limiter sized to its @@ -164,3 +192,7 @@ do: `zfin cache clear` wipes it (everything re-fetches next run). See [Offline use and refreshing data](../guides/offline-and-refresh.md). + +For the low-level implementation -- on-disk layout, the freshness model, +negative caching, and the fetch-decision flowcharts -- see +[Cache implementation](../dev/caching-implementation.md).