update docs/include dev docs on caching implementation

2026-06-30 17:15:52 -07:00 · 2026-06-30 17:15:52 -07:00 · 2e9fed48ff
commit 2e9fed48ff
parent d15e4d024a
5 changed files with 393 additions and 4 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@ -441,7 +441,7 @@ User input -> main.zig (CLI dispatch) or tui.zig (TUI event loop)

 - **Providers are lazily initialized.** `DataService` fields like `td`, `pg`, `fh` start as `null` and are created on first use via `getProvider()`. The provider field name is derived from the type name at comptime.

- **Cache uses SRF format.** [SRF](https://git.lerch.org/lobo/srf) (Simple Record Format) is a line-oriented key-value format. Cache layout: `{cache_dir}/{SYMBOL}/{data_type}.srf`. Freshness is determined by file mtime vs TTL.
+- **Cache uses SRF format.** [SRF](https://git.lerch.org/lobo/srf) (Simple Record Format) is a line-oriented key-value format. Cache layout: `{cache_dir}/{SYMBOL}/{data_type}.srf`. Freshness is determined by an `#!expires=` directive written into each cache file at fetch time (`now + TTL`) vs wall-clock time - NOT file mtime.

 - **Candles use incremental updates.** On cache miss, only candles newer than the last cached date are fetched (not the full 10-year history). The `candles_meta.srf` file tracks the last date and provider without deserializing the full candle file.

--- a/README.md
+++ b/README.md
@ -100,6 +100,10 @@ caching/TTL model, and the complete environment-variable list, see

 ## Architecture

+The data/cache/fetch flow -- freshness, negative caching, incremental
+candle updates, and the optional server tier -- is documented in
+[Cache implementation](docs/dev/caching-implementation.md).
+
 ```
 src/
  main.zig                CLI dispatch + TUI entry point
--- a/docs/README.md
+++ b/docs/README.md
@ -75,6 +75,7 @@ Background and the "why" behind zfin's behavior.

 - [Core concepts](explanation/concepts.md)
 - [Caching and data freshness](explanation/caching.md)
+  - [Cache implementation](dev/caching-implementation.md) -- contributor-level internals: layout, freshness model, fetch flowcharts
 - [Why multiple data providers](explanation/data-providers.md)
 - [Returns and performance](explanation/returns-and-performance.md)
 - [The retirement projection model](explanation/projections-model.md)
--- a/docs/dev/caching-implementation.md
+++ b/docs/dev/caching-implementation.md
@ -0,0 +1,352 @@
+# Cache implementation (developer reference)
+
+This is the low-level, contributor-facing companion to the user-facing
+[Caching and data freshness](../explanation/caching.md) page. It
+documents *how* the cache is built: the on-disk layout, the freshness
+model, the fetch-decision flow, negative caching, and the optional
+server (L2) tier. If you are changing anything in `src/cache/store.zig`
+or the fetch paths in `src/service.zig`, read this first.
+
+Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
+natively.
+
+## Where the data lives
+
+`DataService` (`src/service.zig`) is the sole data-access layer. Both
+the CLI and the TUI go through it; nothing else calls a provider
+directly. It reads and writes a per-symbol, per-type SRF file cache via
+`Store` (`src/cache/store.zig`):
+
+```
+{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
+  AAPL/
+    candles_daily.srf             OHLCV bars (append-only)
+    candles_meta.srf              last_close, last_date, provider + freshness
+    dividends.srf
+    splits.srf
+    options.srf
+    earnings.srf
+    classification.srf
+    etf_metrics.srf
+  _edgar/
+    tickers_companies.srf         shared EDGAR ticker -> CIK maps
+    tickers_funds.srf
+  0000320193/
+    entity_facts.srf              per-CIK XBRL facts
+```
+
+[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
+format. Files carry `#!`-prefixed directives (`#!expires=`,
+`#!created=`) ahead of their records.
+
+### The candle two-file split
+
+Candles are stored as **two** files, and the split is load-bearing:
+
+- `candles_daily.srf` holds the actual OHLCV records and grows
+  append-only: on a cache miss only bars newer than `last_date` are
+  fetched and appended, never the full history.
+- `candles_meta.srf` holds a single small record (`last_close`,
+  `last_date`, `provider`, `fail_count`) plus the `#!expires=` and
+  `#!created=` directives.
+
+Keeping the metadata separate lets every freshness check and last-price
+read touch a ~100-byte file instead of deserializing a multi-megabyte
+candle history. The price fast-path in `loadAllPrices` never
+deserializes `candles_daily.srf` - at most it peeks at the first bytes
+to detect a negative entry.
+
+The two files are a unit: `DataService.invalidate` and the torn-file
+self-heal clear both together, and a negative cache entry for a
+candle-less symbol is keyed off `candles_daily.srf` (see
+[Negative caching](#negative-caching)).
+
+## Freshness is the `#!expires=` directive, not mtime
+
+A cache entry is fresh when the wall clock is earlier than the
+`#!expires=` epoch-seconds directive embedded in the file. File
+modification time is **not** consulted for freshness anywhere - mtime is
+a fragile signal (it changes on copy, restore, rsync, and filesystem
+quirks), so the expiry is written into the content itself.
+
+- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
+  for the data type, optionally offset by a per-key deterministic jitter
+  to avoid thundering-herd refreshes.
+- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
+  it to `Timestamp.now(io, .real)`.
+- An entry with **no** `#!expires=` is treated as stale by zfin's
+  `.fresh_only` reads (a deliberate override of SRF's "no expiry =
+  always fresh" default), except for negative entries, which are always
+  fresh.
+
+### TTLs by data type
+
+Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
+in `DataType.ttl()`.
+
+| Data type            | TTL              | Jitter | Notes                                            |
+|----------------------|------------------|--------|--------------------------------------------------|
+| Daily candles        | market-aware     | n/a    | Boundary set by `market.nextCandleExpiry` (below)|
+| Historical candles   | never (`-1`)     | n/a    | Bars older than ~1 day are immutable             |
+| Dividends            | 14 days          | 11%    | Declared well in advance                         |
+| Splits               | 14 days          | 11%    | Rare corporate events                            |
+| Options              | 1 hour           | 0      | Move continuously during market hours            |
+| Earnings             | 30 days          | 8%     | Smart-refresh after an announcement date passes  |
+| Classification       | 90 days          | 8%     | Sector/industry/country from Wikidata            |
+| ETF metrics          | 90 days          | 8%     | NPORT-P profile, quarterly cadence               |
+| Entity facts (XBRL)  | 30 days          | 8%     | Per-CIK, quarterly filing cadence                |
+| EDGAR ticker maps    | 30 days          | 8%     | ticker -> CIK; very stable upstream              |
+| Quotes               | never cached     | n/a    | Live by definition (see below)                   |
+
+### Market-aware candle freshness
+
+Daily bars are only meaningful once the session settles, so candle
+expiry is keyed to the market clock rather than a rolling window
+(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
+`market.shouldRefresh`):
+
+- Equities/ETFs expire at **16:55 ET** on the next trading day.
+- Mutual funds (NAV) expire at **03:25 ET** the next morning.
+
+If a refresh fires but the provider has not posted the just-closed bar
+yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
+overdue, the code concludes the session was an un-modeled closure (Good
+Friday, weather) and falls back to the next normal boundary instead of
+thrashing all day. See the user page for the cron-timing rationale.
+
+## The fetch decision
+
+### Tiers (the big picture)
+
+```mermaid
+flowchart TD
+    A["Data request via DataService"] --> B{"Local cache fresh?"}
+    B -->|yes| C["Deserialize and return, no network"]
+    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
+    D -->|yes| E["GET server, write bytes verbatim"]
+    E --> F{"Synced entry fresh?"}
+    F -->|yes| C
+    F -->|no| G["Provider fetch"]
+    D -->|no| G
+    G --> H{"Result?"}
+    H -->|ok| I["Write cache with new expiry, return"]
+    H -->|NotFound| J["Write negative cache, return FetchFailed"]
+    H -->|transient| K["Return error, retry next run"]
+```
+
+The `--refresh-data` policy maps to `FetchOptions`:
+
+- `auto` (default): all tiers, honor TTL.
+- `force` -> `force_refresh = true`: skip the local-cache and server
+  tiers, go straight to the provider, re-stamp the cache. Bypasses
+  negative entries (so it retries dead lookups).
+- `never` -> `skip_network = true`: stop at the local cache; return
+  stale data if present, never touch the network.
+
+### `getCandles` (single symbol)
+
+This is the most involved path because of the daily/meta split, the
+incremental-update logic, and the TwelveData carve-out.
+
+```mermaid
+flowchart TD
+    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
+    NG -->|yes| FF["return FetchFailed, no network"]
+    NG -->|no| RM{"candles_meta exists?"}
+
+    RM -->|yes| TW{"provider is twelvedata?"}
+    TW -->|yes| FULL
+    TW -->|no| FR{"meta fresh and not force_refresh?"}
+    FR -->|yes| RET["return cached candles"]
+    FR -->|no| SS1["syncCandlesFromServer"]
+    SS1 --> SF1{"fresh now?"}
+    SF1 -->|yes| RET
+    SF1 -->|no| INC{"shouldRefresh?"}
+    INC -->|no| BUMP["bump TTL, return cached"]
+    INC -->|yes| INCF["incremental fetch from last_date+1"]
+
+    RM -->|no| SN{"skip_network?"}
+    SN -->|yes| FF
+    SN -->|no| SS2["syncCandlesFromServer"]
+    SS2 --> SF2{"fresh now?"}
+    SF2 -->|yes| RET
+    SF2 -->|no| FULL["populateAllFromTiingo, full history"]
+    FULL --> RES{"result?"}
+    RES -->|ok| RET2["return fetched"]
+    RES -->|NotFound| WN["writeNegative candles_daily"]
+    WN --> FF
+    RES -->|transient| TR["bump fail_count, TransientError"]
+    RES -->|other| FF
+```
+
+Key invariant: the negative marker for a candle-less symbol lives in
+`candles_daily.srf`, and **every** candle decision honors it there -
+`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
+cache-only display path), and the `getCandles` short-circuit above. This
+matters because `candles_meta.srf` is never created for a symbol that
+has no candles, so anything keying freshness off the meta file alone
+would treat such a symbol as perpetually stale and re-fetch it forever.
+
+### `fetchCached` (dividends, splits, options, earnings, ...)
+
+Everything that is not candles flows through the generic `fetchCached`,
+which is simpler because each type is a single file:
+
+1. `.fresh_only` read; a fresh entry (including a negative one) returns
+   immediately.
+2. `skip_network`: return any cached entry, even stale; else
+   `FetchFailed`.
+3. Server sync (if configured); a fresh synced entry returns.
+4. Provider fetch; on success write with the type's TTL; on
+   `NotFound` write a negative entry; on transient error return
+   `FetchFailed` without poisoning the cache.
+
+### `loadAllPrices` (portfolio + watchlist price load)
+
+The portfolio price load batches all symbols through three phases.
+Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
+fallback that calls `getCandles`.
+
+```mermaid
+flowchart TD
+    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
+    P1 --> CF{"cache fresh and not force_refresh?"}
+    CF -->|yes| HIT["use cached last close, cached_count++"]
+    CF -->|no| ADD["add to needs_fetch"]
+    HIT --> CHK
+    ADD --> CHK{"needs_fetch empty?"}
+    CHK -->|yes| DONE["return"]
+    CHK -->|no| OFF{"skip_network?"}
+    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
+    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
+    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
+    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
+    P2 --> REM["unsynced to server_failures"]
+    ALLF --> P3
+    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
+    P3 --> END["return prices + counts"]
+```
+
+## Negative caching
+
+When a provider says a symbol genuinely has no data of a type - an
+`error.NotFound` - zfin writes a **negative cache entry** so it does not
+re-run the dead lookup on every invocation. The entry is the sentinel:
+
+```
+#!srfv1
+# fetch_failed
+```
+
+(`Store.negative_cache_content`). Rules:
+
+- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
+  write. Rate-limit, 5xx, connection, auth, and parse failures are
+  transient - they fail the call but leave the cache untouched so the
+  next run retries. (Auth/parse looking permanent but being transient is
+  exactly why they must not poison a now-sticky negative cache.)
+- **Negative entries are always fresh.** They have no `#!expires=`;
+  `readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
+  as fresh, so they stick until `--refresh-data=force` or `cache clear`.
+- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
+  writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
+  `getCandles` short-circuit all recognize it there. `candles_meta.srf`
+  is intentionally not created for a no-data symbol.
+
+## Candle-less symbols (crypto and friends)
+
+Some held symbols have **no daily candles available from the candle
+provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` /
+`BTC-USD` shape are the common case, and delisted or invalid tickers
+behave identically. For these symbols `getCandles` writes a negative
+entry and never produces a price from history.
+
+Such symbols are still priced, through two mechanisms that do **not**
+touch the candle cache:
+
+- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
+  intraday price from Yahoo, which *does* serve crypto. The TUI overlays
+  these live quotes on top of the candle-close price map on refresh and
+  on every streaming tick, so a candle-less holding shows its real
+  current price there. Live quotes are never cached.
+- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
+  value. When neither a candle close nor a live quote is available,
+  `buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
+  position's average cost and flags it as a manual/estimated price
+  (rendered in a warning color).
+
+Practical consequence by surface: the plain CLI `portfolio` command does
+not apply the live-quote overlay, so a candle-less holding shows its
+average-cost fallback (break-even, warning color); the TUI shows the
+live Yahoo price. This is expected - historical-candle commands (`perf`,
+charts) simply have no data for these symbols, while quote-driven views
+do.
+
+If you want a candle-less symbol to be re-checked against the provider
+(for example a ticker that has since started trading), clear its
+negative entry with `cache clear` or `--refresh-data=force`; the live
+quote and manual-price paths are unaffected by the negative cache.
+
+## Server sync (the optional L2 tier)
+
+`ZFIN_SERVER` points zfin at a
+[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
+shared cache between your local cache and the upstream providers. When
+unset, every server-sync path silently no-ops.
+
+Client side (`syncFromServer`, `syncCandlesFromServer`,
+`parallelServerSync` in `service.zig`):
+
+- Triggered on a local miss/stale entry, before any provider call
+  (skipped under `force_refresh`).
+- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
+  (sha256 ETag, completeness check) and written to the local cache
+  **verbatim** via `writeRaw` - the client does not re-stamp
+  `#!expires=`, so it inherits the server's freshness boundary.
+- `parallelServerSync` fans out one task per symbol for the portfolio
+  price load (each worker uses its own HTTP client; the allocator is
+  thread-safe).
+
+Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):
+
+- A **present** file is served as-is, even if stale - the server's cron
+  is the freshness authority; reads never trigger a refetch.
+- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
+  same `getCandles` / `fetchCached` code through the shared zfin
+  library), then re-reads; if still absent, it returns 404.
+
+Because the server runs the same library, the negative-cache rules above
+apply there too: a candle-less symbol gets a negative `candles_daily.srf`
+on first miss and is served from it thereafter, rather than re-hitting
+the upstream provider on every request.
+
+## Invalidation and atomicity
+
+- `DataService.invalidate(symbol)` clears a symbol's entries; for
+  candles it removes the `candles_daily` + `candles_meta` pair together.
+- `cache clear` wipes the whole cache directory; everything re-fetches
+  next run.
+- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
+  and renames into place, so a reader never sees a torn file. A
+  defensively detected torn candle file self-heals by wiping the pair.
+
+## Key code references
+
+| Concern                         | Location                                            |
+|---------------------------------|-----------------------------------------------------|
+| Data-access entry point         | `DataService` - `src/service.zig`                   |
+| Per-type generic fetch          | `fetchCached` - `src/service.zig`                   |
+| Candle fetch + incremental      | `getCandles` - `src/service.zig`                    |
+| Batch price load                | `loadAllPrices` - `src/service.zig`                 |
+| Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`    |
+| Cache store, read/write         | `Store` - `src/cache/store.zig`                     |
+| Freshness check                 | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig`  |
+| TTLs and expiry computation     | `Ttl`, `computeExpires` - `src/cache/store.zig`     |
+| Negative cache                  | `writeNegative`, `isNegative` - `src/cache/store.zig` |
+| NotFound classification         | `isPermanentProviderFailure` - `src/service.zig`    |
+| Market-aware candle expiry      | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
+| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
+| Server endpoints                | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
+
+For the user-facing summary and the `--refresh-data` walkthrough, see
+[Caching and data freshness](../explanation/caching.md).
--- a/docs/explanation/caching.md
+++ b/docs/explanation/caching.md
@ -21,9 +21,11 @@ can satisfy it:
 3. **Provider.** Otherwise zfin fetches from the upstream provider,
   writes the result to the cache, and returns it.

-Freshness is decided by the cache file's modification time versus the
-TTL for that data type. The cache directory defaults to `~/.cache/zfin`
-and is set with `ZFIN_CACHE_DIR`.
+Freshness is decided by an expiry timestamp written into each cache
+file when it is fetched (an `#!expires=` line), compared against the
+current time -- not the file's modification time. Each data type sets
+its own TTL, which determines that expiry (see below). The cache
+directory defaults to `~/.cache/zfin` and is set with `ZFIN_CACHE_DIR`.

 The `--refresh-data` policy decides which tiers run:

@ -120,6 +122,32 @@ ticker, say -- zfin records a negative cache entry so it doesn't retry
 the same dead lookup on every run. (Transient failures like rate limits
 are not cached this way; they're retried.)

+## Symbols without candle data (crypto)
+
+A few holdings have no daily price history available from zfin's candle
+provider -- cryptocurrencies (held as `DOGE-USD`, `BTC-USD`, and the
+like) are the common case, and long-delisted tickers behave the same
+way. zfin negative-caches these so it stops asking for candles that will
+never arrive.
+
+Such holdings are still valued, just not from price history:
+
+- **In the TUI** they are priced from **live quotes** (Yahoo serves
+  crypto), refreshed on load and on every streaming tick.
+- A **manual price** wins when set: a `price::` field on the lot in
+  `portfolio.srf` pins the value.
+- Otherwise zfin falls back to your **average cost** for the holding and
+  flags it in a warning color, so a candle-less position shows at
+  break-even rather than disappearing.
+
+Because the plain `portfolio` CLI command doesn't fetch live quotes, a
+candle-less holding shows its average-cost fallback there; open the TUI
+(or set a `price::`) to see a live value. History-based commands like
+[`perf`](../reference/cli/perf.md) and the price charts simply have no
+data for these symbols. To force a re-check (say a ticker started
+trading), clear the entry with `--refresh-data=force` or `zfin cache
+clear`.
+
 ## Rate limiting

 Each provider has a client-side token-bucket limiter sized to its
@ -164,3 +192,7 @@ do:
  `zfin cache clear` wipes it (everything re-fetches next run).

 See [Offline use and refreshing data](../guides/offline-and-refresh.md).
+
+For the low-level implementation -- on-disk layout, the freshness model,
+negative caching, and the fetch-decision flowcharts -- see
+[Cache implementation](../dev/caching-implementation.md).