# Cache implementation (developer reference) This is the low-level, contributor-facing companion to the user-facing [Caching and data freshness](../explanation/caching.md) page. It documents *how* the cache is built: the on-disk layout, the freshness model, the fetch-decision flow, negative caching, and the optional server (L2) tier. If you are changing anything in `src/cache/store.zig` or the fetch paths in `src/service.zig`, read this first. Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders natively. ## Where the data lives `DataService` (`src/service.zig`) is the sole data-access layer. Both the CLI and the TUI go through it; nothing else calls a provider directly. It reads and writes a per-symbol, per-type SRF file cache via `Store` (`src/cache/store.zig`): ``` {cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR AAPL/ candles_daily.srf OHLCV bars (append-only) candles_meta.srf last_close, last_date, provider + freshness dividends.srf splits.srf options.srf earnings.srf classification.srf etf_metrics.srf _edgar/ tickers_companies.srf shared EDGAR ticker -> CIK maps tickers_funds.srf 0000320193/ entity_facts.srf per-CIK XBRL facts ``` [SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value format. Files carry `#!`-prefixed directives (`#!expires=`, `#!created=`) ahead of their records. ### The candle two-file split Candles are stored as **two** files, and the split is load-bearing: - `candles_daily.srf` holds the actual OHLCV records and grows append-only: on a cache miss only bars newer than `last_date` are fetched and appended, never the full history. - `candles_meta.srf` holds a single small record (`last_close`, `last_date`, `provider`, `fail_count`) plus the `#!expires=` and `#!created=` directives. Keeping the metadata separate lets every freshness check and last-price read touch a ~100-byte file instead of deserializing a multi-megabyte candle history. The price fast-path in `loadAllPrices` never deserializes `candles_daily.srf` - at most it peeks at the first bytes to detect a negative entry. The two files are a unit: `DataService.invalidate` and the torn-file self-heal clear both together, and a negative cache entry for a candle-less symbol is keyed off `candles_daily.srf` (see [Negative caching](#negative-caching)). ## Freshness is the `#!expires=` directive, not mtime A cache entry is fresh when the wall clock is earlier than the `#!expires=` epoch-seconds directive embedded in the file. File modification time is **not** consulted for freshness anywhere - mtime is a fragile signal (it changes on copy, restore, rsync, and filesystem quirks), so the expiry is written into the content itself. - On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL` for the data type, optionally offset by a per-key deterministic jitter to avoid thundering-herd refreshes. - On read, the SRF iterator parses `#!expires=` and `isFresh` compares it to `Timestamp.now(io, .real)`. - An entry with **no** `#!expires=` is treated as stale by zfin's `.fresh_only` reads (a deliberate override of SRF's "no expiry = always fresh" default), except for negative entries, which are always fresh. ### TTLs by data type Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site in `DataType.ttl()`. | Data type | TTL | Jitter | Notes | |----------------------|------------------|--------|--------------------------------------------------| | Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)| | Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable | | Dividends | 14 days | 11% | Declared well in advance | | Splits | 14 days | 11% | Rare corporate events | | Options | 1 hour | 0 | Move continuously during market hours | | Earnings | 30 days | 8% | Smart-refresh after an announcement date passes | | Classification | 90 days | 8% | Sector/industry/country from Wikidata | | ETF metrics | 90 days | 8% | NPORT-P profile, quarterly cadence | | Entity facts (XBRL) | 30 days | 8% | Per-CIK, quarterly filing cadence | | EDGAR ticker maps | 30 days | 8% | ticker -> CIK; very stable upstream | | Quotes | never cached | n/a | Live by definition (see below) | ### Market-aware candle freshness Daily bars are only meaningful once the session settles, so candle expiry is keyed to the market clock rather than a rolling window (`market.nextCandleExpiry` / `market.staleCandleExpiry`, `market.shouldRefresh`): - Equities/ETFs expire at **16:55 ET** on the next trading day. - Mutual funds (NAV) expire at **03:25 ET** the next morning. If a refresh fires but the provider has not posted the just-closed bar yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes overdue, the code concludes the session was an un-modeled closure (Good Friday, weather) and falls back to the next normal boundary instead of thrashing all day. See the user page for the cron-timing rationale. ## The fetch decision ### Tiers (the big picture) ```mermaid flowchart TD A["Data request via DataService"] --> B{"Local cache fresh?"} B -->|yes| C["Deserialize and return, no network"] B -->|no| D{"ZFIN_SERVER set and not force_refresh?"} D -->|yes| E["GET server, write bytes verbatim"] E --> F{"Synced entry fresh?"} F -->|yes| C F -->|no| G["Provider fetch"] D -->|no| G G --> H{"Result?"} H -->|ok| I["Write cache with new expiry, return"] H -->|NotFound| J["Write negative cache, return FetchFailed"] H -->|transient| K["Return error, retry next run"] ``` The `--refresh-data` policy maps to `FetchOptions`: - `auto` (default): all tiers, honor TTL. - `force` -> `force_refresh = true`: skip the local-cache and server tiers, go straight to the provider, re-stamp the cache. Bypasses negative entries (so it retries dead lookups). - `never` -> `skip_network = true`: stop at the local cache; return stale data if present, never touch the network. ### `getCandles` (single symbol) This is the most involved path because of the daily/meta split, the incremental-update logic, and the TwelveData carve-out. ```mermaid flowchart TD S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"} NG -->|yes| FF["return FetchFailed, no network"] NG -->|no| RM{"candles_meta exists?"} RM -->|yes| TW{"provider is twelvedata?"} TW -->|yes| FULL TW -->|no| FR{"meta fresh and not force_refresh?"} FR -->|yes| RET["return cached candles"] FR -->|no| SS1["syncCandlesFromServer"] SS1 --> SF1{"fresh now?"} SF1 -->|yes| RET SF1 -->|no| INC{"shouldRefresh?"} INC -->|no| BUMP["bump TTL, return cached"] INC -->|yes| INCF["incremental fetch from last_date+1"] RM -->|no| SN{"skip_network?"} SN -->|yes| FF SN -->|no| SS2["syncCandlesFromServer"] SS2 --> SF2{"fresh now?"} SF2 -->|yes| RET SF2 -->|no| FULL["populateAllFromTiingo, full history"] FULL --> RES{"result?"} RES -->|ok| RET2["return fetched"] RES -->|NotFound| WN["writeNegative candles_daily"] WN --> FF RES -->|transient| TR["bump fail_count, TransientError"] RES -->|other| FF ``` Key invariant: the negative marker for a candle-less symbol lives in `candles_daily.srf`, and **every** candle decision honors it there - `isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the cache-only display path), and the `getCandles` short-circuit above. This matters because `candles_meta.srf` is never created for a symbol that has no candles, so anything keying freshness off the meta file alone would treat such a symbol as perpetually stale and re-fetch it forever. ### `fetchCached` (dividends, splits, options, earnings, ...) Everything that is not candles flows through the generic `fetchCached`, which is simpler because each type is a single file: 1. `.fresh_only` read; a fresh entry (including a negative one) returns immediately. 2. `skip_network`: return any cached entry, even stale; else `FetchFailed`. 3. Server sync (if configured); a fresh synced entry returns. 4. Provider fetch; on success write with the type's TTL; on `NotFound` write a negative entry; on transient error return `FetchFailed` without poisoning the cache. ### `loadAllPrices` (portfolio + watchlist price load) The portfolio price load batches all symbols through three phases. Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider fallback that calls `getCandles`. ```mermaid flowchart TD ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"] P1 --> CF{"cache fresh and not force_refresh?"} CF -->|yes| HIT["use cached last close, cached_count++"] CF -->|no| ADD["add to needs_fetch"] HIT --> CHK ADD --> CHK{"needs_fetch empty?"} CHK -->|yes| DONE["return"] CHK -->|no| OFF{"skip_network?"} OFF -->|yes| STALE["stale-cache fallback or failed_count++"] OFF -->|no| HASSRV{"ZFIN_SERVER set?"} HASSRV -->|yes| P2["Phase 2: parallelServerSync"] HASSRV -->|no| ALLF["all needs_fetch to server_failures"] P2 --> REM["unsynced to server_failures"] ALLF --> P3 REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"] P3 --> END["return prices + counts"] ``` ## Negative caching When a provider says a symbol genuinely has no data of a type - an `error.NotFound` - zfin writes a **negative cache entry** so it does not re-run the dead lookup on every invocation. The entry is the sentinel: ``` #!srfv1 # fetch_failed ``` (`Store.negative_cache_content`). Rules: - **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the write. Rate-limit, 5xx, connection, auth, and parse failures are transient - they fail the call but leave the cache untouched so the next run retries. (Auth/parse looking permanent but being transient is exactly why they must not poison a now-sticky negative cache.) - **Negative entries are always fresh.** They have no `#!expires=`; `readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel as fresh, so they stick until `--refresh-data=force` or `cache clear`. - **Candles key the negative off `candles_daily.srf`.** `writeNegative` writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the `getCandles` short-circuit all recognize it there. `candles_meta.srf` is intentionally not created for a no-data symbol. ## Candle-less symbols (crypto and friends) Some held symbols have **no daily candles available from the candle provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` / `BTC-USD` shape are the common case, and delisted or invalid tickers behave identically. For these symbols `getCandles` writes a negative entry and never produces a price from history. Such symbols are still priced, through two mechanisms that do **not** touch the candle cache: - **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an intraday price from Yahoo, which *does* serve crypto. The TUI overlays these live quotes on top of the candle-close price map on refresh and on every streaming tick, so a candle-less holding shows its real current price there. Live quotes are never cached. - **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a value. When neither a candle close nor a live quote is available, `buildFallbackPrices` (`analytics/valuation.zig`) falls back to the position's average cost and flags it as a manual/estimated price (rendered in a warning color). Practical consequence by surface: the plain CLI `portfolio` command does not apply the live-quote overlay, so a candle-less holding shows its average-cost fallback (break-even, warning color); the TUI shows the live Yahoo price. This is expected - historical-candle commands (`perf`, charts) simply have no data for these symbols, while quote-driven views do. If you want a candle-less symbol to be re-checked against the provider (for example a ticker that has since started trading), clear its negative entry with `cache clear` or `--refresh-data=force`; the live quote and manual-price paths are unaffected by the negative cache. ## Server sync (the optional L2 tier) `ZFIN_SERVER` points zfin at a [zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a shared cache between your local cache and the upstream providers. When unset, every server-sync path silently no-ops. Client side (`syncFromServer`, `syncCandlesFromServer`, `parallelServerSync` in `service.zig`): - Triggered on a local miss/stale entry, before any provider call (skipped under `force_refresh`). - `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated (sha256 ETag, completeness check) and written to the local cache **verbatim** via `writeRaw` - the client does not re-stamp `#!expires=`, so it inherits the server's freshness boundary. - `parallelServerSync` fans out one task per symbol for the portfolio price load (each worker uses its own HTTP client; the allocator is thread-safe). Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`): - A **present** file is served as-is, even if stale - the server's cron is the freshness authority; reads never trigger a refetch. - An **absent** file triggers a one-shot `fetchOnMiss` (which calls the same `getCandles` / `fetchCached` code through the shared zfin library), then re-reads; if still absent, it returns 404. Because the server runs the same library, the negative-cache rules above apply there too: a candle-less symbol gets a negative `candles_daily.srf` on first miss and is served from it thereafter, rather than re-hitting the upstream provider on every request. ## Invalidation and atomicity - `DataService.invalidate(symbol)` clears a symbol's entries; for candles it removes the `candles_daily` + `candles_meta` pair together. - `cache clear` wipes the whole cache directory; everything re-fetches next run. - All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs, and renames into place, so a reader never sees a torn file. A defensively detected torn candle file self-heals by wiping the pair. ## Key code references | Concern | Location | |---------------------------------|-----------------------------------------------------| | Data-access entry point | `DataService` - `src/service.zig` | | Per-type generic fetch | `fetchCached` - `src/service.zig` | | Candle fetch + incremental | `getCandles` - `src/service.zig` | | Batch price load | `loadAllPrices` - `src/service.zig` | | Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` | | Cache store, read/write | `Store` - `src/cache/store.zig` | | Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` | | TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` | | Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` | | NotFound classification | `isPermanentProviderFailure` - `src/service.zig` | | Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` | | Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` | | Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` | For the user-facing summary and the `--refresh-data` walkthrough, see [Caching and data freshness](../explanation/caching.md).