zfin/docs/dev/caching-implementation.md

Cache implementation (developer reference)

This is the low-level, contributor-facing companion to the user-facing Caching and data freshness page. It documents how the cache is built: the on-disk layout, the freshness model, the fetch-decision flow, negative caching, and the optional server (L2) tier. If you are changing anything in src/cache/store.zig or the fetch paths in src/service.zig, read this first.

Diagrams use Mermaid, which Forgejo renders natively.

Where the data lives

DataService (src/service.zig) is the sole data-access layer. Both the CLI and the TUI go through it; nothing else calls a provider directly. It reads and writes a per-symbol, per-type SRF file cache via Store (src/cache/store.zig):

{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
  AAPL/
    candles_daily.srf             OHLCV bars (append-only)
    candles_meta.srf              last_close, last_date, provider + freshness
    dividends.srf
    splits.srf
    options.srf
    earnings.srf
    classification.srf
    etf_metrics.srf
  _edgar/
    tickers_companies.srf         shared EDGAR ticker -> CIK maps
    tickers_funds.srf
  0000320193/
    entity_facts.srf              per-CIK XBRL facts

SRF is a line-oriented key-value format. Files carry #!-prefixed directives (#!expires=, #!created=) ahead of their records.

The candle two-file split

Candles are stored as two files, and the split is load-bearing:

• candles_daily.srf holds the actual OHLCV records and grows append-only: on a cache miss only bars newer than last_date are fetched and appended, never the full history.
• candles_meta.srf holds a single small record (last_close, last_date, provider, fail_count) plus the #!expires= and #!created= directives.

Keeping the metadata separate lets every freshness check and last-price read touch a ~100-byte file instead of deserializing a multi-megabyte candle history. The price fast-path in loadAllPrices never deserializes candles_daily.srf - at most it peeks at the first bytes to detect a negative entry.

The two files are a unit: DataService.invalidate and the torn-file self-heal clear both together, and a negative cache entry for a candle-less symbol is keyed off candles_daily.srf (see Negative caching).

Freshness is the #!expires= directive, not mtime

A cache entry is fresh when the wall clock is earlier than the #!expires= epoch-seconds directive embedded in the file. File modification time is not consulted for freshness anywhere - mtime is a fragile signal (it changes on copy, restore, rsync, and filesystem quirks), so the expiry is written into the content itself.

• On write, computeExpires (store.zig) sets #!expires = now + TTL for the data type, optionally offset by a per-key deterministic jitter to avoid thundering-herd refreshes.
• On read, the SRF iterator parses #!expires= and isFresh compares it to Timestamp.now(io, .real).
• An entry with no #!expires= is treated as stale by zfin's .fresh_only reads (a deliberate override of SRF's "no expiry = always fresh" default), except for negative entries, which are always fresh.

TTLs by data type

Base TTLs live in Ttl (store.zig); jitter is applied per call site in DataType.ttl().

Data type	TTL	Jitter	Notes
Daily candles	market-aware	n/a	Boundary set by market.nextCandleExpiry (below)
Historical candles	never (-1)	n/a	Bars older than ~1 day are immutable
Dividends	14 days	11%	Declared well in advance
Splits	14 days	11%	Rare corporate events
Options	1 hour	0	Move continuously during market hours
Earnings	30 days	8%	Smart-refresh after an announcement date passes
Classification	90 days	8%	Sector/industry/country from Wikidata
ETF metrics	90 days	8%	NPORT-P profile, quarterly cadence
Entity facts (XBRL)	30 days	8%	Per-CIK, quarterly filing cadence
EDGAR ticker maps	30 days	8%	ticker -> CIK; very stable upstream
Quotes	never cached	n/a	Live by definition (see below)

Market-aware candle freshness

Daily bars are only meaningful once the session settles, so candle expiry is keyed to the market clock rather than a rolling window (market.nextCandleExpiry / market.staleCandleExpiry, market.shouldRefresh):

• Equities/ETFs expire at 16:55 ET on the next trading day.
• Mutual funds (NAV) expire at 03:25 ET the next morning.

If a refresh fires but the provider has not posted the just-closed bar yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes overdue, the code concludes the session was an un-modeled closure (Good Friday, weather) and falls back to the next normal boundary instead of thrashing all day. See the user page for the cron-timing rationale.

The fetch decision

Tiers (the big picture)

flowchart TD
    A["Data request via DataService"] --> B{"Local cache fresh?"}
    B -->|yes| C["Deserialize and return, no network"]
    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
    D -->|yes| E["GET server, write bytes verbatim"]
    E --> F{"Synced entry fresh?"}
    F -->|yes| C
    F -->|no| G["Provider fetch"]
    D -->|no| G
    G --> H{"Result?"}
    H -->|ok| I["Write cache with new expiry, return"]
    H -->|NotFound| J["Write negative cache, return FetchFailed"]
    H -->|transient| K["Return error, retry next run"]

The --refresh-data policy maps to FetchOptions:

• auto (default): all tiers, honor TTL.
• force -> force_refresh = true: skip the local-cache and server tiers, go straight to the provider, re-stamp the cache. Bypasses negative entries (so it retries dead lookups).
• never -> skip_network = true: stop at the local cache; return stale data if present, never touch the network.

getCandles (single symbol)

This is the most involved path because of the daily/meta split, the incremental-update logic, and the TwelveData carve-out.

flowchart TD
    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
    NG -->|yes| FF["return FetchFailed, no network"]
    NG -->|no| RM{"candles_meta exists?"}

    RM -->|yes| TW{"provider is twelvedata?"}
    TW -->|yes| FULL
    TW -->|no| FR{"meta fresh and not force_refresh?"}
    FR -->|yes| RET["return cached candles"]
    FR -->|no| SS1["syncCandlesFromServer"]
    SS1 --> SF1{"fresh now?"}
    SF1 -->|yes| RET
    SF1 -->|no| INC{"shouldRefresh?"}
    INC -->|no| BUMP["bump TTL, return cached"]
    INC -->|yes| INCF["incremental fetch from last_date+1"]

    RM -->|no| SN{"skip_network?"}
    SN -->|yes| FF
    SN -->|no| SS2["syncCandlesFromServer"]
    SS2 --> SF2{"fresh now?"}
    SF2 -->|yes| RET
    SF2 -->|no| FULL["populateAllFromTiingo, full history"]
    FULL --> RES{"result?"}
    RES -->|ok| RET2["return fetched"]
    RES -->|NotFound| WN["writeNegative candles_daily"]
    WN --> FF
    RES -->|transient| TR["bump fail_count, TransientError"]
    RES -->|other| FF

Key invariant: the negative marker for a candle-less symbol lives in candles_daily.srf, and every candle decision honors it there - isCandleMetaFresh (the price fast-path gate), getCachedCandles (the cache-only display path), and the getCandles short-circuit above. This matters because candles_meta.srf is never created for a symbol that has no candles, so anything keying freshness off the meta file alone would treat such a symbol as perpetually stale and re-fetch it forever.

fetchCached (dividends, splits, options, earnings, ...)

Everything that is not candles flows through the generic fetchCached, which is simpler because each type is a single file:

1. .fresh_only read; a fresh entry (including a negative one) returns immediately.
2. skip_network: return any cached entry, even stale; else FetchFailed.
3. Server sync (if configured); a fresh synced entry returns.
4. Provider fetch; on success write with the type's TTL; on NotFound write a negative entry; on transient error return FetchFailed without poisoning the cache.

loadAllPrices (portfolio + watchlist price load)

The portfolio price load batches all symbols through three phases. Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider fallback that calls getCandles.

flowchart TD
    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
    P1 --> CF{"cache fresh and not force_refresh?"}
    CF -->|yes| HIT["use cached last close, cached_count++"]
    CF -->|no| ADD["add to needs_fetch"]
    HIT --> CHK
    ADD --> CHK{"needs_fetch empty?"}
    CHK -->|yes| DONE["return"]
    CHK -->|no| OFF{"skip_network?"}
    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
    P2 --> REM["unsynced to server_failures"]
    ALLF --> P3
    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
    P3 --> END["return prices + counts"]

Negative caching

When a provider says a symbol genuinely has no data of a type - an error.NotFound - zfin writes a negative cache entry so it does not re-run the dead lookup on every invocation. The entry is the sentinel:

#!srfv1
# fetch_failed

(Store.negative_cache_content). Rules:

• Only NotFound qualifies. isPermanentProviderFailure gates the write. Rate-limit, 5xx, connection, auth, and parse failures are transient - they fail the call but leave the cache untouched so the next run retries. (Auth/parse looking permanent but being transient is exactly why they must not poison a now-sticky negative cache.)
• Negative entries are always fresh. They have no #!expires=; readSlice, read, and isCandleMetaFresh special-case the sentinel as fresh, so they stick until --refresh-data=force or cache clear.
• Candles key the negative off candles_daily.srf. writeNegative writes that file; isCandleMetaFresh, getCachedCandles, and the getCandles short-circuit all recognize it there. candles_meta.srf is intentionally not created for a no-data symbol.

Candle-less symbols (crypto and friends)

Some held symbols have no daily candles available from the candle provider (Tiingo) - cryptocurrencies on the Yahoo DOGE-USD / BTC-USD shape are the common case, and delisted or invalid tickers behave identically. For these symbols getCandles writes a negative entry and never produces a price from history.

Such symbols are still priced, through two mechanisms that do not touch the candle cache:

• Live quotes (Yahoo). loadLiveQuotes / getQuote fetch an intraday price from Yahoo, which does serve crypto. The TUI overlays these live quotes on top of the candle-close price map on refresh and on every streaming tick, so a candle-less holding shows its real current price there. Live quotes are never cached.
• Manual price. A price:: field on a lot in portfolio.srf pins a value. When neither a candle close nor a live quote is available, buildFallbackPrices (analytics/valuation.zig) falls back to the position's average cost and flags it as a manual/estimated price (rendered in a warning color).

Practical consequence by surface: the plain CLI portfolio command does not apply the live-quote overlay, so a candle-less holding shows its average-cost fallback (break-even, warning color); the TUI shows the live Yahoo price. This is expected - historical-candle commands (perf, charts) simply have no data for these symbols, while quote-driven views do.

If you want a candle-less symbol to be re-checked against the provider (for example a ticker that has since started trading), clear its negative entry with cache clear or --refresh-data=force; the live quote and manual-price paths are unaffected by the negative cache.

Server sync (the optional L2 tier)

ZFIN_SERVER points zfin at a zfin-server instance - a shared cache between your local cache and the upstream providers. When unset, every server-sync path silently no-ops.

Client side (syncFromServer, syncCandlesFromServer, parallelServerSync in service.zig):

• Triggered on a local miss/stale entry, before any provider call (skipped under force_refresh).
• GET {ZFIN_SERVER}/{SYMBOL}/{type}; the response body is validated (sha256 ETag, completeness check) and written to the local cache verbatim via writeRaw - the client does not re-stamp #!expires=, so it inherits the server's freshness boundary.
• parallelServerSync fans out one task per symbol for the portfolio price load (each worker uses its own HTTP client; the allocator is thread-safe).

Server side (serveSrfFile, fetchOnMiss in zfin-server):

• A present file is served as-is, even if stale - the server's cron is the freshness authority; reads never trigger a refetch.
• An absent file triggers a one-shot fetchOnMiss (which calls the same getCandles / fetchCached code through the shared zfin library), then re-reads; if still absent, it returns 404.

Because the server runs the same library, the negative-cache rules above apply there too: a candle-less symbol gets a negative candles_daily.srf on first miss and is served from it thereafter, rather than re-hitting the upstream provider on every request.

Invalidation and atomicity

• DataService.invalidate(symbol) clears a symbol's entries; for candles it removes the candles_daily + candles_meta pair together.
• cache clear wipes the whole cache directory; everything re-fetches next run.
• All writes are crash-safe: atomic.zig writes to a temp file, fsyncs, and renames into place, so a reader never sees a torn file. A defensively detected torn candle file self-heals by wiping the pair.

Key code references

Concern	Location
Data-access entry point	DataService - src/service.zig
Per-type generic fetch	fetchCached - src/service.zig
Candle fetch + incremental	getCandles - src/service.zig
Batch price load	loadAllPrices - src/service.zig
Live quotes (uncached)	loadLiveQuotes, getQuote - src/service.zig
Cache store, read/write	Store - src/cache/store.zig
Freshness check	isFresh (SRF), isCandleMetaFresh - store.zig
TTLs and expiry computation	Ttl, computeExpires - src/cache/store.zig
Negative cache	writeNegative, isNegative - src/cache/store.zig
NotFound classification	isPermanentProviderFailure - src/service.zig
Market-aware candle expiry	nextCandleExpiry, shouldRefresh - src/market.zig
Price fallback (manual/avg-cost)	buildFallbackPrices - src/analytics/valuation.zig
Server endpoints	serveSrfFile, fetchOnMiss - zfin-server/src/main.zig

For the user-facing summary and the --refresh-data walkthrough, see Caching and data freshness.