zfin/docs/dev/caching-implementation.md

Cache implementation (developer reference)

This is the low-level, contributor-facing companion to the user-facing Caching and data freshness page. It documents how the cache is built: the on-disk layout, the freshness model, the fetch-decision flow, negative caching, and the optional server (L2) tier. If you are changing anything in src/cache/store.zig or the fetch paths in src/service.zig, read this first.

Diagrams use Mermaid, which Forgejo renders natively.

Where the data lives

DataService (src/service.zig) is the sole data-access layer. Both the CLI and the TUI go through it; nothing else calls a provider directly. It reads and writes a per-symbol, per-type SRF file cache via Store (src/cache/store.zig):

{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
  AAPL/
    candles_daily.srf             OHLCV bars (appended; replaced on restatement)
    candles_meta.srf              last_close, last_date, provider, adj_basis + freshness
    dividends.srf
    splits.srf
    options.srf
    earnings.srf
    classification.srf
    etf_metrics.srf
  _edgar/
    tickers_companies.srf         shared EDGAR ticker -> CIK maps
    tickers_funds.srf
  0000320193/
    entity_facts.srf              per-CIK XBRL facts

SRF is a line-oriented key-value format. Files carry #!-prefixed directives (#!expires=, #!created=) ahead of their records.

The candle two-file split

Candles are stored as two files, and the split is load-bearing:

• candles_daily.srf holds the actual OHLCV records and normally grows by appending: on a cache miss only bars newer than last_date are fetched and appended, not the full history. The exception is a restatement, which replaces the file wholesale - see The adjustment basis.
• candles_meta.srf holds a single small record (last_close, last_date, provider, fail_count, tiingo_retry_after_s, adj_basis) plus the #!expires= and #!created= directives.

Keeping the metadata separate lets every freshness check and last-price read touch a ~100-byte file instead of deserializing a multi-megabyte candle history. The price fast-path in loadAllPrices never deserializes candles_daily.srf - at most it peeks at the first bytes to detect a negative entry.

The two files are a unit: DataService.invalidate and the torn-file self-heal clear both together, and a negative cache entry for a candle-less symbol is keyed off candles_daily.srf (see Negative caching).

The adjustment basis

Appending bars cannot restate the bars already on disk, and sometimes they need it.

Providers compute adj_close by scaling raw close by the product of the adjustment factors for every distribution after that bar. So a series' adjustment basis is only as current as the fetch that produced it. Newly appended bars arrive with adj_close == close, because nothing has gone ex after them yet, while every previously cached bar keeps the basis it was originally fetched with. When the next distribution goes ex, the bars behind it should be marked down by its factor - and an append does not do that. Every total return spanning that ex-date then reads low by roughly the missed yield.

CandleMeta.adj_basis records how current a series' basis is. It is the date of the newest bar present at the last full fetch:

• Not a wall-clock timestamp. A fetch that runs before the day's bar is published gets a basis one bar behind - which is the honest claim, since a provider's adjusted series is only ever as complete as its newest bar.
• Not an ex-date. It is compared against ex-dates but is never one.
• Never advanced by an append. Store.appendCandles takes the existing meta and overrides only last_close / last_date; only Store.cacheCandles, which replaces the whole file, sets a new basis. That asymmetry is the mechanism, so keep it.

getCandles escalates from append to full refetch when freshness.adjustmentBasisStale says the basis predates a corporate action that has already gone ex:

newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale     = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)

The already-ex bound lives in newestCorporateAction, not in the verdict, and that placement is load-bearing in both directions. A declared-but-not-yet-ex distribution is reflected in no provider's adjusted series, so treating it as something to catch up to would refetch on every pass forever. But bounding the verdict instead - "is the newest action of all still in the future? then nothing to do" - lets one forward announcement hide every older unapplied action behind it. That shipped: a quarterly payer announcing a quarter ahead was permanently unable to restate.

The check runs on the stale path only, not on the fresh-cache early-return. Detection is therefore at most one trading day behind, which the candle TTL guarantees, and the hot portfolio-pricing path pays nothing.

Steady state for a quarterly payer is four full-history fetches a year, each landing within about a trading day of an ex-date. zfin diagnose SYMBOL reports the basis against the newest already-ex action.

Freshness is the #!expires= directive, not mtime

A cache entry is fresh when the wall clock is earlier than the #!expires= epoch-seconds directive embedded in the file. File modification time is not consulted for freshness anywhere - mtime is a fragile signal (it changes on copy, restore, rsync, and filesystem quirks), so the expiry is written into the content itself.

• On write, computeExpires (store.zig) sets #!expires = now + TTL for the data type, optionally offset by a per-key deterministic jitter to avoid thundering-herd refreshes.
• On read, the SRF iterator parses #!expires= and isFresh compares it to Timestamp.now(io, .real).
• An entry with no #!expires= is treated as stale by zfin's .fresh_only reads (a deliberate override of SRF's "no expiry = always fresh" default), except for negative entries, which are always fresh.

TTLs by data type

Base TTLs live in Ttl (store.zig); jitter is applied per call site in DataType.ttl().

Data type	TTL	Jitter	Notes
Daily candles	market-aware	n/a	Boundary set by market.nextCandleExpiry (below)
Historical candles	never (-1)	n/a	Bars older than ~1 day are immutable
Dividends	14 days	11%	Declared well in advance
Splits	14 days	11%	Rare corporate events
Options	1 hour	0	Move continuously during market hours
Earnings	30 days	8%	Smart-refresh after an announcement date passes
Classification	90 days	8%	Sector/industry/country from Wikidata
ETF metrics	90 days	8%	NPORT-P profile, quarterly cadence
Entity facts (XBRL)	30 days	8%	Per-CIK, quarterly filing cadence
EDGAR ticker maps	30 days	8%	ticker -> CIK; very stable upstream
Quotes	never cached	n/a	Live by definition (see below)

Market-aware candle freshness

Daily bars are only meaningful once the session settles, so candle expiry is keyed to the market clock rather than a rolling window (market.nextCandleExpiry / market.staleCandleExpiry, market.shouldRefresh):

• Equities/ETFs expire at 16:55 ET on the next trading day.
• Mutual funds (NAV) expire at 03:25 ET the next morning.

If a refresh fires but the provider has not posted the just-closed bar yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes overdue, the code concludes the session was an un-modeled closure (Good Friday, weather) and falls back to the next normal boundary instead of thrashing all day. See the user page for the cron-timing rationale.

The fetch decision

Tiers (the big picture)

flowchart TD
    A["Data request via DataService"] --> B{"Local cache fresh?"}
    B -->|yes| C["Deserialize and return, no network"]
    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
    D -->|yes| E["GET server, write bytes verbatim"]
    E --> F{"Synced entry fresh?"}
    F -->|yes| C
    F -->|no| G["Provider fetch"]
    D -->|no| G
    G --> H{"Result?"}
    H -->|ok| I["Write cache with new expiry, return"]
    H -->|NotFound| J["Write negative cache, return FetchFailed"]
    H -->|transient| K["Return error, retry next run"]

The --refresh-data policy maps to FetchOptions:

• auto (default): all tiers, honor TTL.
• force -> force_refresh = true: skip the local-cache and server tiers, go straight to the provider, re-stamp the cache. Bypasses negative entries (so it retries dead lookups).
• never -> skip_network = true: stop at the local cache; return stale data if present, never touch the network.

getCandles (single symbol)

This is the most involved path because of the daily/meta split, the incremental-update logic, the adjustment-basis escalation, and the TwelveData carve-out.

Note also that force re-asks the provider but does not rebuild candle history: it skips the TTL and the server tier, then takes the same incremental top-up. Replacing the series is the restatement path's job (or zfin cache clear).

Provider routing is keyed off CandleMeta.tiingo_retry_after_s, not off provider. provider is pure provenance - "where did these bars come from" - and using it to route made a single non-transient Tiingo failure permanent: Yahoo got tried first, succeeded, rewrote provider = .yahoo, and Tiingo was never consulted again. The backoff is armed only by a genuine 404, expires after Ttl.tiingo_backoff with per-symbol jitter, and clears the moment Tiingo serves the symbol again.

flowchart TD
    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
    NG -->|yes| FF["return FetchFailed, no network"]
    NG -->|no| RM{"candles_meta exists?"}

    RM -->|yes| SK{"skip_network?"}
    SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
    SK -->|no| TW{"provider is twelvedata?"}
    TW -->|yes| FULL
    TW -->|no| FR{"meta fresh and not force_refresh?"}
    FR -->|yes| RET["return cached candles"]
    FR -->|no| SS1["syncCandlesFromServer"]
    SS1 --> SF1{"fresh AND adj_basis current?"}
    SF1 -->|yes| RET
    SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
    AB -->|yes| REST["refetchFullHistory: restate whole series"]
    REST --> RR{"ok?"}
    RR -->|yes| RET2["return fetched"]
    RR -->|no| INC
    AB -->|no| INC{"shouldRefresh?"}
    INC -->|no| BUMP["bump TTL, return cached"]
    INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]

    RM -->|no| SN{"skip_network?"}
    SN -->|yes| FF
    SN -->|no| SS2["syncCandlesFromServer"]
    SS2 --> SF2{"fresh now?"}
    SF2 -->|yes| RET
    SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
    FULL --> RES{"result?"}
    RES -->|ok| RET2
    RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"]
    WN --> FF
    RES -->|transient| TR["bump fail_count, TransientError"]
    RES -->|other| FF

Two things about this shape are easy to get wrong.

The basis check precedes shouldRefresh. A symbol that needs both a top-up and a restatement costs one full fetch, not an append followed by a second pass. And a failed restatement falls through to the ordinary top-up rather than erroring: the existing series is untouched and still usable, just understated by the missed adjustment, so it retries on the next stale pass.

Only a unanimous NotFound earns a negative entry. writeNegative overwrites candles_daily.srf with a marker, so a verdict of "no such symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and refetchFullHistory returns error.NotFound only when every provider disclaims the symbol. Anything else (auth trouble, a malformed body, a network blip) fails the call but leaves the cache alone. The restatement path above never writes a negative entry at all, for the same reason: it is reached while holding a working series.

The no-prior-cache branch does not re-check the basis after a server sync, only freshness. A stale basis inherited from the server is caught on the next invocation, which takes the meta-exists branch. One invocation of understated returns, then it self-corrects.

Key invariant: the negative marker for a candle-less symbol lives in candles_daily.srf, and every candle decision honors it there - isCandleMetaFresh (the price fast-path gate), getCachedCandles (the cache-only display path), and the getCandles short-circuit above. This matters because candles_meta.srf is never created for a symbol that has no candles, so anything keying freshness off the meta file alone would treat such a symbol as perpetually stale and re-fetch it forever.

fetchCached (dividends, splits, options, earnings, ...)

Everything that is not candles flows through the generic fetchCached, which is simpler because each type is a single file:

1. .fresh_only read; a fresh entry (including a negative one) returns immediately.
2. skip_network: return any cached entry, even stale; else FetchFailed.
3. Server sync (if configured); a fresh synced entry returns.
4. Provider fetch; on success write with the type's TTL; on NotFound write a negative entry; on transient error return FetchFailed without poisoning the cache.

loadAllPrices (portfolio + watchlist price load)

The portfolio price load batches all symbols through three phases. Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider fallback that calls getCandles.

flowchart TD
    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
    P1 --> CF{"cache fresh and not force_refresh?"}
    CF -->|yes| HIT["use cached last close, cached_count++"]
    CF -->|no| ADD["add to needs_fetch"]
    HIT --> CHK
    ADD --> CHK{"needs_fetch empty?"}
    CHK -->|yes| DONE["return"]
    CHK -->|no| OFF{"skip_network?"}
    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
    P2 --> REM["unsynced to server_failures"]
    ALLF --> P3
    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
    P3 --> END["return prices + counts"]

Negative caching

When a provider says a symbol genuinely has no data of a type - an error.NotFound - zfin writes a negative cache entry so it does not re-run the dead lookup on every invocation. The entry is the sentinel:

#!srfv1
# fetch_failed

(Store.negative_cache_content). Rules:

• Only NotFound qualifies. isPermanentProviderFailure gates the write. Rate-limit, 5xx, connection, auth, and parse failures are transient - they fail the call but leave the cache untouched so the next run retries. (Auth/parse looking permanent but being transient is exactly why they must not poison a now-sticky negative cache.)
• Negative entries are always fresh. They have no #!expires=; readSlice, read, and isCandleMetaFresh special-case the sentinel as fresh, so they stick until --refresh-data=force or cache clear.
• Candles key the negative off candles_daily.srf. writeNegative writes that file; isCandleMetaFresh, getCachedCandles, and the getCandles short-circuit all recognize it there. candles_meta.srf is intentionally not created for a no-data symbol.

Candle-less symbols (crypto and friends)

Some held symbols have no daily candles available from any candle provider - cryptocurrencies on the Yahoo DOGE-USD / BTC-USD shape are the common case, and delisted or invalid tickers behave identically. For these symbols getCandles writes a negative entry and never produces a price from history. "Any" is literal: Tiingo and Yahoo must both disclaim the symbol, because the marker overwrites candles_daily.srf.

Such symbols are still priced, through two mechanisms that do not touch the candle cache:

• Live quotes (Yahoo). loadLiveQuotes / getQuote fetch an intraday price from Yahoo, which does serve crypto. The TUI overlays these live quotes on top of the candle-close price map on refresh and on every streaming tick, so a candle-less holding shows its real current price there. Live quotes are never cached.
• Manual price. A price:: field on a lot in portfolio.srf pins a value. When neither a candle close nor a live quote is available, buildFallbackPrices (analytics/valuation.zig) falls back to the position's average cost and flags it as a manual/estimated price (rendered in a warning color).

Practical consequence by surface: the plain CLI portfolio command does not apply the live-quote overlay, so a candle-less holding shows its average-cost fallback (break-even, warning color); the TUI shows the live Yahoo price. This is expected - historical-candle commands (perf, charts) simply have no data for these symbols, while quote-driven views do.

If you want a candle-less symbol to be re-checked against the provider (for example a ticker that has since started trading), clear its negative entry with cache clear or --refresh-data=force; the live quote and manual-price paths are unaffected by the negative cache.

Server sync (the optional L2 tier)

ZFIN_SERVER points zfin at a zfin-server instance - a shared cache between your local cache and the upstream providers. When unset, every server-sync path silently no-ops.

Client side (syncFromServer, syncCandlesFromServer, parallelServerSync in service.zig):

• Triggered on a local miss/stale entry, before any provider call (skipped under force_refresh).
• GET {ZFIN_SERVER}/{SYMBOL}/{type}; the response body is validated (sha256 ETag, completeness check) and written to the local cache verbatim via writeRaw - the client does not re-stamp #!expires=, so it inherits the server's freshness boundary.
• parallelServerSync fans out one task per symbol for the portfolio price load (each worker uses its own HTTP client; the allocator is thread-safe).

Server side (serveSrfFile, fetchOnMiss in zfin-server):

• A present file is served as-is, even if stale - the server's cron is the freshness authority; reads never trigger a refetch.
• An absent file triggers a one-shot fetchOnMiss (which calls the same getCandles / fetchCached code through the shared zfin library), then re-reads; if still absent, it returns 404.

Because the server runs the same library, the negative-cache rules above apply there too: a candle-less symbol gets a negative candles_daily.srf on first miss and is served from it thereafter, rather than re-hitting the upstream provider on every request.

Invalidation and atomicity

• DataService.invalidate(symbol) clears a symbol's entries; for candles it removes the candles_daily + candles_meta pair together.
• cache clear wipes the whole cache directory; everything re-fetches next run.
• All writes are crash-safe: atomic.zig writes to a temp file, fsyncs, and renames into place, so a reader never sees a torn file. A defensively detected torn candle file self-heals by wiping the pair.

Key code references

Concern	Location
Data-access entry point	DataService - src/service.zig
Per-type generic fetch	fetchCached - src/service.zig
Candle fetch + incremental	getCandles - src/service.zig
Batch price load	loadAllPrices - src/service.zig
Live quotes (uncached)	loadLiveQuotes, getQuote - src/service.zig
Cache store, read/write	Store - src/cache/store.zig
Freshness check	isFresh (SRF), isCandleMetaFresh - store.zig
TTLs and expiry computation	Ttl, computeExpires - src/cache/store.zig
Negative cache	writeNegative, isNegative - src/cache/store.zig
NotFound classification	isPermanentProviderFailure - src/service.zig
Market-aware candle expiry	nextCandleExpiry, shouldRefresh - src/market.zig
Price fallback (manual/avg-cost)	buildFallbackPrices - src/analytics/valuation.zig
Server endpoints	serveSrfFile, fetchOnMiss - zfin-server/src/main.zig

For the user-facing summary and the --refresh-data walkthrough, see Caching and data freshness.