zfin/docs/dev/caching-implementation.md

Cache implementation (developer reference)

This is the low-level, contributor-facing companion to the user-facing Caching and data freshness page. It documents how the cache is built: the on-disk layout, the freshness model, the fetch-decision flow, negative caching, and the optional server (L2) tier. If you are changing anything in src/cache/store.zig or the fetch paths in src/service.zig, read this first.

Diagrams use Mermaid, which Forgejo renders natively.

Where the data lives

DataService (src/service.zig) is the sole data-access layer. Both the CLI and the TUI go through it; nothing else calls a provider directly. It reads and writes a per-symbol, per-type SRF file cache via Store (src/cache/store.zig):

{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
  AAPL/
    candles_daily.srf             OHLCV bars (appended; replaced on restatement)
    candles_meta.srf              last_close, last_date, provider, adj_basis + freshness
    dividends.srf
    splits.srf
    options.srf
    earnings.srf
    classification.srf
    etf_metrics.srf
  _edgar/
    tickers_companies.srf         shared EDGAR ticker -> CIK maps
    tickers_funds.srf
  0000320193/
    entity_facts.srf              per-CIK XBRL facts

SRF is a line-oriented key-value format. Files carry #!-prefixed directives (#!expires=, #!created=) ahead of their records.

The candle two-file split

Candles are stored as two files, and the split is load-bearing:

• candles_daily.srf holds the actual OHLCV records and normally grows by appending: on a cache miss only bars newer than last_date are fetched and appended, not the full history. The exception is a restatement, which replaces the file wholesale - see The adjustment basis.
• candles_meta.srf holds a single small record (last_close, last_date, provider, fail_count, tiingo_retry_after_s, adj_basis) plus the #!expires= and #!created= directives.

Keeping the metadata separate lets every freshness check and last-price read touch a ~100-byte file instead of deserializing a multi-megabyte candle history. The price fast-path in loadAllPrices never deserializes candles_daily.srf - at most it peeks at the first bytes to detect a negative entry.

The two files are a unit: DataService.invalidate and the torn-file self-heal clear both together, and a negative cache entry for a candle-less symbol is keyed off candles_daily.srf (see Negative caching).

The adjustment basis

Appending bars cannot restate the bars already on disk, and sometimes they need it.

Providers compute adj_close by scaling raw close by the product of the adjustment factors for every distribution after that bar. So a series' adjustment basis is only as current as the fetch that produced it. Newly appended bars arrive with adj_close == close, because nothing has gone ex after them yet, while every previously cached bar keeps the basis it was originally fetched with. When the next distribution goes ex, the bars behind it should be marked down by its factor - and an append does not do that. Every total return spanning that ex-date then reads low by roughly the missed yield.

CandleMeta.adj_basis records how current a series' basis is. It is the date of the newest bar present at the last full fetch:

• Not a wall-clock timestamp. A fetch that runs before the day's bar is published gets a basis one bar behind - which is the honest claim, since a provider's adjusted series is only ever as complete as its newest bar.
• Not an ex-date. It is compared against ex-dates but is never one.
• Never advanced by an append. Store.appendCandles takes the existing meta and overrides only last_close / last_date; only Store.cacheCandles, which replaces the whole file, sets a new basis. That asymmetry is the mechanism, so keep it.

getCandles escalates from append to full refetch when freshness.adjustmentBasisStale says the basis predates a corporate action that has already gone ex:

newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale     = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)

The already-ex bound lives in newestCorporateAction, not in the verdict, and that placement is load-bearing in both directions. A declared-but-not-yet-ex distribution is reflected in no provider's adjusted series, so treating it as something to catch up to would refetch on every pass forever. But bounding the verdict instead - "is the newest action of all still in the future? then nothing to do" - lets one forward announcement hide every older unapplied action behind it. That shipped: a quarterly payer announcing a quarter ahead was permanently unable to restate.

The check runs on the stale path only, not on the fresh-cache early-return. Detection is therefore at most one trading day behind, which the candle TTL guarantees, and the hot portfolio-pricing path pays nothing.

Steady state for a quarterly payer is four full-history fetches a year, each landing within about a trading day of an ex-date. zfin diagnose SYMBOL reports the basis against the newest already-ex action.

Freshness is the #!expires= directive, not mtime

A cache entry is fresh when the wall clock is earlier than the #!expires= epoch-seconds directive embedded in the file. File modification time is not consulted for freshness anywhere - mtime is a fragile signal (it changes on copy, restore, rsync, and filesystem quirks), so the expiry is written into the content itself.

• On write, computeExpires (store.zig) sets #!expires = now + TTL for the data type, optionally offset by a per-key deterministic jitter to avoid thundering-herd refreshes.
• On read, the SRF iterator parses #!expires= and isFresh compares it to Timestamp.now(io, .real).
• An entry with no #!expires= is treated as stale by zfin's .fresh_only reads (a deliberate override of SRF's "no expiry = always fresh" default), except for negative entries, which are always fresh.

TTLs by data type

Base TTLs live in Ttl (store.zig); jitter is applied per call site in DataType.ttl().

Data type	TTL	Jitter	Notes
Daily candles	market-aware	n/a	Boundary set by market.nextCandleExpiry (below)
Historical candles	never (-1)	n/a	Bars older than ~1 day are immutable
Dividends	6 days	11%	Backstop only; see dividend smart refresh below
Splits	14 days	11%	Rare corporate events, announced weeks ahead
Options	1 hour	0	Move continuously during market hours
Earnings	30 days	8%	Smart-refresh after an announcement date passes
Classification	90 days	8%	Sector/industry/country from Wikidata
ETF metrics	90 days	8%	NPORT-P profile, quarterly cadence
Entity facts (XBRL)	30 days	8%	Per-CIK, quarterly filing cadence
EDGAR ticker maps	30 days	8%	ticker -> CIK; very stable upstream
Quotes	never cached	n/a	Live by definition (see below)

Market-aware candle freshness

Daily bars are only meaningful once the session settles, so candle expiry is keyed to the market clock rather than a rolling window (market.nextCandleExpiry / market.staleCandleExpiry, market.shouldRefresh):

• Equities/ETFs expire at 16:55 ET on the next trading day.
• Mutual funds (NAV) expire at 03:25 ET the next morning.

If a refresh fires but the provider has not posted the just-closed bar yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes overdue, the code concludes the session was an un-modeled closure (Good Friday, weather) and falls back to the next normal boundary instead of thrashing all day. See the user page for the cron-timing rationale.

The fetch decision

Tiers (the big picture)

flowchart TD
    A["Data request via DataService"] --> B{"Local cache fresh?"}
    B -->|yes| C["Deserialize and return, no network"]
    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
    D -->|yes| E["GET server, write bytes verbatim"]
    E --> F{"Synced entry fresh?"}
    F -->|yes| C
    F -->|no| G["Provider fetch"]
    D -->|no| G
    G --> H{"Result?"}
    H -->|ok| I["Write cache with new expiry, return"]
    H -->|NotFound| J["Write negative cache, return FetchFailed"]
    H -->|transient| K["Return error, retry next run"]

The --refresh-data policy maps to FetchOptions:

• auto (default): all tiers, honor TTL.
• force -> force_refresh = true: skip the local-cache and server tiers, go straight to the provider, re-stamp the cache. Bypasses negative entries (so it retries dead lookups).
• never -> skip_network = true: stop at the local cache; return stale data if present, never touch the network.

getCandles (single symbol)

This is the most involved path because of the daily/meta split, the incremental-update logic, the adjustment-basis escalation, and the TwelveData carve-out.

Note also that force re-asks the provider but does not rebuild candle history: it skips the TTL and the server tier, then takes the same incremental top-up. Replacing the series is the restatement path's job (or zfin cache clear).

Provider routing is keyed off CandleMeta.tiingo_retry_after_s, not off provider. provider is pure provenance - "where did these bars come from" - and using it to route made a single non-transient Tiingo failure permanent: Yahoo got tried first, succeeded, rewrote provider = .yahoo, and Tiingo was never consulted again. The backoff is armed only by a genuine 404, expires after Ttl.tiingo_backoff with per-symbol jitter, and clears the moment Tiingo serves the symbol again.

CandleProvider has four variants - twelvedata, yahoo, tiingo, external - and exactly one of them is special-cased anywhere: twelvedata, whose adj_close was unreliable, so a cache carrying it is treated as unusable. That check exists in two places, one per entry into the meta-exists branch (the skip_network path reports the symbol unavailable; the online path forces a full re-fetch). external is covered under Externally-managed candle series; it is inert on every serve path and vetoes only the negative-cache write.

flowchart TD
    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
    NG -->|yes| FF["return FetchFailed, no network"]
    NG -->|no| RM{"candles_meta exists?"}

    RM -->|yes| SK{"skip_network?"}
    SK -->|yes| TWS{"provider is twelvedata?"}
    TWS -->|yes| FF
    TWS -->|no| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
    SK -->|no| TW{"provider is twelvedata?"}
    TW -->|yes| FULL
    TW -->|no| FR{"meta fresh and not force_refresh?"}
    FR -->|yes| RET["return cached candles"]
    FR -->|no| SS1["syncCandlesFromServer"]
    SS1 --> SF1{"fresh AND adj_basis current?"}
    SF1 -->|yes| RET
    SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
    AB -->|yes| REST["refetchFullHistory: restate whole series"]
    REST --> RR{"ok?"}
    RR -->|yes| RET2["return fetched"]
    RR -->|no| INC
    AB -->|no| INC{"shouldRefresh?"}
    INC -->|no| BUMP["bump TTL, return cached"]
    INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]

    RM -->|no| SN{"skip_network?"}
    SN -->|yes| FF
    SN -->|no| SS2["syncCandlesFromServer"]
    SS2 --> SF2{"fresh now?"}
    SF2 -->|yes| RET
    SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
    FULL --> RES{"result?"}
    RES -->|ok| RET2
    RES -->|"NotFound (EVERY provider disclaims it)"| SNC{"prior meta says provider is external?"}
    SNC -->|yes| FF
    SNC -->|no| WN["writeNegative candles_daily"]
    WN --> FF
    RES -->|transient| TR["bump fail_count, TransientError"]
    RES -->|other| FF

Two things about this shape are easy to get wrong.

The basis check precedes shouldRefresh. A symbol that needs both a top-up and a restatement costs one full fetch, not an append followed by a second pass. And a failed restatement falls through to the ordinary top-up rather than erroring: the existing series is untouched and still usable, just understated by the missed adjustment, so it retries on the next stale pass.

Only a unanimous NotFound earns a negative entry. writeNegative overwrites candles_daily.srf with a marker, so a verdict of "no such symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and refetchFullHistory returns error.NotFound only when every provider disclaims the symbol. Anything else (auth trouble, a malformed body, a network blip) fails the call but leaves the cache alone. The restatement path above never writes a negative entry at all, for the same reason: it is reached while holding a working series.

The no-prior-cache branch does not re-check the basis after a server sync, only freshness. A stale basis inherited from the server is caught on the next invocation, which takes the meta-exists branch. One invocation of understated returns, then it self-corrects.

Key invariant: the negative marker for a candle-less symbol lives in candles_daily.srf, and every candle decision honors it there - isCandleMetaFresh (the price fast-path gate), getCachedCandles (the cache-only display path), and the getCandles short-circuit above. This matters because candles_meta.srf is never created for a symbol that has no candles, so anything keying freshness off the meta file alone would treat such a symbol as perpetually stale and re-fetch it forever.

fetchCached (dividends, splits, options, earnings)

Everything that is not candles flows through the generic fetchCached, which is simpler because each type is a single file:

1. .fresh_only read; a fresh entry (including a negative one) returns immediately -- unless the type supplies a needsRefresh hook that says the fresh entry is incomplete (see the freshness levels below).
2. skip_network: return any cached entry, even stale; else FetchFailed.
3. Server sync (if configured); a fresh synced entry returns, subject to the same needsRefresh check.
4. Provider fetch; on success write with the TTL from ttlSpecFor; on NotFound write a negative entry; on NoApiKey return NoApiKey; on any other transient error return FetchFailed. Neither of the last two poisons the cache.

The three freshness levels

Dividends and earnings do not get a single freshness verdict; they get three, in a fixed order. The user-facing explanation is in Caching and data freshness; this is where each level lives in code.

Level	Name	Question	Implementation
1	expiry clock	"Is this entry old?"	#!expires= vs now, enforced by Store.read(.fresh_only)
2	completeness check	"Is this entry missing something it should already have?"	needsRefresh hook, via serveFreshOrDiscard
3	recheck floor	"Did we already ask the provider recently?"	askedRecently, 12h

The ordering is the part to get right, and it is the opposite of the natural reading. Level 2 is a second opinion on a "still fresh" verdict from Level 1, not an extra gate in front of a refresh:

• Level 1 says stale -> Store.read(.fresh_only) returns null, the if body in step 1 never executes, and serveFreshOrDiscard is never called. Levels 2 and 3 do not run at all. There is nothing for them to contribute: a refetch is already happening.
• Level 1 says fresh -> Level 2 runs, and its whole job is to be able to overrule that with "no it isn't, a payment has happened that this entry doesn't contain."
• Level 2 says incomplete -> Level 3 runs, and can overrule Level 2 back to "serve the cache anyway" if we asked too recently.

So each level can only be reached by the previous one declining to decide. If you ever find yourself reasoning about Level 2 firing on a stale entry, the model is inverted.

skip_network short-circuits between Levels 1 and 2: offline mode serves the fresh entry without consulting either later level, since it can never refetch and discarding could only turn a served result into a failure.

NoApiKey is propagated rather than collapsed into FetchFailed because it is the one distinction callers act on differently: commands/earnings.zig and tui/earnings_tab.zig both name the environment variable to set, which they cannot do from a generic failure. It also must not negative-cache -- nothing is wrong with the symbol, and poisoning it would suppress the fetch after the key is finally configured.

getEarnings is a thin wrapper over this, not a parallel implementation: it short-circuits mutual funds (no quarterly earnings exist, so there is nothing to cache or fetch) and then delegates, supplying earningsPostProcess and earningsNeedsRefreshHook.

The three comptime hooks

fetchCached takes three optional comptime function pointers. Passing null for all three gives the behaviour every type had before they existed, which is what getSplits and getOptions do.

Hook	Signature	Question it answers
postProcess	fn (*T, Allocator) anyerror!void	Any derived field to recompute after parsing?
needsRefresh	fn ([]const T, Date) bool	This entry is fresh, but is it complete?
ttlFor	fn ([]const T) cache.TtlSpec	How long should this data stay fresh?

Who passes what:

Getter	postProcess	needsRefresh	ttlFor
getDividends	-	dividendsNeedRefresh	dividendTtl
getEarnings	earningsPostProcess	earningsNeedsRefreshHook	-
getSplits	-	-	-
getOptions	-	-	-

postProcess is not for duping strings -- Store.read already does that. Only EarningsEvent uses it, to rebuild surprise.

needsRefresh is consulted at both fresh-read sites (steps 1 and 3 above) through serveFreshOrDiscard, which owns the free so a caller cannot decide to discard and then leak. Covering step 3 is not optional: with a ZFIN_SERVER configured, a sync can leave a still-incomplete entry on disk that step 3 would otherwise serve, defeating the mechanism entirely. (This is not theoretical - it was observed against a live server whose copy was missing the same record the local cache was.) The hook is suppressed under skip_network, since offline mode never refetches and discarding could only turn a served result into a failure.

ttlFor exists because the right expiry time for dividends depends on the records being written, not just on their type - see dividendTtl. It is applied through ttlSpecFor at both write sites (the normal path and the rate-limit retry) so a retry cannot stamp a different clock than the first attempt would have.

All three branches are comptime-elided when the argument is null, which is what lets types without a freeSlice (such as Split) keep compiling.

The recheck floor

A needsRefresh hook has a failure mode that is easy to miss. When the hook says something is overdue and the provider does not have it yet - the normal case, not an error - the cache is rewritten with a fresh expiry but the same records. The hook therefore still says "overdue", and the next command asks again. On a quarter-end run touching fifteen funds that is fifteen provider calls per command, repeated every time anything runs.

smart_refresh_recheck_interval (12 hours) is the floor, enforced by askedRecently. It applies to any type with a needsRefresh hook.

It reads #!expires=, not #!created=, and that distinction is the whole trick. serializeWithMeta re-stamps created on every write, and dividend rows also arrive as a side effect of a Tiingo candle fetch via writeSupplement. So created means "file last touched", which a candle refresh changes without anyone asking Polygon anything - keying the floor off it would suppress a genuinely due refetch for 12 hours. expires is moved only by a primary-provider fetch (ExpiryPolicy.bump); a supplement deliberately puts the old value back (.preserve). There is a regression test for exactly this.

Recovering the write time from expires is exact, not approximate, because computeExpires derives its jitter from a hash of the symbol - the same symbol always gets the same offset. Calling it with a zero "now" returns precisely the TTL that was applied, so expires - that is the write time to the second.

Two cases fall back to "ask again", which is the safe direction: a file with no #!expires= at all, and a symbol that crossed from unknown-schedule to known-schedule since it was written (stored under the short TTL, recomputed with the long one, so the write looks older than it was).

Dividend schedule prediction

dividendsNeedRefresh returns true when at least one full cadence period has elapsed since the newest cached ex-date and the payment that should have landed in it is still absent.

It exists because a dividend cache being fresh and being complete are unrelated. An operating company declares weeks ahead of the ex-date and pays about a month later. An ETF cannot: its distribution is the pass-through of whatever the fund earned, so the amount does not exist until roughly its own ex-date, and it pays 1-7 days later. The period in which the record exists but the cash has not landed is about a week, so any longer expiry can be written before the record exists and still be fresh after the payment. Quarter-end clustering makes it systematic -- a dozen funds go ex within a few days, so one refresh pass puts all of them on the same expiry date with the same blind spot.

Cadence comes from dividendCadenceDays: the median of up to the five newest gaps between cached ex-dates. Median rather than mean (a special distribution landing days after a regular one drags the mean) and rather than minimum (the same outlier destroys the minimum outright, predicting a next ex-date days away and firing for the whole chase window after every payment). newestExDates sorts rather than trusting file order, because writeSupplement's merge interleaves two providers' records and a cadence computed from negative gaps would forecast into the past forever.

Three properties are load-bearing and each has its own test:

• A full period must elapse. Otherwise it refires immediately after a successful fetch, on the record it just stored.
• It rolls forward. The expected date is the latest one at or before today, not the first one after the newest cached record -- so a symbol two periods behind stays recoverable instead of being permanently past its window.
• The chase window is bounded (dividend_chase_days, 14). An unbounded chase refetches forever whenever a sponsor skips a period or the cadence estimate is wrong.

dividendTtl then picks the expiry from the same cadence calculation: Ttl.dividends_scheduled (14 days) when a cadence exists, because prediction is what finds due payments; Ttl.dividends (6 days) when it does not, because then the clock is the only mechanism and it has to expire before the next weekly review. Six is the largest value that guarantees that -- 6 x 1.11 jitter = 6.66 days, under 7, where 7 would reach 7.77. "TTL constants are reasonable" in store.zig asserts that property rather than just the number.

Known blind spots, all left to the expiry time rather than papered over with a more eager statistic that would cost requests on every symbol to protect against a rare event on one:

• A cadence change (quarterly to monthly) keeps the old median for several periods, so the forecast runs late.
• An off-cycle special is on no schedule, so prediction never fires for it.
• A symbol with fewer than three cached records has no cadence at all -- which is exactly the case that gets the 6-day expiry.

Why Tiingo can never discover a new dividend

Tiingo returns dividends and splits alongside candles in a single request (populateAllFromTiingo), written through writeSupplement so the merge lands new records without moving the primary provider's #!expires= clock. That makes it look like a free second source for Level 2, and it is worth being explicit that it is not - for two independent reasons.

Structural. Tiingo carries a dividend as a divCash field on a daily price bar, so a payment can only appear once a bar exists for it. A declared-but-not-yet-ex dividend has no bar and therefore cannot be represented at all. This is exactly why Polygon is the primary source - see the comment in fetchFromProvider, which cites a forward-dated ARCC ex-date as the case Tiingo's response cannot express.

Circular. populateAllFromTiingo is reached from only two places:

1. A cold candle cache - the symbol has no bars at all (getCandles's full-history path). Here Tiingo genuinely earns its keep: one request bulk-loads the symbol's entire dividend history.
2. An adjustment-basis restatement - cached adj_close values predate a corporate action they should reflect, so the series is refetched in full.

Trigger 2 is decided by freshness.adjustmentBasisStale fed by freshness.newestCorporateAction, and that function reads the cached dividend file to find the newest action. So zfin only knows to restate after the dividend record is already on disk - which means Polygon must have supplied it first. By the time Tiingo's rows merge in, they add nothing about that payment.

Net: Tiingo bulk-loads history and fills field-level gaps in old records. Every new dividend arrives via Polygon. This is also why the Level 3 floor reads #!expires= and not #!created= - a Tiingo supplement rewrites the file, and therefore created, without having asked Polygon anything.

For the user-facing version of all this, including a worked quarterly example, see Caching and data freshness.

loadAllPrices (portfolio + watchlist price load)

The portfolio price load batches all symbols through three phases. Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider fallback that calls getCandles.

flowchart TD
    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
    P1 --> CF{"cache fresh and not force_refresh?"}
    CF -->|yes| HIT["use cached last close, cached_count++"]
    CF -->|no| ADD["add to needs_fetch"]
    HIT --> CHK
    ADD --> CHK{"needs_fetch empty?"}
    CHK -->|yes| DONE["return"]
    CHK -->|no| OFF{"skip_network?"}
    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
    P2 --> REM["unsynced to server_failures"]
    ALLF --> P3
    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
    P3 --> END["return prices + counts"]

Negative caching

When a provider says a symbol genuinely has no data of a type - an error.NotFound - zfin writes a negative cache entry so it does not re-run the dead lookup on every invocation. The entry is the sentinel:

#!srfv1
# fetch_failed

(Store.negative_cache_content). Rules:

• Only NotFound qualifies. isPermanentProviderFailure gates the write. Rate-limit, 5xx, connection, auth, and parse failures are transient - they fail the call but leave the cache untouched so the next run retries. (Auth/parse looking permanent but being transient is exactly why they must not poison a now-sticky negative cache.)
• Negative entries are always fresh. They have no #!expires=; readSlice, read, and isCandleMetaFresh special-case the sentinel as fresh, so they stick until --refresh-data=force or cache clear.
• Candles key the negative off candles_daily.srf. writeNegative writes that file; isCandleMetaFresh, getCachedCandles, and the getCandles short-circuit all recognize it there. candles_meta.srf is intentionally not created for a no-data symbol.
• Externally-managed series are exempt. A prior meta saying provider::external vetoes the write via DataService.shouldNegativeCache, because for a series no provider was ever going to carry, the marker is unrecoverable rather than merely sticky. See Externally-managed candle series.

Candle-less symbols (crypto and friends)

Some held symbols have no daily candles available from any candle provider - cryptocurrencies on the Yahoo DOGE-USD / BTC-USD shape are the common case, and delisted or invalid tickers behave identically. For these symbols getCandles writes a negative entry and never produces a price from history. "Any" is literal: Tiingo and Yahoo must both disclaim the symbol, because the marker overwrites candles_daily.srf.

Such symbols are still priced, through two mechanisms that do not touch the candle cache:

• Live quotes (Yahoo). loadLiveQuotes / getQuote fetch an intraday price from Yahoo, which does serve crypto. The TUI overlays these live quotes on top of the candle-close price map on refresh and on every streaming tick, so a candle-less holding shows its real current price there. Live quotes are never cached.
• Manual price. A price:: field on a lot in portfolio.srf pins a value. When neither a candle close nor a live quote is available, buildFallbackPrices (analytics/valuation.zig) falls back to the position's average cost and flags it as a manual/estimated price (rendered in a warning color).

Practical consequence by surface: the plain CLI portfolio command does not apply the live-quote overlay, so a candle-less holding shows its average-cost fallback (break-even, warning color); the TUI shows the live Yahoo price. This is expected - historical-candle commands (perf, charts) simply have no data for these symbols, while quote-driven views do.

If you want a candle-less symbol to be re-checked against the provider (for example a ticker that has since started trading), clear its negative entry with cache clear or --refresh-data=force; the live quote and manual-price paths are unaffected by the negative cache.

Externally-managed candle series

CandleMeta.provider == .external labels a series that was produced and is maintained outside zfin, by something that writes candles_daily.srf and candles_meta.srf directly into a cache directory. The motivating case is a unitized trust with no ticker and no CUSIP, whose daily unit values come from a plan recordkeeper's feed; every market-data provider 404s it.

Like the previous section, this is a symbol no provider carries. The difference is that a candle-less symbol has no history anywhere, so negative-caching it is correct, whereas an externally-managed symbol has a real series that simply did not come from a provider. The label exists so candles_meta.srf can say so honestly instead of naming a provider that never served it.

Two population routes:

• Server: a job writes the pair straight into the ZFIN_SERVER cache directory. That copy is the master.
• Client: the existing server-sync path (phase 2 of loadAllPrices, and the two syncCandlesFromServer calls inside getCandles) pulls the bytes down verbatim, exactly as it does for any other symbol. No client-side special-casing is involved.

The label is pure provenance and drives no routing. zfin still walks the normal provider chain for such a symbol and still 404s on every pass; restricting it to the server tier and skipping the chain entirely is deferred work. There is exactly one behavior attached to the label, and it is a veto rather than a routing decision:

A negative-cache entry on an external symbol is unrecoverable. writeNegative overwrites candles_daily.srf with a marker, negative entries never expire, and getCandles short-circuits on isNegative before it reaches syncCandlesFromServer - so the marker makes the only surviving copy permanently unreachable. On a server it is worse: the externally-populated cache is the master copy, so nothing survives. DataService.shouldNegativeCache therefore refuses the write when the prior meta says .external.

That guard is deliberately partial. It consults the prior meta, so it covers the case that destroys data: meta present, candles_daily.srf missing or unreadable (a partially-completed server sync writes the two files in sequence and needs both to report success), which falls through to the cold-start path. It cannot cover a true cold start with both files absent - there is no label to read - so an external symbol first touched while ZFIN_SERVER is unreachable is poisoned until --refresh-data=force or cache clear. Closing that needs the isNegative short-circuit moved after the server tier, which would re-hit the network for every legitimately candle-less symbol on every run. It belongs with the deferred routing work.

The label is not sticky, and must not become sticky

external describes where a series came from now, not a permanent property of the symbol. If a provider ever does serve it, the bars in the cache genuinely came from that provider, and applyTiingoCoverage overwriting the label with .tiingo / .yahoo is the correct outcome: the provenance changed, and the symbol has rejoined the ordinary provider path under its own steam. That is a good day, not a bug.

So do not add a guard preserving .external across a successful fetch, and any future routing built on the variant must keep that escape hatch open. Making it a one-way door recreates the latch bug that keying routing off provider == .yahoo already produced - see the tiingo_retry_after_s rationale - except worse, because the pinned symbol would be pinned to a tier that only one machine can populate.

Adding a CandleProvider variant is a coordinated deployment

Not a backward-compatible change. provider has no default, so it is never elided, and SRF's enum coercion has no lenient mode and no srfParse hook (custom parsers are consulted for struct/union fields only). A build that does not know a variant cannot fall back:

• srf >= ea2c358 (zfin's current pin) returns StringValueNotValidEnumMember. readCandleMeta swallows it into a null, which reads as a cache miss and routes the symbol down the cold-start path above - destructive for precisely the series that cannot be re-fetched.
• srf <= 4a3e5f0 unwraps a null optional and panics. That is what zfin-server 7103ced vendors, and a panic is not interceptable by the catch return null every caller depends on. The blast radius is not one symbol: handleDiagnostics runs freshness.collect over every key in the cache directory, so a single unreadable file takes down /<any-symbol>/diagnostics and the cron refresh sweep. The byte-serving routes (/candles, /candles_meta) are unaffected, since they never parse.

Skew is dangerous in both directions: an old client syncing candles_meta bytes from a new server fails the same way, because looksCompleteSrf and the ETag check validate shape and integrity, not enum values.

So the ordering is: release zfin, then rebuild and deploy every server that reads the affected cache directory, and only then may a file carrying the new value exist.

Server sync (the optional L2 tier)

ZFIN_SERVER points zfin at a zfin-server instance - a shared cache between your local cache and the upstream providers. When unset, every server-sync path silently no-ops.

Client side (syncFromServer, syncCandlesFromServer, parallelServerSync in service.zig):

• Triggered on a local miss/stale entry, before any provider call (skipped under force_refresh).
• GET {ZFIN_SERVER}/{SYMBOL}/{type}; the response body is validated (sha256 ETag, completeness check) and written to the local cache verbatim via writeRaw - the client does not re-stamp #!expires=, so it inherits the server's freshness boundary.
• parallelServerSync fans out one task per symbol for the portfolio price load (each worker uses its own HTTP client; the allocator is thread-safe).

Server side (serveSrfFile, fetchOnMiss in zfin-server):

• A present file is served as-is, even if stale - the server's cron is the freshness authority; reads never trigger a refetch.
• An absent file triggers a one-shot fetchOnMiss (which calls the same getCandles / fetchCached code through the shared zfin library), then re-reads; if still absent, it returns 404.

Because the server runs the same library, the negative-cache rules above apply there too: a candle-less symbol gets a negative candles_daily.srf on first miss and is served from it thereafter, rather than re-hitting the upstream provider on every request.

Invalidation and atomicity

• DataService.invalidate(symbol) clears a symbol's entries; for candles it removes the candles_daily + candles_meta pair together.
• cache clear wipes the whole cache directory; everything re-fetches next run.
• All writes are crash-safe: atomic.zig writes to a temp file, fsyncs, and renames into place, so a reader never sees a torn file. A defensively detected torn candle file self-heals by wiping the pair.

Key code references

Concern	Location
Data-access entry point	DataService - src/service.zig
Per-type generic fetch	fetchCached - src/service.zig
Fresh-but-incomplete decision	serveFreshOrDiscard - src/service.zig
Recheck floor (12h)	askedRecently, smart_refresh_recheck_interval - src/service.zig
Dividend schedule prediction	dividendsNeedRefresh, dividendCadenceDays - src/service.zig
Dividend expiry choice	dividendTtl, ttlSpecFor - src/service.zig
Earnings smart refresh	earningsNeedsRefresh, earningsNeedsRefreshHook - src/service.zig
Candle fetch + incremental	getCandles - src/service.zig
Batch price load	loadAllPrices - src/service.zig
Live quotes (uncached)	loadLiveQuotes, getQuote - src/service.zig
Cache store, read/write	Store - src/cache/store.zig
Freshness check	isFresh (SRF), isCandleMetaFresh - store.zig
TTLs and expiry computation	Ttl, computeExpires - src/cache/store.zig
Negative cache	writeNegative, isNegative - src/cache/store.zig
Negative-cache veto	shouldNegativeCache - src/service.zig
Candle provenance labels	CandleProvider - src/cache/store.zig
NotFound classification	isPermanentProviderFailure - src/service.zig
Market-aware candle expiry	nextCandleExpiry, shouldRefresh - src/market.zig
Price fallback (manual/avg-cost)	buildFallbackPrices - src/analytics/valuation.zig
Server endpoints	serveSrfFile, fetchOnMiss - zfin-server/src/main.zig

For the user-facing summary and the --refresh-data walkthrough, see Caching and data freshness.