# Cache implementation (developer reference)
This is the low-level, contributor-facing companion to the user-facing
[Caching and data freshness](../explanation/caching.md) page. It
documents *how* the cache is built: the on-disk layout, the freshness
model, the fetch-decision flow, negative caching, and the optional
server (L2) tier. If you are changing anything in `src/cache/store.zig`
or the fetch paths in `src/service.zig`, read this first.
Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
natively.
## Where the data lives
`DataService` (`src/service.zig`) is the sole data-access layer. Both
the CLI and the TUI go through it; nothing else calls a provider
directly. It reads and writes a per-symbol, per-type SRF file cache via
`Store` (`src/cache/store.zig`):
```
{cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR
AAPL/
candles_daily.srf OHLCV bars (appended; replaced on restatement)
candles_meta.srf last_close, last_date, provider, adj_basis + freshness
dividends.srf
splits.srf
options.srf
earnings.srf
classification.srf
etf_metrics.srf
_edgar/
tickers_companies.srf shared EDGAR ticker -> CIK maps
tickers_funds.srf
0000320193/
entity_facts.srf per-CIK XBRL facts
```
[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
format. Files carry `#!`-prefixed directives (`#!expires=`,
`#!created=`) ahead of their records.
### The candle two-file split
Candles are stored as **two** files, and the split is load-bearing:
- `candles_daily.srf` holds the actual OHLCV records and normally grows
by appending: on a cache miss only bars newer than `last_date` are
fetched and appended, not the full history. The exception is a
**restatement**, which replaces the file wholesale - see
[The adjustment basis](#the-adjustment-basis).
- `candles_meta.srf` holds a single small record (`last_close`,
`last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`,
`adj_basis`) plus the `#!expires=` and `#!created=` directives.
Keeping the metadata separate lets every freshness check and last-price
read touch a ~100-byte file instead of deserializing a multi-megabyte
candle history. The price fast-path in `loadAllPrices` never
deserializes `candles_daily.srf` - at most it peeks at the first bytes
to detect a negative entry.
The two files are a unit: `DataService.invalidate` and the torn-file
self-heal clear both together, and a negative cache entry for a
candle-less symbol is keyed off `candles_daily.srf` (see
[Negative caching](#negative-caching)).
### The adjustment basis
Appending bars cannot restate the bars already on disk, and sometimes
they need it.
Providers compute `adj_close` by scaling raw `close` by the product of
the adjustment factors for every distribution *after* that bar. So a
series' adjustment basis is only as current as the fetch that produced
it. Newly appended bars arrive with `adj_close == close`, because
nothing has gone ex after them yet, while every previously cached bar
keeps the basis it was originally fetched with. When the next
distribution goes ex, the bars behind it should be marked down by its
factor - and an append does not do that. Every total return spanning
that ex-date then reads low by roughly the missed yield.
`CandleMeta.adj_basis` records how current a series' basis is. It is
**the date of the newest bar present at the last full fetch**:
- Not a wall-clock timestamp. A fetch that runs before the day's bar is
published gets a basis one bar behind - which is the honest claim,
since a provider's adjusted series is only ever as complete as its
newest bar.
- Not an ex-date. It is compared against ex-dates but is never one.
- Never advanced by an append. `Store.appendCandles` takes the existing
meta and overrides only `last_close` / `last_date`; only
`Store.cacheCandles`, which replaces the whole file, sets a new
basis. That asymmetry is the mechanism, so keep it.
`getCandles` escalates from append to full refetch when
`freshness.adjustmentBasisStale` says the basis predates a corporate
action that has **already gone ex**:
```zig
newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)
```
The already-ex bound lives in `newestCorporateAction`, not in the
verdict, and that placement is load-bearing in both directions. A
declared-but-not-yet-ex distribution is reflected in no provider's
adjusted series, so treating it as something to catch up to would
refetch on every pass forever. But bounding the *verdict* instead -
"is the newest action of all still in the future? then nothing to do" -
lets one forward announcement hide every older unapplied action behind
it. That shipped: a quarterly payer announcing a quarter ahead was
permanently unable to restate.
The check runs on the stale path only, not on the fresh-cache
early-return. Detection is therefore at most one trading day behind,
which the candle TTL guarantees, and the hot portfolio-pricing path
pays nothing.
Steady state for a quarterly payer is four full-history fetches a year,
each landing within about a trading day of an ex-date. `zfin diagnose
SYMBOL` reports the basis against the newest already-ex action.
## Freshness is the `#!expires=` directive, not mtime
A cache entry is fresh when the wall clock is earlier than the
`#!expires=` epoch-seconds directive embedded in the file. File
modification time is **not** consulted for freshness anywhere - mtime is
a fragile signal (it changes on copy, restore, rsync, and filesystem
quirks), so the expiry is written into the content itself.
- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
for the data type, optionally offset by a per-key deterministic jitter
to avoid thundering-herd refreshes.
- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
it to `Timestamp.now(io, .real)`.
- An entry with **no** `#!expires=` is treated as stale by zfin's
`.fresh_only` reads (a deliberate override of SRF's "no expiry =
always fresh" default), except for negative entries, which are always
fresh.
### TTLs by data type
Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
in `DataType.ttl()`.
| Data type | TTL | Jitter | Notes |
|----------------------|------------------|--------|--------------------------------------------------|
| Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)|
| Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable |
| Dividends | 6 days | 11% | Backstop only; see dividend smart refresh below |
| Splits | 14 days | 11% | Rare corporate events, announced weeks ahead |
| Options | 1 hour | 0 | Move continuously during market hours |
| Earnings | 30 days | 8% | Smart-refresh after an announcement date passes |
| Classification | 90 days | 8% | Sector/industry/country from Wikidata |
| ETF metrics | 90 days | 8% | NPORT-P profile, quarterly cadence |
| Entity facts (XBRL) | 30 days | 8% | Per-CIK, quarterly filing cadence |
| EDGAR ticker maps | 30 days | 8% | ticker -> CIK; very stable upstream |
| Quotes | never cached | n/a | Live by definition (see below) |
### Market-aware candle freshness
Daily bars are only meaningful once the session settles, so candle
expiry is keyed to the market clock rather than a rolling window
(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
`market.shouldRefresh`):
- Equities/ETFs expire at **16:55 ET** on the next trading day.
- Mutual funds (NAV) expire at **03:25 ET** the next morning.
If a refresh fires but the provider has not posted the just-closed bar
yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
overdue, the code concludes the session was an un-modeled closure (Good
Friday, weather) and falls back to the next normal boundary instead of
thrashing all day. See the user page for the cron-timing rationale.
## The fetch decision
### Tiers (the big picture)
```mermaid
flowchart TD
A["Data request via DataService"] --> B{"Local cache fresh?"}
B -->|yes| C["Deserialize and return, no network"]
B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
D -->|yes| E["GET server, write bytes verbatim"]
E --> F{"Synced entry fresh?"}
F -->|yes| C
F -->|no| G["Provider fetch"]
D -->|no| G
G --> H{"Result?"}
H -->|ok| I["Write cache with new expiry, return"]
H -->|NotFound| J["Write negative cache, return FetchFailed"]
H -->|transient| K["Return error, retry next run"]
```
The `--refresh-data` policy maps to `FetchOptions`:
- `auto` (default): all tiers, honor TTL.
- `force` -> `force_refresh = true`: skip the local-cache and server
tiers, go straight to the provider, re-stamp the cache. Bypasses
negative entries (so it retries dead lookups).
- `never` -> `skip_network = true`: stop at the local cache; return
stale data if present, never touch the network.
### `getCandles` (single symbol)
This is the most involved path because of the daily/meta split, the
incremental-update logic, the adjustment-basis escalation, and the
TwelveData carve-out.
Note also that `force` re-asks the provider but does **not** rebuild
candle history: it skips the TTL and the server tier, then takes the
same incremental top-up. Replacing the series is the restatement path's
job (or `zfin cache clear`).
Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off
`provider`. `provider` is pure provenance - "where did these bars come
from" - and using it to route made a single non-transient Tiingo failure
permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`,
and Tiingo was never consulted again. The backoff is armed only by a
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
and clears the moment Tiingo serves the symbol again.
`CandleProvider` has four variants - `twelvedata`, `yahoo`, `tiingo`,
`external` - and exactly one of them is special-cased anywhere:
`twelvedata`, whose `adj_close` was unreliable, so a cache carrying it is
treated as unusable. That check exists in **two** places, one per entry
into the meta-exists branch (the `skip_network` path reports the symbol
unavailable; the online path forces a full re-fetch). `external` is
covered under [Externally-managed candle
series](#externally-managed-candle-series); it is inert on every serve
path and vetoes only the negative-cache write.
```mermaid
flowchart TD
S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
NG -->|yes| FF["return FetchFailed, no network"]
NG -->|no| RM{"candles_meta exists?"}
RM -->|yes| SK{"skip_network?"}
SK -->|yes| TWS{"provider is twelvedata?"}
TWS -->|yes| FF
TWS -->|no| RETS["return cached even if stale
(FetchFailed if unreadable)"]
SK -->|no| TW{"provider is twelvedata?"}
TW -->|yes| FULL
TW -->|no| FR{"meta fresh and not force_refresh?"}
FR -->|yes| RET["return cached candles"]
FR -->|no| SS1["syncCandlesFromServer"]
SS1 --> SF1{"fresh AND adj_basis current?"}
SF1 -->|yes| RET
SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
AB -->|yes| REST["refetchFullHistory: restate whole series"]
REST --> RR{"ok?"}
RR -->|yes| RET2["return fetched"]
RR -->|no| INC
AB -->|no| INC{"shouldRefresh?"}
INC -->|no| BUMP["bump TTL, return cached"]
INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]
RM -->|no| SN{"skip_network?"}
SN -->|yes| FF
SN -->|no| SS2["syncCandlesFromServer"]
SS2 --> SF2{"fresh now?"}
SF2 -->|yes| RET
SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
FULL --> RES{"result?"}
RES -->|ok| RET2
RES -->|"NotFound (EVERY provider disclaims it)"| SNC{"prior meta says provider is external?"}
SNC -->|yes| FF
SNC -->|no| WN["writeNegative candles_daily"]
WN --> FF
RES -->|transient| TR["bump fail_count, TransientError"]
RES -->|other| FF
```
Two things about this shape are easy to get wrong.
**The basis check precedes `shouldRefresh`.** A symbol that needs both a
top-up and a restatement costs one full fetch, not an append followed by
a second pass. And a failed restatement falls through to the ordinary
top-up rather than erroring: the existing series is untouched and still
usable, just understated by the missed adjustment, so it retries on the
next stale pass.
**Only a unanimous `NotFound` earns a negative entry.** `writeNegative`
*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such
symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and
`refetchFullHistory` returns `error.NotFound` only when every provider
disclaims the symbol. Anything else (auth trouble, a malformed body, a
network blip) fails the call but leaves the cache alone. The restatement
path above never writes a negative entry at all, for the same reason: it
is reached while holding a working series.
The no-prior-cache branch does not re-check the basis after a server
sync, only freshness. A stale basis inherited from the server is caught
on the next invocation, which takes the meta-exists branch. One
invocation of understated returns, then it self-corrects.
Key invariant: the negative marker for a candle-less symbol lives in
`candles_daily.srf`, and **every** candle decision honors it there -
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
cache-only display path), and the `getCandles` short-circuit above. This
matters because `candles_meta.srf` is never created for a symbol that
has no candles, so anything keying freshness off the meta file alone
would treat such a symbol as perpetually stale and re-fetch it forever.
### `fetchCached` (dividends, splits, options, earnings)
Everything that is not candles flows through the generic `fetchCached`,
which is simpler because each type is a single file:
1. `.fresh_only` read; a fresh entry (including a negative one) returns
immediately -- **unless** the type supplies a `needsRefresh` hook that
says the fresh entry is incomplete (see the freshness levels below).
2. `skip_network`: return any cached entry, even stale; else
`FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns, subject to
the same `needsRefresh` check.
4. Provider fetch; on success write with the TTL from `ttlSpecFor`; on
`NotFound` write a negative entry; on `NoApiKey` return `NoApiKey`;
on any other transient error return `FetchFailed`. Neither of the
last two poisons the cache.
#### The three freshness levels
Dividends and earnings do not get a single freshness verdict; they get
three, in a fixed order. The user-facing explanation is in
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check);
this is where each level lives in code.
| Level | Name | Question | Implementation |
|-------|--------------------|-----------------------------------------------------------|------------------------------------------------------------|
| **1** | expiry clock | "Is this entry old?" | `#!expires=` vs now, enforced by `Store.read(.fresh_only)` |
| **2** | completeness check | "Is this entry missing something it should already have?" | `needsRefresh` hook, via `serveFreshOrDiscard` |
| **3** | recheck floor | "Did we already ask the provider recently?" | `askedRecently`, 12h |
**The ordering is the part to get right, and it is the opposite of the
natural reading.** Level 2 is a **second opinion on a "still fresh" verdict
from Level 1**, not an extra gate in front of a refresh:
- Level 1 says **stale** -> `Store.read(.fresh_only)` returns `null`, the
`if` body in step 1 never executes, and `serveFreshOrDiscard` is never
called. **Levels 2 and 3 do not run at all.** There is nothing for them
to contribute: a refetch is already happening.
- Level 1 says **fresh** -> Level 2 runs, and its whole job is to be able
to overrule that with "no it isn't, a payment has happened that this
entry doesn't contain."
- Level 2 says **incomplete** -> Level 3 runs, and can overrule Level 2
back to "serve the cache anyway" if we asked too recently.
So each level can only be reached by the previous one declining to
decide. If you ever find yourself reasoning about Level 2 firing on a
stale entry, the model is inverted.
`skip_network` short-circuits between Levels 1 and 2: offline mode serves
the fresh entry without consulting either later level, since it can never
refetch and discarding could only turn a served result into a failure.
`NoApiKey` is propagated rather than collapsed into `FetchFailed`
because it is the one distinction callers act on differently:
`commands/earnings.zig` and `tui/earnings_tab.zig` both name the
environment variable to set, which they cannot do from a generic
failure. It also must not negative-cache -- nothing is wrong with the
symbol, and poisoning it would suppress the fetch after the key is
finally configured.
`getEarnings` is a thin wrapper over this, not a parallel
implementation: it short-circuits mutual funds (no quarterly earnings
exist, so there is nothing to cache or fetch) and then delegates,
supplying `earningsPostProcess` and `earningsNeedsRefreshHook`.
#### The three comptime hooks
`fetchCached` takes three optional comptime function pointers. Passing
`null` for all three gives the behaviour every type had before they
existed, which is what `getSplits` and `getOptions` do.
| Hook | Signature | Question it answers |
|----------------|--------------------------------------|-----------------------------------------------|
| `postProcess` | `fn (*T, Allocator) anyerror!void` | Any derived field to recompute after parsing? |
| `needsRefresh` | `fn ([]const T, Date) bool` | This entry is fresh, but is it *complete*? |
| `ttlFor` | `fn ([]const T) cache.TtlSpec` | How long should this data stay fresh? |
Who passes what:
| Getter | postProcess | needsRefresh | ttlFor |
|---------------|------------------------|-----------------------------|----------------|
| `getDividends`| - | `dividendsNeedRefresh` | `dividendTtl` |
| `getEarnings` | `earningsPostProcess` | `earningsNeedsRefreshHook` | - |
| `getSplits` | - | - | - |
| `getOptions` | - | - | - |
`postProcess` is **not** for duping strings -- `Store.read` already does
that. Only `EarningsEvent` uses it, to rebuild `surprise`.
`needsRefresh` is consulted at **both** fresh-read sites (steps 1 and 3
above) through `serveFreshOrDiscard`, which owns the free so a caller
cannot decide to discard and then leak. Covering step 3 is not optional:
with a `ZFIN_SERVER` configured, a sync can leave a still-incomplete entry
on disk that step 3 would otherwise serve, defeating the mechanism
entirely. (This is not theoretical - it was observed against a live
server whose copy was missing the same record the local cache was.) The
hook is suppressed under `skip_network`, since offline mode never
refetches and discarding could only turn a served result into a failure.
`ttlFor` exists because the right expiry time for dividends depends on the
records being written, not just on their type - see `dividendTtl`. It is
applied through `ttlSpecFor` at both write sites (the normal path and the
rate-limit retry) so a retry cannot stamp a different clock than the
first attempt would have.
All three branches are comptime-elided when the argument is `null`, which
is what lets types without a `freeSlice` (such as `Split`) keep compiling.
#### The recheck floor
A `needsRefresh` hook has a failure mode that is easy to miss. When the
hook says something is overdue and the provider does not have it yet -
the *normal* case, not an error - the cache is rewritten with a fresh
expiry but the same records. The hook therefore still says "overdue", and
the next command asks again. On a quarter-end run touching fifteen funds
that is fifteen provider calls per command, repeated every time anything
runs.
`smart_refresh_recheck_interval` (12 hours) is the floor, enforced by
`askedRecently`. It applies to any type with a `needsRefresh` hook.
**It reads `#!expires=`, not `#!created=`, and that distinction is the
whole trick.** `serializeWithMeta` re-stamps `created` on every write, and
dividend rows also arrive as a side effect of a Tiingo candle fetch via
`writeSupplement`. So `created` means "file last touched", which a candle
refresh changes without anyone asking Polygon anything - keying the floor
off it would suppress a genuinely due refetch for 12 hours. `expires` is
moved only by a primary-provider fetch (`ExpiryPolicy.bump`); a supplement
deliberately puts the old value back (`.preserve`). There is a regression
test for exactly this.
Recovering the write time from `expires` is *exact*, not approximate,
because `computeExpires` derives its jitter from a hash of the symbol -
the same symbol always gets the same offset. Calling it with a zero "now"
returns precisely the TTL that was applied, so `expires - that` is the
write time to the second.
Two cases fall back to "ask again", which is the safe direction: a file
with no `#!expires=` at all, and a symbol that crossed from
unknown-schedule to known-schedule since it was written (stored under the
short TTL, recomputed with the long one, so the write looks older than it
was).
#### Dividend schedule prediction
`dividendsNeedRefresh` returns true when at least one full cadence period
has elapsed since the newest cached ex-date and the payment that should
have landed in it is still absent.
It exists because a dividend cache being *fresh* and being *complete* are
unrelated. An operating company declares weeks ahead of the ex-date and
pays about a month later. An ETF cannot: its distribution is the
pass-through of whatever the fund earned, so the amount does not exist
until roughly its own ex-date, and it pays 1-7 days later. The period in
which the record exists but the cash has not landed is about a week, so
any longer expiry can be written before the record exists and still be
fresh after the payment. Quarter-end clustering makes it systematic -- a
dozen funds go ex within a few days, so one refresh pass puts all of them
on the same expiry date with the same blind spot.
Cadence comes from `dividendCadenceDays`: the **median** of up to the five
newest gaps between cached ex-dates. Median rather than mean (a special
distribution landing days after a regular one drags the mean) and rather
than minimum (the same outlier destroys the minimum outright, predicting a
next ex-date days away and firing for the whole chase window after every
payment). `newestExDates` sorts rather than trusting file order, because
`writeSupplement`'s merge interleaves two providers' records and a cadence
computed from negative gaps would forecast into the past forever.
Three properties are load-bearing and each has its own test:
- **A full period must elapse.** Otherwise it refires immediately after a
successful fetch, on the record it just stored.
- **It rolls forward.** The expected date is the latest one at or before
today, not the first one after the newest cached record -- so a symbol
two periods behind stays recoverable instead of being permanently past
its window.
- **The chase window is bounded** (`dividend_chase_days`, 14). An
unbounded chase refetches forever whenever a sponsor skips a period or
the cadence estimate is wrong.
`dividendTtl` then picks the expiry from the same cadence calculation:
`Ttl.dividends_scheduled` (14 days) when a cadence exists, because
prediction is what finds due payments; `Ttl.dividends` (6 days) when it
does not, because then the clock is the only mechanism and it has to
expire before the next weekly review. Six is the largest value that
guarantees that -- 6 x 1.11 jitter = 6.66 days, under 7, where 7 would
reach 7.77. `"TTL constants are reasonable"` in store.zig asserts that
property rather than just the number.
Known blind spots, all left to the expiry time rather than papered over
with a more eager statistic that would cost requests on every symbol to
protect against a rare event on one:
- A **cadence change** (quarterly to monthly) keeps the old median for
several periods, so the forecast runs late.
- An **off-cycle special** is on no schedule, so prediction never fires
for it.
- A symbol with **fewer than three cached records** has no cadence at all
-- which is exactly the case that gets the 6-day expiry.
#### Why Tiingo can never discover a new dividend
Tiingo returns dividends and splits alongside candles in a single request
(`populateAllFromTiingo`), written through `writeSupplement` so the merge
lands new records without moving the primary provider's `#!expires=`
clock. That makes it look like a free second source for Level 2, and it is
worth being explicit that it is not - for two independent reasons.
**Structural.** Tiingo carries a dividend as a `divCash` field on a daily
price bar, so a payment can only appear once a bar exists for it. A
declared-but-not-yet-ex dividend has no bar and therefore cannot be
represented at all. This is exactly why Polygon is the primary source -
see the comment in `fetchFromProvider`, which cites a forward-dated ARCC
ex-date as the case Tiingo's response cannot express.
**Circular.** `populateAllFromTiingo` is reached from only two places:
1. A **cold candle cache** - the symbol has no bars at all
(`getCandles`'s full-history path). Here Tiingo genuinely earns its
keep: one request bulk-loads the symbol's entire dividend history.
2. An **adjustment-basis restatement** - cached `adj_close` values predate
a corporate action they should reflect, so the series is refetched in
full.
Trigger 2 is decided by `freshness.adjustmentBasisStale` fed by
`freshness.newestCorporateAction`, and that function reads the **cached
dividend file** to find the newest action. So zfin only knows to restate
*after* the dividend record is already on disk - which means Polygon must
have supplied it first. By the time Tiingo's rows merge in, they add
nothing about that payment.
Net: Tiingo bulk-loads history and fills field-level gaps in old records.
Every *new* dividend arrives via Polygon. This is also why the Level 3
floor reads `#!expires=` and not `#!created=` - a Tiingo supplement
rewrites the file, and therefore `created`, without having asked Polygon
anything.
For the user-facing version of all this, including a worked quarterly
example, see
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
### `loadAllPrices` (portfolio + watchlist price load)
The portfolio price load batches all symbols through three phases.
Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
fallback that calls `getCandles`.
```mermaid
flowchart TD
ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
P1 --> CF{"cache fresh and not force_refresh?"}
CF -->|yes| HIT["use cached last close, cached_count++"]
CF -->|no| ADD["add to needs_fetch"]
HIT --> CHK
ADD --> CHK{"needs_fetch empty?"}
CHK -->|yes| DONE["return"]
CHK -->|no| OFF{"skip_network?"}
OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
P2 --> REM["unsynced to server_failures"]
ALLF --> P3
REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
P3 --> END["return prices + counts"]
```
## Negative caching
When a provider says a symbol genuinely has no data of a type - an
`error.NotFound` - zfin writes a **negative cache entry** so it does not
re-run the dead lookup on every invocation. The entry is the sentinel:
```
#!srfv1
# fetch_failed
```
(`Store.negative_cache_content`). Rules:
- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
write. Rate-limit, 5xx, connection, auth, and parse failures are
transient - they fail the call but leave the cache untouched so the
next run retries. (Auth/parse looking permanent but being transient is
exactly why they must not poison a now-sticky negative cache.)
- **Negative entries are always fresh.** They have no `#!expires=`;
`readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
as fresh, so they stick until `--refresh-data=force` or `cache clear`.
- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
`getCandles` short-circuit all recognize it there. `candles_meta.srf`
is intentionally not created for a no-data symbol.
- **Externally-managed series are exempt.** A prior meta saying
`provider::external` vetoes the write via
`DataService.shouldNegativeCache`, because for a series no provider was
ever going to carry, the marker is unrecoverable rather than merely
sticky. See [Externally-managed candle
series](#externally-managed-candle-series).
## Candle-less symbols (crypto and friends)
Some held symbols have **no daily candles available from any candle
provider** - cryptocurrencies on the Yahoo `DOGE-USD` /
`BTC-USD` shape are the common case, and delisted or invalid tickers
behave identically. For these symbols `getCandles` writes a negative
entry and never produces a price from history. "Any" is literal: Tiingo
and Yahoo must both disclaim the symbol, because the marker overwrites
`candles_daily.srf`.
Such symbols are still priced, through two mechanisms that do **not**
touch the candle cache:
- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
intraday price from Yahoo, which *does* serve crypto. The TUI overlays
these live quotes on top of the candle-close price map on refresh and
on every streaming tick, so a candle-less holding shows its real
current price there. Live quotes are never cached.
- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
value. When neither a candle close nor a live quote is available,
`buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
position's average cost and flags it as a manual/estimated price
(rendered in a warning color).
Practical consequence by surface: the plain CLI `portfolio` command does
not apply the live-quote overlay, so a candle-less holding shows its
average-cost fallback (break-even, warning color); the TUI shows the
live Yahoo price. This is expected - historical-candle commands (`perf`,
charts) simply have no data for these symbols, while quote-driven views
do.
If you want a candle-less symbol to be re-checked against the provider
(for example a ticker that has since started trading), clear its
negative entry with `cache clear` or `--refresh-data=force`; the live
quote and manual-price paths are unaffected by the negative cache.
## Externally-managed candle series
`CandleMeta.provider == .external` labels a series that was produced and
is maintained **outside zfin**, by something that writes
`candles_daily.srf` and `candles_meta.srf` directly into a cache
directory. The motivating case is a unitized trust with no ticker and no
CUSIP, whose daily unit values come from a plan recordkeeper's feed;
every market-data provider 404s it.
Like the previous section, this is a symbol no provider carries. The
difference is that a candle-less symbol has no history *anywhere*, so
negative-caching it is correct, whereas an externally-managed symbol has
a real series that simply did not come from a provider. The label exists
so `candles_meta.srf` can say so honestly instead of naming a provider
that never served it.
Two population routes:
- **Server**: a job writes the pair straight into the `ZFIN_SERVER`
cache directory. That copy is the master.
- **Client**: the existing server-sync path (phase 2 of `loadAllPrices`,
and the two `syncCandlesFromServer` calls inside `getCandles`) pulls
the bytes down verbatim, exactly as it does for any other symbol. No
client-side special-casing is involved.
The label is **pure provenance and drives no routing**. zfin still walks
the normal provider chain for such a symbol and still 404s on every pass;
restricting it to the server tier and skipping the chain entirely is
deferred work. There is exactly one behavior attached to the label, and
it is a veto rather than a routing decision:
**A negative-cache entry on an external symbol is unrecoverable.**
`writeNegative` overwrites `candles_daily.srf` with a marker, negative
entries never expire, and `getCandles` short-circuits on `isNegative`
*before* it reaches `syncCandlesFromServer` - so the marker makes the
only surviving copy permanently unreachable. On a server it is worse:
the externally-populated cache is the master copy, so nothing survives.
`DataService.shouldNegativeCache` therefore refuses the write when the
prior meta says `.external`.
That guard is deliberately partial. It consults the prior meta, so it
covers the case that destroys data: meta present, `candles_daily.srf`
missing or unreadable (a partially-completed server sync writes the two
files in sequence and needs both to report success), which falls through
to the cold-start path. It cannot cover a **true cold start** with both
files absent - there is no label to read - so an external symbol first
touched while `ZFIN_SERVER` is unreachable is poisoned until
`--refresh-data=force` or `cache clear`. Closing that needs the
`isNegative` short-circuit moved after the server tier, which would
re-hit the network for every legitimately candle-less symbol on every
run. It belongs with the deferred routing work.
### The label is not sticky, and must not become sticky
`external` describes where a series came from *now*, not a permanent
property of the symbol. If a provider ever does serve it, the bars in
the cache genuinely came from that provider, and `applyTiingoCoverage`
overwriting the label with `.tiingo` / `.yahoo` is the correct outcome:
the provenance changed, and the symbol has rejoined the ordinary
provider path under its own steam. That is a good day, not a bug.
So do **not** add a guard preserving `.external` across a successful
fetch, and any future routing built on the variant must keep that escape
hatch open. Making it a one-way door recreates the latch bug that keying
routing off `provider == .yahoo` already produced - see [the
`tiingo_retry_after_s` rationale](#getcandles-single-symbol) - except
worse, because the pinned symbol would be pinned to a tier that only one
machine can populate.
### Adding a `CandleProvider` variant is a coordinated deployment
Not a backward-compatible change. `provider` has no default, so it is
never elided, and SRF's enum coercion has no lenient mode and no
`srfParse` hook (custom parsers are consulted for struct/union fields
only). A build that does not know a variant cannot fall back:
- srf `>= ea2c358` (zfin's current pin) returns
`StringValueNotValidEnumMember`. `readCandleMeta` swallows it into a
`null`, which reads as a cache miss and routes the symbol down the
cold-start path above - destructive for precisely the series that
cannot be re-fetched.
- srf `<= 4a3e5f0` unwraps a null optional and **panics**. That is what
zfin-server `7103ced` vendors, and a panic is not interceptable by the
`catch return null` every caller depends on. The blast radius is not
one symbol: `handleDiagnostics` runs `freshness.collect` over every key
in the cache directory, so a single unreadable file takes down
`//diagnostics` and the cron refresh sweep. The
byte-serving routes (`/candles`, `/candles_meta`) are unaffected, since
they never parse.
Skew is dangerous in both directions: an old **client** syncing
`candles_meta` bytes from a new server fails the same way, because
`looksCompleteSrf` and the ETag check validate shape and integrity, not
enum values.
So the ordering is: release zfin, then rebuild and deploy every server
that reads the affected cache directory, and only then may a file
carrying the new value exist.
## Server sync (the optional L2 tier)
`ZFIN_SERVER` points zfin at a
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
shared cache between your local cache and the upstream providers. When
unset, every server-sync path silently no-ops.
Client side (`syncFromServer`, `syncCandlesFromServer`,
`parallelServerSync` in `service.zig`):
- Triggered on a local miss/stale entry, before any provider call
(skipped under `force_refresh`).
- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
(sha256 ETag, completeness check) and written to the local cache
**verbatim** via `writeRaw` - the client does not re-stamp
`#!expires=`, so it inherits the server's freshness boundary.
- `parallelServerSync` fans out one task per symbol for the portfolio
price load (each worker uses its own HTTP client; the allocator is
thread-safe).
Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):
- A **present** file is served as-is, even if stale - the server's cron
is the freshness authority; reads never trigger a refetch.
- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
same `getCandles` / `fetchCached` code through the shared zfin
library), then re-reads; if still absent, it returns 404.
Because the server runs the same library, the negative-cache rules above
apply there too: a candle-less symbol gets a negative `candles_daily.srf`
on first miss and is served from it thereafter, rather than re-hitting
the upstream provider on every request.
## Invalidation and atomicity
- `DataService.invalidate(symbol)` clears a symbol's entries; for
candles it removes the `candles_daily` + `candles_meta` pair together.
- `cache clear` wipes the whole cache directory; everything re-fetches
next run.
- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
and renames into place, so a reader never sees a torn file. A
defensively detected torn candle file self-heals by wiping the pair.
## Key code references
| Concern | Location |
|---------------------------------|------------------------------------------------------------------------|
| Data-access entry point | `DataService` - `src/service.zig` |
| Per-type generic fetch | `fetchCached` - `src/service.zig` |
| Fresh-but-incomplete decision | `serveFreshOrDiscard` - `src/service.zig` |
| Recheck floor (12h) | `askedRecently`, `smart_refresh_recheck_interval` - `src/service.zig` |
| Dividend schedule prediction | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` |
| Dividend expiry choice | `dividendTtl`, `ttlSpecFor` - `src/service.zig` |
| Earnings smart refresh | `earningsNeedsRefresh`, `earningsNeedsRefreshHook` - `src/service.zig` |
| Candle fetch + incremental | `getCandles` - `src/service.zig` |
| Batch price load | `loadAllPrices` - `src/service.zig` |
| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` |
| Cache store, read/write | `Store` - `src/cache/store.zig` |
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| Negative-cache veto | `shouldNegativeCache` - `src/service.zig` |
| Candle provenance labels | `CandleProvider` - `src/cache/store.zig` |
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
| Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
For the user-facing summary and the `--refresh-data` walkthrough, see
[Caching and data freshness](../explanation/caching.md).